What is computer vision?
This lecture defines computer vision (CV), lists its goals and applications, and explains why CV is difficult. It also shows how the CSE411 course is organised.
The goal: perceive the story behind a picture
A photograph is a grid of numbers. A human looks at it and sees "a bus on a street, a flag, a man walking". Computer vision makes a computer get that description from the pixels. The slides define it as: compute properties of the world from images. These properties are 3D shape, the names of people and objects, and what happened.
Definition. Computer vision builds algorithms that take images or video as input. The output is a useful description of the scene: geometry (3D shape, depth, motion), semantics (objects, people, scene type, activity) or an improved image.
Can computers match human perception?
The slides answer "yes and no (mainly no)".
- Computers are better at "easy" tasks. They measure exact distances, read thousands of licence plates per minute and do not get tired.
- Humans are better at "hard" tasks: context, rare situations and intent.
- Deep learning made progress much faster after 2015. The boundary of "hard" continues to move.
Humans also make errors: optical illusions fool us. But we recognise a scene from a 32x32 thumbnail (the "80 million tiny images" example by Torralba et al.). Human vision uses prior knowledge to fill in what the pixels do not show.
Goals of computer vision (slide order)
| Goal | What it means | Example |
|---|---|---|
| Compute 3D shape of the world | Recover depth and geometry from images | ZED 2i stereo camera |
| Recognise objects and people | Name every region: sky, building, bus, car, face, flag, street lamp | Terminator 2 (1991) "vision"; scene labelling by Fei-Fei, Fergus, Torralba |
| "Enhance" images | Sharpen, denoise, zoom | Film-style "enhance!" |
| Forensics | Recover hidden information from images | "Eyes for Relighting" (Nayar and Nishino): the reflection in a person's cornea shows what the person looked at and the lighting |
| Improve photos (computational photography) | Combine software and sensors to go past hardware limits | Super-resolution, phone depth-of-field (portrait mode), Google Magic Eraser (object removal), low-light photography (Hasinoff et al., SIGGRAPH Asia 2016) |
Why study computer vision?
People capture billions of images and videos every day. Humans never look at most of them, so software must analyse them. The table lists the applications from the slides.
Applications from the slides
| Area | Examples |
|---|---|
| Optical character recognition (OCR) | Scanner software; digit recognition at AT&T Labs in the 1990s (LeCun's LeNet); licence plate readers (ANPR); sudoku grabber; automatic cheque reading |
| Face detection | Almost every camera detects faces in real time (for focus and "smart" cropping) |
| Face analysis and recognition | Identify a person, estimate expression |
| Vision-based biometrics | Iris patterns (the "Afghan girl" was identified by her iris), fingerprints, face unlock; login without a password |
| Bird identification | Merlin Bird ID (Cornell Tech) |
| Special effects | Shape capture (The Matrix), motion capture (Pirates of the Caribbean), 3D face tracking with consumer cameras (Snapchat lenses, Face2Face) |
| Image synthesis | GANs (Progressive GAN faces, "Which face is real?"); text-to-image models: DALL-E 2, Imagen |
| Sports | Sportvision first-down line in American football broadcasts |
| Smart and self-driving cars | Mobileye, Tesla Autopilot, lane and collision warnings; Waymo |
| Robotics | NASA Mars Curiosity rover, Amazon Picking Challenge, Amazon warehouse robots |
| Medical imaging | 3D imaging (MRI, CT); skin-cancer classification with deep learning (Esteva et al., Nature) |
| Virtual and augmented reality | 6-DoF head tracking, hand and body tracking, 3D scene understanding, 3D-360 video capture |
A frequent question is "list any five applications of computer vision with one line each". Take one from each row: OCR, face unlock, self-driving, medical imaging, AR/VR. Add one line on how it works. Example: face unlock = face detection, then embedding, then match.
Current state of the art. CV is an active research area that changes fast. Deep learning drives many modern applications. Many startups work in robotics, autonomous vehicles, medical imaging, construction, inspection, VR/AR, precision agriculture and GeoAI.
Why is computer vision difficult?
One object can give very different pixel values. Learn all the challenges from the slides:
| Challenge | Meaning | Example |
|---|---|---|
| Viewpoint variation | The object looks different from different camera positions | A chair from the front vs from above |
| Illumination | Lighting changes pixel values, shadows and highlights | One face in sun vs in a dark room |
| Scale | The object size in the image changes with distance | A car close up vs far away |
| Motion | Moving objects blur and change shape | A running dog, a shaking camera |
| Intra-class variation | Objects of one category look very different | Hundreds of chair designs, all "chair" |
| Background clutter | The object blends into a busy background (camouflage) | Leopard in dry grass |
| Occlusion | Part of the object is hidden | Person behind a tree |
| Local ambiguity | A small patch alone does not show what the object is | A patch of gray pixels can be a car roof or a wall (Fei-Fei, Fergus, Torralba) |
Local ambiguity: a tiny patch alone does not tell you what it is. Humans use the context around the patch. The slides say there are "lots of visual cues we can use": shading, perspective, familiar sizes and texture gradients.
Bottom line: perception is ambiguous
Image formation loses depth. The camera projects a 3D world onto a 2D image. Thus many different 3D scenes can give the same 2D image. The slide shows Julian Beever's anamorphic pavement art: a Coke bottle drawn flat on the ground looks 3D from one viewpoint. To remove the ambiguity, a vision system must use prior knowledge about the structure of the world: objects are solid, the ground is flat, light comes from above. For this reason, modern systems learn priors from data.
Do not write "vision is hard because computers are slow". The root cause is that the problem is inherently ambiguous (many 3D scenes, one 2D image). Large variation (viewpoint, light, scale, clutter, occlusion) adds to it.
Computer vision vs related fields
Lecture 2 separates the related fields by their input and output.
| Output: image | Output: description | |
|---|---|---|
| Input: image | Digital image processing | Computer vision |
| Input: description | Computer graphics | Artificial intelligence |
Image processing improves an image. Computer vision understands it. Graphics renders an image from a model of the scene. Thus CV is the inverse of graphics.
Levels of vision and course roadmap
The lecture ends with low-level vision: basic image processing and image formation. Typical operations are filtering (convolution, written $f * h$), edge detection, feature extraction and image formation.
$f$ = input image, $h$ = filter kernel, $g$ = filtered output. Edge detection and blurring both use this operation. Only the kernel is different.
The CSE411 syllabus (Dr. Alkha Mohan) uses the usual low, mid and high level split:
| Unit | Topics |
|---|---|
| Introduction | Human visual system, why vision is difficult, retina, LGN and primary visual cortex, computational models of recognition |
| Features and filters (low level) | Linear filters, edges and contours, binary image analysis, background subtraction, texture, motion and optical flow |
| Grouping and fitting (mid level) | Segmentation, clustering, Hough transform, fitting lines and curves, robust fitting, RANSAC, deformable contours, interactive segmentation |
| Multiple views | Local invariant features, image transformations and alignment, planar homography, epipolar geometry and stereo, object instance recognition |
| Recognition (high level) | Categorisation, detection, supervised classification, probabilistic models for sequences, attributes, dimensionality reduction, deep learning and CNNs |
| Applications | OCR, face detection and recognition, motion estimation, human tracking, gesture recognition |
| Lab | Projective geometry; Harris, Hessian, LoG, DoG, SIFT, PCA-SIFT, GLOH, SURF, HOG; camera calibration methods; Roberts and Sobel edges; thresholding segmentation; morphology |
Course objectives:
- Understand the principles of image formation, image processing and visual perception.
- Analyse feature extraction, segmentation and stereo vision.
- Apply algorithms for detection, recognition and scene understanding.
- Build real-world solutions with modern machine learning.
Evaluation: theory 70 marks (continuous 30 = two quizzes 20 + assignment 10; mid-sem 25 scaled to 50-mark paper; end-sem 45% of a 100-mark paper) and lab 30 marks (weekly 15, lab exam 5, project plus viva 15; group project of 4 to 5 students on a social-good, SDG-linked CV application). Main text: Szeliski, Computer Vision: Algorithms and Applications; also Forsyth and Ponce.
Quick self-test
1. Define computer vision and state its three broad goals from the lecture.
CV gets a useful description of the world from images or video. Goals: (a) compute 3D shape and geometry, (b) recognise objects, people and activities, (c) enhance or improve images (computational photography, forensics).
2. Why is vision an ill-posed / ambiguous problem?
Projection from 3D to 2D loses depth. Thus many different 3D scenes give the same image (e.g. Julian Beever's pavement art). We must add prior knowledge about the world to select the most probable interpretation.
3. List six challenges that make object recognition difficult.
Viewpoint variation, illumination, scale, motion, intra-class variation, background clutter, occlusion, local ambiguity. Give any six, each with a one-line example.
4. Differentiate image processing, computer vision, computer graphics and AI by input and output.
Image to image = image processing. Image to description = computer vision. Description to image = computer graphics. Description to description = AI.
5. Name five real applications of CV and the task each solves.
OCR (read text, licence plates), face unlock (biometric identification), self-driving (detect lanes, cars, pedestrians), medical imaging (skin cancer classification, MRI/CT 3D), AR/VR (6-DoF tracking), sports first-down line, image synthesis (GANs).
6. Can computers match human perception? Justify.
"Yes and no, mainly no". Computers are better at precise, repetitive or large-scale tasks. Humans are better at "hard" tasks that need context. Deep learning after 2015 made the gap smaller, and the boundary of "hard" continues to move.
7. What is local ambiguity? How do humans resolve it?
A small image patch alone does not show what it is. Humans use the context around it and prior knowledge: shading, perspective, familiar size.
8. What does low-level vision cover, and what are mid-level and high-level vision?
Low level: image formation, filtering, edges, features. Mid level: grouping and fitting (segmentation, Hough, RANSAC). High level: recognition, detection, scene and activity understanding.
- CV changes pixels into a description of the world: 3D shape, objects and people, activities, or better images.
- Computers are better at "easy" precise tasks. Humans are better at "hard" context tasks. Deep learning (after 2015) closed much of the gap.
- Applications: OCR, face detection and recognition, biometrics, special effects, image synthesis (GANs, DALL-E, Imagen), sports, smart and self-driving cars, robotics, medical imaging, AR/VR.
- Challenges: viewpoint, illumination, scale, motion, intra-class variation, clutter, occlusion, local ambiguity.
- Root cause: 3D to 2D projection is ambiguous, so the system needs priors about the world.
- Image to image = image processing; image to description = CV; description to image = graphics.
- Course path: low level (filters, edges), then mid level (segmentation, fitting), then multiple views, then high-level recognition (deep learning).