Recognition & high-level vision
Low-level vision gives edges and corners. High-level vision answers three questions: what is this, where is it, and what is happening? This page covers the 9-level understanding hierarchy, the main recognition tasks (object, scene, activity, detection), visual attributes and active learning.
What high-level vision covers
The lecture plan has these topics: object, scene and activity categorisation, object detection, supervised classification algorithms, probabilistic models for sequence data, visual attributes and active learning. This lecture (L16) gives the big picture. L17 covers the classifiers and sequence models. L18 covers detection.
Hierarchical understanding
Hierarchical understanding means to interpret an image at several levels of abstraction, from raw numbers up to a full semantic story. Humans do it naturally. They find edges and shapes, recognise objects, see how the objects relate, name the environment and then infer the activity or event.
Compare it to reading. Pixels are ink strokes, features are letters, parts are words, objects are sentences, and semantic understanding is the meaning of the paragraph. Each level uses the level below it.
Slide example: a child plays football in a park. A system does these steps:
- Detect edges, corners and textures.
- Find shapes that look like a person and a ball.
- Label the shapes "child" and "football".
- Recognise the environment as "park".
- Understand that the child kicks the ball.
- Infer that a football game takes place.
The levels (low to high)
| Level | What the system knows | Example / techniques |
|---|---|---|
| 1. Pixel | Only numbers: intensity, RGB values, coordinates | Pixels like [120,130,125]. Operations: brightness and colour analysis, noise removal, enhancement |
| 2. Feature | Edges, corners, blobs, texture patterns | Sobel, Canny, Harris, SIFT, SURF, ORB. Output: edges, corners, texture descriptors, keypoints |
| 3. Object part | Meaningful parts of objects | Human: head, arms, legs, torso. Car: wheels, doors, windshield. Helps when the object is partly occluded. Use: pose estimation, face analysis, body tracking |
| 4. Object | Complete objects, with a confidence | Person, football, goalpost, tree. Traditional: bag of visual words, SVM, random forest. Deep: CNN, ResNet, ViT. Output e.g. "Football, 98%" |
| 5. Relationship | How objects relate to each other | Person holding a ball, car parked near a building, dog sitting on a sofa. Gives context |
| 6. Scene | The overall environment | Trees + grass + bench + playground gives "park". GIST descriptors, CNN scene classifiers, Places365 |
| 7. Activity | What action is being performed | Person + football + foot near ball gives "playing football". Optical flow, RNN, LSTM, 3D CNN, Transformers |
| 8. Event | Several activities combined into a bigger event | Running + kicking + goalkeeper diving + crowd cheering gives "football match" |
| 9. Semantic understanding | Meaning and context of the whole scene | Flooded road + stranded vehicles + heavy rain gives "flood disaster". Approaches human-like understanding |
The slide infographic shows 8 stacked layers (pixel to event). The written slides add a 9th layer, semantic understanding. If the question says "list the levels in the hierarchy", write 9 levels. Add that the picture merges the last level into the event level.
Exam favourite: "Explain hierarchical understanding with an example." Use the child-and-football chain. Name the levels with one technique each (Level 2: Harris/SIFT, Level 4: CNN/SVM, Level 6: GIST/Places365, Level 7: 3D CNN/LSTM).
Object categorisation (image classification)
Object categorisation assigns one class label to the whole image, based on the main object in it. It answers "what object is in this image?". The model takes an $H\times W\times C$ pixel array. It outputs a probability distribution over a fixed set of categories.
Input: a $224\times224$ RGB image of a cat on a rug. Output: "Cat" with confidence 0.98. Input size here is $224\times224\times3 = 150{,}528$ numbers. Applications: photo organisation, species identification apps.
Scene categorisation
Scene categorisation classifies the global environment, not single foreground objects. It answers "where was this image taken?". The model uses global texture, background patterns, lighting and spatial layout across the full frame.
Input: sand, ocean waves, blue sky and a beach umbrella. Output: "Beach" (or "Coastline"). Applications: context-aware robotics, automated video indexing, environmental monitoring.
Activity categorisation (action recognition)
Activity categorisation identifies the behaviour or event in a scene. It usually uses a sequence of video frames, but it can also use a pose in a still image. It answers "what is happening?". It needs appearance (pixels) and motion (optical flow or frame sequences). Thus it uses 3D CNNs or video Transformers.
Input: a 3-second clip of a person running up to a bar and jumping over it. Output: "High jump". Applications: sports analytics, surveillance anomaly detection, human-computer interaction.
Object detection (the slide's output format)
Detection means to localise every object and classify each one. It answers "what objects are present, and where are they?". Algorithms such as YOLO and Faster R-CNN give three outputs for each object: a bounding box $[x_{min}, y_{min}, \text{width}, \text{height}]$, a class label and a confidence score. The Object detection page (L18) covers this in full.
A single frame of an intersection gives three detections:
- Car at $[x:120, y:340, w:200, h:150]$
- Pedestrian at $[x:450, y:310, w:50, h:120]$
- Traffic light at $[x:300, y:50, w:40, h:90]$
Convert the car to corner format: $x_{max}=120+200=320$ and $y_{max}=340+150=490$. So $(x_{min},y_{min},x_{max},y_{max})=(120,340,320,490)$. Pedestrian: $(450,310,500,430)$. Traffic light: $(300,50,340,140)$.
| Task | Question answered | Output |
|---|---|---|
| Object classification | What object is in this image? | One label (+ confidence) for the image |
| Scene classification | Where was this taken? | One scene label from global context |
| Activity recognition | What is happening? | Action label for a clip, uses motion |
| Object detection | What objects, and where? | Boxes + labels + scores, many per image |
Classification says "there is a car". Detection says "a car at these coordinates". A detection image can contain several objects. A classification image gets one label.
Visual attributes
Visual attributes are human-understandable properties of objects, scenes or people. A system can read them from visual data. Recognition says what it is. Attribute recognition says what it looks like. Example, a dog image: category = dog, attributes = brown, furry, four-legged, large, standing.
Category names alone are often not enough. Attributes give richer descriptions. They also describe things without a trained category (for example, an unseen animal that is "striped, four-legged, large").
| Attribute type | Describes | Examples |
|---|---|---|
| Colour | Colour properties | red, brown |
| Shape | Geometric appearance | circular, square, rectangular, triangular, oval |
| Texture | Surface patterns | smooth, rough, striped, spotted, glossy |
| Material | Composition | wooden, metallic, plastic, glass, rubber |
| Human | Used in surveillance and person recognition | male, female, tall, short, wearing hat, wearing glasses |
| Scene | Environments | indoor, outdoor, natural, urban, sunny, cloudy |
Active learning
Labelling data is expensive (a doctor who labels X-rays costs much money). In active learning, the model chooses the most informative samples. It asks a human to label only those samples. The goal is to minimise the annotation effort for a given accuracy.
The loop (slide order) has these steps: unlabelled data, initial training set, train, select the most informative samples, a human labels them, retrain, repeat.
Components
- Oracle: the human annotator (doctor, researcher, domain expert) who gives the correct labels.
- Learner: the machine learning model.
- Query strategy: the rule that decides which samples get labelled.
Query strategies
Uncertainty sampling (ask about what the model is least sure of):
- Least confidence: pick the sample whose top class probability is smallest.
- Margin sampling: $\text{margin} = p_{(1)} - p_{(2)}$ (top two probabilities). Small margin means high uncertainty.
- Entropy sampling: $$ H = -\sum_i p_i \log p_i $$ $p_i$ is the predicted probability of class $i$. Higher entropy means more uncertainty. Maximum when all classes are equally likely.
Practice numerical. The model gives outputs for two images over 3 classes: $A=(0.5,0.3,0.2)$, $B=(0.9,0.05,0.05)$. Which image must the oracle label?
Least confidence: the top probability of A is 0.5 and of B is 0.9. A is less confident.
Margin: A: $0.5-0.3=0.2$. B: $0.9-0.05=0.85$. A has the smaller margin.
Entropy (log base 2): $H_A=-(0.5\log_2 0.5+0.3\log_2 0.3+0.2\log_2 0.2)=0.5+0.521+0.464=1.485$ bits. $H_B=-(0.9\log_2 0.9+2\times0.05\log_2 0.05)=0.137+0.432=0.569$ bits. A has higher entropy.
All three criteria agree: label A.
- Query by committee: several models vote. Example: Model A says Cat, Model B says Dog, Model C says Cat. The disagreement is high, so request a label.
- Diversity sampling: do not pick near-identical samples. Choose varied samples (different colours, viewpoints, backgrounds).
- Density-based sampling: pick representative samples from dense regions. Do not pick outliers.
Typical short answer: define active learning and draw the loop. Name the three components (oracle, learner, query strategy). List the uncertainty (least confidence, margin, entropy), committee, diversity and density strategies. A good combination in practice is uncertainty plus density. It picks samples that are unsure and representative, not outliers.
Quick self-test
What is hierarchical understanding in CV? Give the chain for "child playing football in a park".
It is the interpretation of visual information at several abstraction levels, from low-level features to full semantics. The chain: edges/corners/textures, then person and ball shapes, then "child" and "football", then "park", then "child kicks the ball", then "a football game takes place".
Which level of the hierarchy helps with partially occluded objects, and why?
The object part level helps. The door and wheels of a car can be visible when the body is hidden. Then part-based recognition can still infer the car. Examples: head/arms/legs for people, wheels/doors/windshield for cars.
Difference between object categorisation, scene categorisation and object detection?
Object categorisation: one label for the main object ("what object?"). Scene categorisation: global label from background, texture and layout ("where?"). Detection: boxes + labels + scores for every object ("what and where?").
A detector outputs a car at [x:120, y:340, w:200, h:150]. Give the (xmin, ymin, xmax, ymax) form.
$x_{max}=120+200=320$, $y_{max}=340+150=490$. So $(120,340,320,490)$.
What are visual attributes? Why are they useful beyond object categories?
They are human-understandable properties (colour, shape, texture, material, human, scene attributes). Categories answer "what is it?". Attributes answer "what does it look like?". Attributes give richer descriptions and can describe unseen categories.
Why use active learning? Name its three components.
Labelling is costly. Active learning labels only the most informative samples. Components: oracle (human annotator), learner (model), query strategy (which samples to label).
Two predictions: P=(0.6,0.3,0.1) and Q=(0.34,0.33,0.33). Which one does entropy sampling pick and why?
Q. Its probabilities are almost uniform, so entropy is near the maximum $\log_2 3=1.585$ bits. P has entropy $\approx 1.30$ bits (0.442+0.521+0.332). Q is closer to uniform, so it is more uncertain.
What is the difference between uncertainty sampling and query by committee?
Uncertainty sampling uses one model and picks samples with low confidence, small margin or high entropy. Query by committee uses several models and picks samples on which they disagree most.
- High-level vision moves from pixels to features to parts to objects to relationships to scenes to activities to events to semantics.
- Pixel level: raw numbers. Feature level: Sobel/Canny/Harris/SIFT/ORB. Part level: helps occlusion. Object level: SVM/CNN/ViT. Scene: GIST, Places365. Activity: optical flow, LSTM, 3D CNN.
- Object categorisation = one label per image; scene categorisation = global context; activity recognition = motion over frames; detection = boxes + label + confidence.
- Attributes describe appearance (colour, shape, texture, material, human, scene).
- Active learning cuts the labelling cost. The oracle labels what the learner is unsure of, chosen by uncertainty (least confidence, margin, entropy), committee, diversity or density strategies. Entropy is $-\sum p\log p$.