Eye, image formation, colour, cameras
This lecture shows how light becomes numbers. It starts with the human eye. Then it explains how a camera forms and digitises an image, and how pixels, bits and resolution work. It ends with the colour models (RGB, CMY, CMYK, HSI, YCbCr), with all conversion formulas and worked numericals.
Human visual system
Vision is a pipeline. The eye is the camera (sensor), the neurons are the cable (optic nerve), and the brain is the processor. The lecture has three parts: the structure of the eye, image formation in the eye, and brightness adaptation and discrimination.
Structure of the eye
Light enters through the cornea and the pupil. The pupil is a small hole, and the iris controls its size. The pupil contracts in bright light. The lens focuses the light on the retina, which works like camera film. The eye has three coats (sclera/cornea, choroid, retina). They enclose the clear aqueous humour, the lens and the vitreous body (clear jelly). The optic nerve sends signals to the brain. The macula/fovea is the small central area of sharpest vision. The slide also labels these parts: conjunctiva, ciliary body, suspensory ligaments and trabecular meshwork. The ciliary body produces aqueous humour, and its muscles change the lens shape.
Photoreceptors: cones and rods
| Feature | Cones | Rods |
|---|---|---|
| Number | 6 to 7 million | 75 to 150 million |
| Location | Mostly in the fovea (centre) | Distributed over the whole retinal surface |
| Sensitivity | Highly sensitive to colour, need bright light | Sensitive to low light, no colour |
| Vision type | Photopic (bright-light) vision | Scotopic (dim-light) vision |
| Detail | Fine detail (each cone has its own nerve) | Coarse overall view (many rods share one nerve) |
Mnemonic: Cones = Colour and day. Rods = night (and peripheral) vision. Rods are about 20 times more numerous.
Image formation in the eye
The lens focuses the outside world on the retina. The photoreceptors detect photons and send neural impulses. Unlike a camera lens, the eye's lens has a fixed distance to the retina (about 17 mm). It focuses when the ciliary muscles change its shape: thick for near objects, flat for far objects.
$H$ = real object height, $D$ = distance to the lens centre, $h$ = image height on the retina, 17 mm = lens-to-retina distance.
Slide example (tree 15 m high, 100 m away).
Step 1: $H/D = 15/100 = 0.15$.
Step 2: $h = 17 \times 0.15 = 2.55$ mm.
The tree forms an inverted image, 2.55 mm tall, on the retina.
Practice: a 2 m person stands 10 m away, so $h = 17\times 2/10 = 3.4$ mm.
Brightness adaptation and discrimination
The eye handles a huge range of intensities (about $10^{10}$ from the scotopic threshold to the glare limit), but not all at once. The eye adapts to one level at a time. Perceived (subjective) brightness is a logarithmic function of the real intensity. The slide graph plots subjective brightness against the log of intensity in mL. It shows the scotopic threshold at the bottom, the glare limit at the top and the photopic curve. It also shows a short "adaptation range" $B_a$ to $B_b$ around the current adaptation level.
- Brightness adaptation: the eye adjusts to the prevailing light level. After exposure to very bright light, a stimulus appears dimmer. The slide says: "an apparent decrease in intensity of a stimulus after exposure to a high level of light". An example is walking from sunlight into a dark cinema.
- Brightness discrimination: the ability to tell small changes in light intensity apart at a specific adaptation level. The Weber ratio $\Delta I_c / I$ measures it. A small ratio means good discrimination.
Do not confuse perceived brightness with measured intensity. Name two effects: Mach bands (edges look over- or undershooting) and simultaneous contrast (a gray square looks darker on a bright background). Judge image-processing results by how the eye sees them.
Image formation
Image formation is the conversion of a 3D scene to a 2D image. It maps what is in front of the camera to the image plane. It has three phases (from the slides):
- A light source illuminates the scene.
- The scene reflects the radiation towards the camera. The camera focuses it.
- The camera sensors sense the radiation. This completes the process.
The incoming light falls on a CCD (Charge Coupled Device) sensor array. CMOS sensors are the common modern alternative. Each sensor cell measures only the total brightness. To get colour, a colour filter array (CFA) sits on top. It is usually the Bayer filter mosaic. The R filter lets only red light through, G only green, B only blue. The Bayer pattern has 50% green, 25% red and 25% blue sites, because human eyes are most sensitive to green. The system estimates the two missing colours at each pixel from the neighbours (demosaicing). The output of this stage is analog. It must be digitised.
| R | G | R | G |
| G | B | G | B |
| R | G | R | G |
| G | B | G | B |
Pinhole camera model (camera models)
The simplest camera is a box with a tiny hole. Each scene point sends one ray through the hole to one image point. The image is sharp but inverted. By similar triangles, a 3D point $(X, Y, Z)$ in camera coordinates projects to the image point $(x, y)$.
$f$ = focal length (pinhole to image plane), $Z$ = depth of the point, $(X,Y)$ = its lateral position. The image scale is $f/Z$, so far objects ($Z$ large) look small. If you place the image plane in front of the pinhole, the minus signs of the inversion vanish. A real lens follows the thin-lens law $\tfrac1f = \tfrac1u + \tfrac1v$, where $u$ = object distance and $v$ = image distance.
Practice. $f = 50$ mm, a pole 2 m tall at $Z = 10$ m.
$y = f\,Y/Z = 50 \times 2/10 = 10$ mm (inverted). If the pole moves to $Z = 20$ m, the image halves to 5 mm.
Thin lens: $f = 50$ mm, $u = 1000$ mm gives $1/v = 1/50 - 1/1000 = 19/1000$, so $v \approx 52.6$ mm.
The eye example above uses the same pinhole (similar-triangles) idea, with $f = 17$ mm.
Image representation: the digital image
A digital image is a numerical representation of a visual scene. Two steps change a continuous image into a digital image:
- Sampling: digitise the spatial coordinates (choose the grid of $M \times N$ points).
- Quantization: digitise the intensity values (round each to one of $L$ levels).
Digital image = sampling + quantization.
$$ f(x,y)=\begin{bmatrix} f(0,0) & f(0,1) & \cdots & f(0,N-1)\\ f(1,0) & f(1,1) & \cdots & f(1,N-1)\\ \vdots & & & \vdots\\ f(M-1,0) & f(M-1,1) & \cdots & f(M-1,N-1)\end{bmatrix}$$$M$ = number of rows, $N$ = number of columns, $f(x,y)$ = intensity at row $x$, column $y$. Each element is a pixel (picture element), the smallest unit of a digital image.
Bits per pixel (bpp) and colour depth
More bits per pixel give more shades and better quality, but also bigger files.
| bpp | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 10 | 16 | 24 | 32 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Colours | 2 | 4 | 8 | 16 | 32 | 64 | 128 | 256 | 1024 | 65,536 | 16,777,216 | 4,294,967,296 |
Numerical. Storage of a 1024 x 768 image.
Grayscale 8 bpp: $1024 \times 768 \times 8 = 6{,}291{,}456$ bits $= 786{,}432$ bytes $= 768$ KB.
RGB 24 bpp: $1024\times768\times 24 = 18{,}874{,}368$ bits $= 2{,}359{,}296$ bytes $= 2.25$ MB.
Binary 1 bpp: $786{,}432$ bits $= 98{,}304$ bytes $= 96$ KB.
Slide question: colour depth of RGB? 8 bits per channel x 3 channels = 24 bpp. This gives $2^{24} = 16{,}777{,}216$ colours.
Intensity, brightness and contrast
- Intensity = the value stored at a pixel (0 = black up to 255 = white for 8 bit).
- Brightness is relative. Pixels X, Y, Z have intensities 10, 50 and 255. Z is the brightest. Y is brighter than X but not as bright as Z.
- Contrast = difference between the maximum and minimum intensities in the image: $C = I_{max} - I_{min}$.
Slide example. Image X has range 20 to 80, image Y has range 50 to 120.
$C_X = 80 - 20 = 60$, $C_Y = 120 - 50 = 70$. So Y has the higher contrast (spread). Y is also the brighter image, because its values are higher on average.
Practice. An image has min 30 and max 230, so contrast $=200$. If you add 20 to every pixel, brightness rises but contrast stays 200.
Do not mix brightness and contrast. Brightness shifts all pixel values up or down (adds a constant). Contrast changes the spread (multiplies).
Pixel resolution vs pixel density
| Resolution | Pixel density |
|---|---|
| Total number of pixels (width x height), e.g. 1920 x 1080 | Number of pixels per inch (PPI/DPI): how close the pixels are to each other |
| Size of the image/display in pixels | Sharpness measure; higher density = sharper |
| Alone does not guarantee quality | Depends on physical size too |
$w, h$ = width and height in pixels, $d$ = screen diagonal in inches.
Numerical. A 1920 x 1080 phone with a 5.5 inch diagonal: $\sqrt{1920^2 + 1080^2} = \sqrt{4{,}852{,}800} \approx 2203$ px. PPI $= 2203/5.5 \approx 400$.
Slide illustration: the same letter "A" at 128x128 (256 PPI), 64x64 (128 PPI) ... down to 2x2 (4 PPI). Fewer pixels per inch give a blocky, unreadable image.
Types of images
| Type | Values | Bits per pixel |
|---|---|---|
| Binary | Only 0 (black) and 1 (white) | 1 |
| Grayscale | Shades from black to white, usually 0 to 255 | 8 (256 shades) |
| Colour (RGB) | Three channels R, G, B, each 0 to 255 | 24 (8 per channel) |
| 0 | 0 | 1 |
| 0 | 1 | 1 |
| 1 | 1 | 1 |
The grid above is a tiny binary image (1 bit per pixel). The highlighted cell is $f(1,1)=1$.
Colour fundamentals
Colour is a powerful descriptor. It helps to identify objects and extract them from a scene. Visible light is a narrow band of the electromagnetic spectrum (roughly 400 to 700 nm). The spectrum has six broad regions with no sharp boundaries: violet, blue, green, yellow, orange, red.
Measuring a light source
| Quantity | Meaning | Unit |
|---|---|---|
| Radiance | Total energy flowing out of the source | Watts (W) |
| Luminance | Energy an observer perceives from it | Lumens (lm) |
| Brightness | Subjective, achromatic notion of intensity (hard to measure) | - |
Telling one colour from another
- Brightness: the achromatic notion of intensity.
- Hue: the dominant wavelength in a mixture of light waves (what we call "red", "green").
- Saturation: purity, the amount of white light mixed into a hue. Saturation is inversely related to the amount of white light added. Pure spectrum colours are fully saturated. Pink is red plus white, so it is less saturated.
Hue and saturation together are the chromaticity. The amounts of red, green and blue that form a colour are the tristimulus values, written $X$, $Y$, $Z$ (the CIE system).
Colour models
A colour model (colour space or system) is a standard way to specify colours. It is a coordinate system with a subspace in which each colour is one point. The lecture covers RGB, CMY, CMYK, HSI (HSV) and YCbCr.
RGB model
Cameras, monitors and TVs use RGB. Red, green and blue are additive primaries: light adds up (R + G + B = white). An RGB image is three component images, one per channel. The number of bits per pixel is the pixel depth. 8 bits per channel give 24 bpp ("full colour", $2^{24}$ colours).
The corners are three primaries (R, G, B), three secondaries (cyan, magenta, yellow), black and white. Any point on the black-to-white diagonal has $R=G=B$, which is a gray.
Numerical (8-bit). (255, 255, 0) = yellow; (128, 128, 128) = mid gray; (255, 128, 0) = orange. A simple grayscale conversion is the average $(R+G+B)/3$. The perceptual version is the Y formula below.
CMY model
Cyan, magenta and yellow are the secondary colours of light. They are also the primaries of pigments (subtractive: ink absorbs light). With values normalised to $[0,1]$:
A surface coated with pure cyan reflects no red light ($C = 1 - R$). The same formula works in reverse: $R = 1 - C$, and so on.
Numerical. RGB $=(0.2, 0.5, 1)$. CMY $=(1-0.2,\ 1-0.5,\ 1-1) = (0.8, 0.5, 0)$.
Check: pure red $(1,0,0)$ gives $(0,1,1)$. That is no cyan and full magenta + yellow, which is red ink. White $(1,1,1)$ gives $(0,0,0)$, which is no ink on white paper.
CMYK model
Equal amounts of cyan, magenta and yellow pigment should give black. In practice they give a dark brown. To get true black and save expensive coloured ink, printing adds a fourth colour, K = black. This is CMYK. A common conversion is $K=\min(C,M,Y)$, then $C' = C-K$, $M' = M-K$, $Y' = Y-K$. Some books also scale by $1-K$.
Practice. $(C,M,Y) = (0.8, 0.5, 0.3)$. $K = 0.3$
$C' = 0.5$, $M' = 0.2$, $Y' = 0$. CMYK $=(0.5, 0.2, 0, 0.3)$.
HSI model
RGB and CMY describe colour as mixing amounts. Humans do not describe colour this way ("a bright, pale green"). HSI (Hue, Saturation, Intensity) decouples the intensity component from the colour-carrying information (hue and saturation). It suits algorithms that use colour descriptions. You can change brightness without a change to the colour. You can also segment by hue under uneven lighting.
- Hue: an angle around the colour circle. Red = 0 degrees, green = 120 degrees, blue = 240 degrees (yellow 60, cyan 180, magenta 300).
- Saturation: purity, the distance from the central intensity axis (0 for gray, 1 for fully pure).
- Intensity: height along the black-to-white axis. The model is a double cone (two pyramids), with white at the top and black at the bottom.
RGB to HSI (R, G, B normalised to $[0,1]$):
$I$ = average of channels. $S$ = 1 minus the white fraction. $\theta$ = angle from the red axis. $H$ = hue in degrees (divide by 360 if normalised).
HSI to RGB (first multiply a normalised $H$ by 360 degrees. If $H$ is already in degrees, do not multiply). Use the sector that contains $H$:
Sector $0^\circ \le H < 120^\circ$:
$$ B = I(1-S),\quad R = I\left[1 + \frac{S\cos H}{\cos(60^\circ - H)}\right],\quad G = 3I - (R+B) $$Sector $120^\circ \le H < 240^\circ$: first $H \leftarrow H - 120^\circ$, then
$$ R = I(1-S),\quad G = I\left[1 + \frac{S\cos H}{\cos(60^\circ - H)}\right],\quad B = 3I-(R+G) $$Sector $240^\circ \le H < 360^\circ$: first $H \leftarrow H - 240^\circ$, then
$$ G = I(1-S),\quad B = I\left[1 + \frac{S\cos H}{\cos(60^\circ - H)}\right],\quad R = 3I-(G+B) $$Pattern to memorise: in each sector, the "opposite" colour gets $I(1-S)$. The colour at the start of the sector gets the cosine formula. The third colour is the remainder, so that $R+G+B = 3I$.
Numerical 1: RGB (0.8, 0.4, 0.2) to HSI.
$I = (0.8+0.4+0.2)/3 = 0.4667$.
$S = 1 - 0.2/0.4667 = 1 - 0.4286 = 0.5714$.
Numerator: $\tfrac12[(0.8-0.4)+(0.8-0.2)] = \tfrac12(0.4+0.6) = 0.5$.
Denominator: $[(0.4)^2 + (0.6)(0.2)]^{1/2} = [0.16+0.12]^{1/2} = 0.5292$.
$\theta = \cos^{-1}(0.5/0.5292) = \cos^{-1}(0.9449) \approx 19.1^\circ$. Since $G \ge B$, $H = 19.1^\circ$ (an orange).
Numerical 2: pure red (1, 0, 0). $I = 1/3$, $S = 1 - 0/(1/3) = 1$, numerator $\tfrac12(1+1)=1$, denominator $[1 + 1\cdot 0]^{1/2} = 1$, $\theta = \cos^{-1}1 = 0^\circ$, so $H = 0^\circ$. The colour is fully saturated with hue 0, which is correct.
Numerical 3: gray (0.5, 0.5, 0.5). $S = 1 - 0.5/0.5 = 0$; the denominator is 0, so hue is undefined (gray has no hue).
Numerical 4: back to RGB, $H = 60^\circ$, $S = 1$, $I = 1/3$. Sector 1. $B = I(1-S) = 0$. $R = \tfrac13\left[1 + \frac{\cos 60^\circ}{\cos(0^\circ)}\right] = \tfrac13(1.5) = 0.5$. $G = 3I - (R+B) = 1 - 0.5 = 0.5$. RGB $= (0.5, 0.5, 0)$, a yellow. This is correct for hue 60.
YCbCr (YUV) model
YCbCr is a family of colour spaces for video and digital photography. It separates brightness (luma, Y) from chroma (colour). $Cb$ = blue-difference, $Cr$ = red-difference. The eye is more sensitive to brightness than to colour. So the system can store the chroma channels at lower resolution (chroma subsampling, e.g. 4:2:0). This is the basis of JPEG and MPEG compression. A black-and-white TV uses only Y, so colour broadcasts stayed backward compatible.
$Y$ = luma (the weights reflect eye sensitivity: green highest, blue lowest). $U$ and $V$ = scaled colour-difference signals (analog YUV, as on the slide). In digital YCbCr the scalings are $Cb = 0.564(B-Y)$ and $Cr = 0.713(R-Y)$ (plus an offset of 128 for 8-bit storage).
Numerical 1. RGB $=(100, 150, 200)$.
$Y = 0.299(100)+0.587(150)+0.114(200) = 29.9+88.05+22.8 = 140.75$.
$U = 0.492(200 - 140.75) = 0.492 \times 59.25 = 29.15$.
$V = 0.877(100 - 140.75) = 0.877 \times (-40.75) = -35.74$.
Numerical 2. Pure red $(1,0,0)$: $Y = 0.299$
$U = 0.492(0 - 0.299) = -0.147$
$V = 0.877(1 - 0.299) = 0.615$.
Check: any gray ($R=G=B$) gives $Y = R$ (the weights sum to 1) and $U = V = 0$.
Comparison of colour models
| Model | Components | Type | Used for |
|---|---|---|---|
| RGB | Red, green, blue | Additive (light) | Cameras, displays |
| CMY | Cyan, magenta, yellow ($=1-$RGB) | Subtractive (pigment) | Colour printers (theory) |
| CMYK | CMY plus black | Subtractive | Printing: true black, saves ink |
| HSI / HSV | Hue, saturation, intensity | Perceptual | Colour-based segmentation, human-friendly editing |
| YCbCr / YUV | Luma + two chroma differences | Luma-chroma | Video, JPEG, compression, skin detection |
Additive primaries are R, G, B. Subtractive primaries are C, M, Y. The slide calls cyan, magenta and yellow the "secondary colours of light" and the "pigment primaries". Both statements are true.
Key stages in digital image processing
The lecture ends with the Gonzalez and Woods block diagram. A typical system starts at the problem domain and goes through these stages:
| Stage | What it does |
|---|---|
| Image acquisition | Capture the image with a sensor and digitise it |
| Image enhancement | Make the image look better for a purpose (contrast, sharpening) – subjective |
| Image restoration | Undo known degradation (blur, noise) – objective, uses a degradation model |
| Colour image processing | Process colour channels and colour spaces |
| Image compression | Reduce storage and bandwidth (JPEG) |
| Morphological processing | Extract shape components using erosion, dilation etc. |
| Segmentation | Partition image into regions or objects |
| Representation and description | Convert regions to boundaries or features usable by a computer |
| Object recognition | Assign a label (e.g. "vehicle") to an object from its descriptors |
Enhancement vs restoration is a classic short question. Enhancement is subjective (what looks good). Restoration is objective (it reverses a known degradation).
Quick self-test
1. A tree 15 m high stands 100 m from an observer. Find its height on the retina (lens to retina = 17 mm).
$h/17 = 15/100$, so $h = 2.55$ mm (inverted).
2. Differentiate rods and cones.
Cones: 6 to 7 million, mostly in the fovea, colour and fine detail, photopic (bright-light) vision. Rods: 75 to 150 million, spread over the retina, low-light sensitivity, no colour, scotopic vision.
3. Explain brightness adaptation and brightness discrimination.
Adaptation: the eye adjusts to the prevailing light level, so the same stimulus looks darker after exposure to bright light. Only a limited range around the adaptation level is usable at once. Discrimination: the ability to detect small changes in intensity at a given adaptation level (Weber ratio).
4. What are the three phases of image formation? What does the Bayer filter do?
The phases are illumination of the scene, reflection towards the camera and sensing by the sensor array (CCD). The Bayer CFA places R, G, B filters (50% G, 25% R, 25% B) over the sensor, so each cell records only one colour. The system interpolates the missing colours. The output is analog and must be digitised.
5. Define a digital image. Distinguish sampling and quantization.
A 2D array $f(x,y)$ of pixels. Sampling digitises coordinates (grid size $M\times N$); quantization digitises intensity into $L=2^k$ levels. Digital image = sampling + quantization.
6. How many bits are needed to store a 640 x 480 RGB image (8 bits per channel)? How many colours?
$640\times480\times24 = 7{,}372{,}800$ bits $= 921{,}600$ bytes $\approx 900$ KB. Colours $=2^{24} = 16{,}777{,}216$.
7. Image A has intensities 20 to 80, image B 50 to 120. Compute contrast of each; which is brighter?
$C_A = 60$, $C_B = 70$, so B has higher contrast. B also has higher intensity values, so it is brighter overall.
8. Convert RGB $(0.2, 0.5, 1)$ to CMY, and RGB $(100,150,200)$ to Y.
CMY $=(0.8, 0.5, 0)$. $Y = 0.299\cdot100 + 0.587\cdot150 + 0.114\cdot200 = 140.75$.
9. Why HSI instead of RGB for colour-based processing? Compute H, S, I for (1, 0, 0).
HSI separates intensity from colour information (hue, saturation). This matches the way humans describe colour, so lighting changes affect only $I$. For (1,0,0): $I=1/3$, $S=1$, $\theta=0^\circ$, $H=0^\circ$.
10. Why does CMYK add a fourth colour? Differentiate resolution and pixel density.
Equal C, M, Y give a dark brown, not true black. Adding K gives true black and saves ink. Resolution = total pixel count (e.g. 1920x1080). Pixel density = pixels per inch (sharpness). A 1920x1080 screen of 5.5 in is about 400 PPI.
- Eye = camera (lens, retina), optic nerve = cable, brain = processor. Cones (6-7 M, colour, photopic), rods (75-150 M, dim light, scotopic). Retinal image by similar triangles: $h = 17\cdot H/D$.
- Brightness adaptation (eye adjusts to a light level) and discrimination (detect small changes); perceived brightness is roughly logarithmic.
- Image formation: illumination, reflection, sensing (CCD) with a Bayer CFA (50% G); analog output must be digitised. Pinhole model: $x = fX/Z$.
- Digital image = sampling + quantization; pixel = smallest element; number of colours $=2^{bpp}$; binary 1 bpp, gray 8 bpp, RGB 24 bpp.
- Contrast = max minus min intensity; brightness is relative; resolution (pixel count) differs from pixel density (PPI).
- Colour: brightness, hue, saturation; radiance (W), luminance (lm); tristimulus X, Y, Z.
- Models: RGB (additive cube), CMY $=1-$RGB, CMYK (true black), HSI (decouples intensity), YCbCr/YUV (luma + chroma, $Y=0.299R+0.587G+0.114B$).
- Pipeline: acquisition, enhancement, restoration, colour processing, compression, morphology, segmentation, representation, recognition.