Thresholding & morphology
This lecture turns a grey image into a clean black-and-white mask (thresholding, Otsu). Then it cleans the mask with shape-based operations (erosion, dilation, opening, closing). After this page you can compute the Otsu threshold by hand. You can also do every morphological operation on a small grid.
Binary images
A binary image has only two pixel values: foreground (the object, usually 1 or white) and background (usually 0 or black). It marks the regions of interest and removes everything else. Binary images are cheap to store and easy to analyse. You can count blobs, measure the area and trace outlines.
The slide shows an $8\times 8$ matrix of 0s and 1s next to its black/white picture. A binary image is only a matrix of 0 and 1. Thresholding is the method that makes one from a grey image.
Thresholding
Thresholding converts a grey image to a binary image. It splits the pixels into two classes with their intensity. Pixels above the threshold get one value (usually white). Pixels below the threshold get the other value (usually black). Thresholding is used for image segmentation, object detection and binarization.
Imagine a cut-off height. Everyone taller than the line goes in the "foreground" queue. Everyone else goes in the "background" queue. The only decision is the position of the line.
Techniques named in the slides: global, adaptive, Otsu, Yanni and Horne, Niblack. The first three have detailed sections below. The slide only lists Yanni-Horne and Niblack. Niblack is a local method with $T = m + k\,s$. Here $m$ is the local mean and $s$ is the local standard deviation. The value $k$ is typically negative, about $-0.2$. Yanni-Horne is a histogram-based automatic method.
Global thresholding
Global thresholding applies one fixed threshold $T$ to every pixel. It works when the histogram is bimodal: two humps (dark background, bright object) with a valley between them. Put $T$ in the valley.
The five OpenCV threshold types
The formulas use the threshold $T$ (thresh), the value $M$ (maxval, usually 255) and the source pixel $\text{src}(x,y)$:
| Type | Rule for pixels above T | Rule for the rest | Formula |
|---|---|---|---|
| THRESH_BINARY | set to maxval (255) | 0 | $dst = M$ if $src \gt T$, else $0$ |
| THRESH_BINARY_INV | set to 0 | 255 | $dst = 0$ if $src \gt T$, else $M$ |
| THRESH_TRUNC | set to T (clipped) | unchanged | $dst = T$ if $src \gt T$, else $src$ |
| THRESH_TOZERO | unchanged | 0 | $dst = src$ if $src \gt T$, else $0$ |
| THRESH_TOZERO_INV | set to 0 | unchanged | $dst = 0$ if $src \gt T$, else $src$ |
The slide demo uses a left-to-right gradient image. BINARY gives black on the left and white on the right. BINARY_INV gives the opposite. TRUNC keeps the dark ramp, then a flat cap at $T$. TOZERO sets the dark half to black and keeps the bright ramp. TOZERO_INV keeps the dark ramp and sets the bright half to black. Only BINARY and BINARY_INV give a true two-level image. The other three keep grey values.
Worked example. One row of pixels, $T = 127$, maxval $= 255$. The test is strictly "greater than", so 127 is "not above".
| 30 | 100 | 127 | 128 | 200 | 255 |
| Type | 30 | 100 | 127 | 128 | 200 | 255 |
|---|---|---|---|---|---|---|
| BINARY | 0 | 0 | 0 | 255 | 255 | 255 |
| BINARY_INV | 255 | 255 | 255 | 0 | 0 | 0 |
| TRUNC | 30 | 100 | 127 | 127 | 127 | 127 |
| TOZERO | 0 | 0 | 0 | 128 | 200 | 255 |
| TOZERO_INV | 30 | 100 | 127 | 0 | 0 | 0 |
img is the grayscale input (read with flag 0). 127 is the threshold. 255 is the maximum value for pixels above the threshold. The last argument is the type. The function returns the threshold used (ret) and the output image.
Adaptive thresholding
Global thresholding fails under uneven lighting. A shadow on one side puts the whole shadowed region below $T$. Adaptive thresholding computes a separate threshold for every pixel from its local neighbourhood. Thus different regions get different thresholds.
The local statistic is the mean, the median or the mid-range:
All three use the neighbourhood around the pixel (e.g. a $3\times3$ or $11\times 11$ window).
Adaptive mean and adaptive Gaussian
$C$ is a constant that you subtract from the local average. Adaptive mean gives all neighbours the same weight. Adaptive Gaussian gives nearby pixels more weight (the weights fall off like a Gaussian bell). A pixel becomes 255 if it is larger than its own $T(x,y)$. Otherwise it becomes 0.
How to pick C
- Small C (2 to 5): the threshold is close to the local mean, so more pixels turn white. This shows a faint foreground but keeps more noise.
- Large C (10 to 15): the test is stricter, so fewer pixels turn white. Use it when the background is noisy or unevenly lit.
- Find C by trial and error for the dataset (documents, medical scans, natural scenes).
Worked example (extra). $3\times3$ window, $C = 5$. The centre pixel is 105. One corner (bottom-right) is a bright 200. All other pixels are 100.
| 100 | 100 | 100 |
| 100 | 105 | 100 |
| 100 | 100 | 200 |
Mean: $(7\cdot100 + 200 + 105)/9 = 1005/9 = 111.67$. $T = 111.67 - 5 = 106.67$. Since $105 \lt 106.67$, the output is 0.
Gaussian: use the weights $\tfrac1{16}\begin{bmatrix}1&2&1\\2&4&2\\1&2&1\end{bmatrix}$. Sum $= (11\cdot100 + 1\cdot200 + 4\cdot105)/16 = 1720/16 = 107.5$. $T = 102.5$. Since $105 \gt 102.5$, the output is 255.
The same pixel gives different answers. The Gaussian version gives less weight to the far bright corner.
| Global | Adaptive |
|---|---|
| One threshold for the whole image | One threshold per pixel from its neighbourhood |
| Needs a bimodal histogram and even lighting | Works with uneven lighting / varying background |
| Fast, single parameter | Slower; parameters are window size and C |
| Fails on shadows and gradients | Can make noisy speckle in flat regions (see the slide's dice image) |
Otsu thresholding
Otsu is an automatic global method. You do not pick $T$. The method tries every possible $T$ and keeps the one that separates the two classes best. It assumes a bimodal histogram and reasonable contrast between background and object.
Split the histogram at every possible cut. A good cut gives two compact groups (low spread inside each group). Otsu picks the cut with the smallest weighted within-class variance. This is the same as the largest spread between the groups.
$t$ = candidate threshold. Class 0 = pixels with intensity $\le t$, class 1 = pixels $\gt t$. $\omega_0,\omega_1$ = class weights = (pixels in the class) / (total pixels). $\sigma_0^2,\sigma_1^2$ = variance of intensities inside each class. Choose $t^* = \arg\min_t \sigma_w^2(t)$.
$$\omega_k = \frac{n_k}{N},\qquad \mu_k = \frac{\sum i\,n_i}{n_k},\qquad \sigma_k^2 = \frac{\sum (i-\mu_k)^2 n_i}{n_k}$$$n_i$ = count of pixels with grey level $i$, $N$ = total pixels, sums run over the grey levels of that class.
Slide example, step by step
The image histogram (20 pixels) is: grey level 0 has 6 pixels, level 1 has 4, level 2 has 2 and level 3 has 8. Try the threshold between 1 and 2.
| level | 0 | 1 | 2 | 3 |
| count | 6 | 4 | 2 | 8 |
Left class (levels 0, 1):
$\omega_0 = \dfrac{6+4}{20} = 0.5$. $\mu_0 = \dfrac{0\cdot6 + 1\cdot4}{10} = 0.4$. $\sigma_0^2 = \dfrac{(0-0.4)^2\cdot 6 + (1-0.4)^2\cdot 4}{10} = \dfrac{0.96 + 1.44}{10} = 0.24$.
Right class (levels 2, 3):
$\omega_1 = \dfrac{2+8}{20} = 0.5$. $\mu_1 = \dfrac{2\cdot2 + 3\cdot8}{10} = 2.8$. $\sigma_1^2 = \dfrac{(2-2.8)^2\cdot 2 + (3-2.8)^2\cdot 8}{10} = \dfrac{1.28+0.32}{10} = 0.16$.
Weighted within-class variance: $\sigma_w^2 = 0.5(0.24) + 0.5(0.16) = 0.2$.
The algorithm repeats this for every threshold and takes the minimum. The other two cuts for the same histogram are:
$t = 0$ (left = {0}, right = {1,2,3}): $\omega_0 = 0.3$, $\sigma_0^2 = 0$. Right has 14 pixels, $\mu_1 = (4+4+24)/14 = 2.286$, $\sigma_1^2 = (4\cdot1.653 + 2\cdot0.082 + 8\cdot0.510)/14 = 0.776$, $\omega_1 = 0.7$. $\sigma_w^2 = 0.7\times0.776 = 0.543$.
$t = 2$ (left = {0,1,2}, right = {3}): left has 12 pixels, $\mu_0 = 8/12 = 0.667$, $\sigma_0^2 = (6\cdot0.444 + 4\cdot0.111 + 2\cdot1.778)/12 = 0.556$, $\omega_0 = 0.6$, right variance 0. $\sigma_w^2 = 0.6\times0.556 = 0.333$.
Comparison: $t=0 \to 0.543$, $t=1 \to 0.200$, $t=2 \to 0.333$. The minimum is at $t = 1$, so the Otsu threshold is 1. Pixels $\le 1$ are class 0 and pixels $\ge 2$ are class 1.
Shortcut that examiners accept: the total variance is fixed, $\sigma_T^2 = \sigma_w^2 + \sigma_b^2$ with $\sigma_b^2 = \omega_0\omega_1(\mu_0-\mu_1)^2$. Here $\sigma_b^2 = 0.25\times(2.4)^2 = 1.44$ and $0.2 + 1.44 = 1.64 = \sigma_T^2$ (check: global mean 1.6). So the minimum of the within-class variance is the same as the maximum of the between-class variance. The between-class variance is cheaper to compute.
Practice numerical. A 12-pixel image has levels 0, 1, 2, 3 with counts 4, 2, 2, 4. Find Otsu's threshold.
$t=0$: $\omega_0 = 1/3$, $\sigma_0^2=0$; right {1,2,3} counts 2,2,4: $\mu=2.25$, $\sigma^2 = 0.6875$, $\omega_1 = 2/3$, $\sigma_w^2 = 0.458$.
$t=1$: left {0,1} counts 4,2: $\mu_0 = 0.333$, $\sigma_0^2 = 0.222$, $\omega_0 = 0.5$; right {2,3} counts 2,4: $\mu_1 = 2.667$, $\sigma_1^2 = 0.222$, $\omega_1=0.5$. $\sigma_w^2 = 0.222$.
$t=2$: by symmetry $0.458$. Best $t^* = 1$.
Otsu with noise: blur first
Noise makes the histogram one lump. Then Otsu (like global $T=127$) fails and gives a speckled result. The fix from the slide: filter the image with a Gaussian first. The histogram becomes clearly bimodal. Then Otsu separates the object from the background cleanly (the noisy rectangle and the noisy target rings in the slides). In OpenCV, pass cv2.THRESH_OTSU together with THRESH_BINARY. OpenCV ignores the threshold argument.
Otsu fails when the histogram is not bimodal (e.g. a tiny object on a huge background, so one hump is almost invisible). It also fails under uneven illumination. For uneven lighting, use adaptive thresholding.
Morphological image processing
Morphology is a set of image processing operations that work on shape. To find each output pixel, compare the matching input pixel with its neighbours. A small template, the structuring element, does this. The output has the same size as the input.
Morphology can pre-process the input to a segmentation. More often, it post-processes a segmentation. It removes imperfections (specks, small holes) and extracts information about shape and structure. The slide's fingerprint example shows a speckled thresholded print that morphology cleans.
Terminology
- Structuring element (SE): a small matrix or template (any shape: square, cross, line). It moves over every position of the image and is compared with the pixels below it.
- Fit: every pixel of the SE lies on object pixels.
- Hit: at least one SE pixel lies on an object pixel.
- Miss: no SE pixel lies on an object pixel.
Fit implies hit. In the slide's figure, a $2\times2$ SE gives a Fit when it is fully inside the blob. It gives a Hit when it only overlaps the edge of the blob. It gives a Miss when it is fully over the background.
Erosion
Erosion shrinks objects. It removes pixels from their boundaries. Place the SE at each pixel. The output is 1 only if the SE fits entirely inside the object there.
$A$ = binary image (set of foreground pixels), $B$ = structuring element, $B_z$ = $B$ with its centre moved to $z$. Erosion keeps the positions $z$ where $B_z$ is fully inside $A$.
Dilation
Dilation grows objects. It adds pixels to their boundaries. The output is 1 if the SE hits the object (any overlap).
The symbols are the same as above. Dilation keeps the positions where $B_z$ and $A$ share at least one pixel. (For symmetric SEs such as the square or cross, no reflection of $B$ is needed.)
Worked example: erosion and dilation with a $3\times3$ square SE. Image (1 = object), $6\times6$. Pixels outside the image are 0. The image has a small blob with one bump and one isolated noise pixel at the bottom-right.
| 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 1 | 1 | 1 | 0 | 0 |
| 0 | 1 | 1 | 1 | 1 | 0 |
| 0 | 1 | 1 | 1 | 0 | 0 |
| 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 0 | 0 | 0 | 0 | 1 |
Erosion. The output pixel is 1 only if all 9 pixels under the SE are 1. Only the centre of the $3\times3$ solid block, pixel (row 2, col 2), has a full $3\times3$ of ones. Pixel (2,3) fails because (1,4) is 0. The noise pixel (5,5) fails. Result:
| 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 0 | 1 | 0 | 0 | 0 |
| 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 0 | 0 | 0 | 0 | 0 |
Dilation of the original. The output is 1 if any pixel under the SE is 1. Each 1 paints a $3\times3$ patch around itself. Row by row:
- Row 0 sees the ones of row 1 (cols 1 to 3), so cols 0 to 4 become 1.
- Rows 1, 2 and 3 see ones in cols 1 to 4, so all cols 0 to 5 become 1.
- Row 4 sees row 3 (cols 1 to 3) and the noise pixel in row 5 (col 5), so all of row 4 becomes 1.
- Row 5 sees only the noise pixel, so cols 4 and 5 are 1.
| 1 | 1 | 1 | 1 | 1 | 0 |
| 1 | 1 | 1 | 1 | 1 | 1 |
| 1 | 1 | 1 | 1 | 1 | 1 |
| 1 | 1 | 1 | 1 | 1 | 1 |
| 1 | 1 | 1 | 1 | 1 | 1 |
| 0 | 0 | 0 | 0 | 1 | 1 |
Dilation vs erosion
| Dilation | Erosion |
|---|---|
| Increases the size of objects | Decreases the size of objects |
| Expands object boundaries | Shrinks object boundaries |
| Output is 1 on a HIT (any overlap) | Output is 1 on a FIT (full overlap) |
| Edge pixels affected more than interior pixels (per the slide) | Interior pixels affected more than edge pixels (per the slide) |
| Fills small gaps, merges nearby/overlapping objects | Removes small objects, separates touching objects |
| More likely to add noise and false detections | Less likely to add noise and false detections |
Applications of erosion: remove small objects or thin structures, separate overlapping objects, find object boundaries (original minus eroded) and make masks for segmentation. Applications of dilation: fill small holes and gaps, enlarge objects, join broken parts of an object and make masks for segmentation.
Erosion and dilation are duals. Erosion of the object is the same as dilation of the background (with the reflected SE), and the reverse. They are not inverses. Erosion then dilation does not in general restore the original image. Opening uses this fact.
Compound operations: opening and closing
- Opening = erosion first, then dilation: $A\circ B = (A\ominus B)\oplus B$. It removes small bright specks and thin protrusions, breaks narrow bridges and smooths contours. The objects that stay keep their size.
- Closing = dilation first, then erosion: $A\bullet B = (A\oplus B)\ominus B$. It fills small holes and narrow gaps and joins nearby pieces.
| Opening | Closing |
|---|---|
| Erode, then dilate | Dilate, then erode |
| Removes small objects/noise (white specks on black) | Fills small holes and gaps (black specks inside objects) |
| Breaks thin connections between objects | Joins nearby objects, closes cracks |
| Result is a subset of the original | Result is a superset of the original |
Opening on the earlier $6\times6$ image. Erosion gave a single pixel at (2,2). Dilation of that pixel with the $3\times3$ SE paints a $3\times3$ block at rows 1 to 3, cols 1 to 3. The noise pixel and the one-pixel bump at (2,4) are gone. The solid block is restored:
| 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 1 | 1 | 1 | 0 | 0 |
| 0 | 1 | 1 | 1 | 0 | 0 |
| 0 | 1 | 1 | 1 | 0 | 0 |
| 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 0 | 0 | 0 | 0 | 0 |
Closing a hole ($3\times3$ SE, $5\times5$ image). The image is a ring of ones with a one-pixel hole in the centre:
| 0 | 0 | 0 | 0 | 0 |
| 0 | 1 | 1 | 1 | 0 |
| 0 | 1 | 0 | 1 | 0 |
| 0 | 1 | 1 | 1 | 0 |
| 0 | 0 | 0 | 0 | 0 |
Dilate: every pixel has a 1 in its $3\times3$ neighbourhood, so the whole $5\times5$ image becomes 1 (the hole is gone). Erode (outside the image is 0): a pixel stays 1 only if its full $3\times3$ neighbourhood is inside the image and all ones. This is the central $3\times3$ block:
| 0 | 0 | 0 | 0 | 0 |
| 0 | 1 | 1 | 1 | 0 |
| 0 | 1 | 1 | 1 | 0 |
| 0 | 1 | 1 | 1 | 0 |
| 0 | 0 | 0 | 0 | 0 |
The hole is filled and the outer shape is unchanged. (Libraries handle the border in different ways. OpenCV treats the border of erosion as "don't care", so it keeps the outer ring as it is. In both cases, the hole is filled.)
Practice (1D). Row: 0 1 1 0 1 1 1 0, SE = three 1s in a row ($1\times3$, centred). Dilation: each output is 1 if any of left, self or right is 1. This gives 1 1 1 1 1 1 1 1. Erosion: the output is 1 only where left, self and right are all 1. Only index 5 qualifies. This gives 0 0 0 0 0 1 0 0. Opening = dilate the erosion = 0 0 0 0 1 1 1 0 (the short two-pixel piece is deleted and the 3-pixel run stays). Closing = erode the dilation (outside = 0) = 0 1 1 1 1 1 1 0 (the one-pixel gap at index 3 is filled).
Extras that examiners like: boundary extraction $= A - (A\ominus B)$. Opening and closing are idempotent: a second run changes nothing. The typical clean-up pipeline after segmentation is opening (remove specks) then closing (fill holes). The cleaning of a thresholded mask is step 5 of background subtraction (next lecture).
Connected components (extra)
After the binary mask is clean, you usually count and label the separate blobs. Two foreground pixels belong to the same connected component if a path of foreground pixels links them. 4-connectivity counts only the up, down, left and right neighbours. 8-connectivity also counts the diagonals. The choice matters. Two pixels that touch only at a corner are one blob under 8-connectivity and two blobs under 4-connectivity. To label, scan the image and give each new blob a label. Merge the labels when blobs touch.
| 1 | 1 | 0 | 0 | 1 |
| 0 | 1 | 0 | 0 | 1 |
| 0 | 0 | 1 | 0 | 0 |
| 0 | 0 | 0 | 0 | 0 |
The L-shaped group at the top-left (3 pixels) and the vertical pair at the right (2 pixels) are separate under both rules. The pixel at (2,2) touches (1,1) only diagonally: 8-connectivity gives 2 components (the diagonal pixel joins the L, making it 4 pixels), 4-connectivity gives 3 components.
Quick self-test
1. Why does global thresholding fail on unevenly lit images, and what is the fix?
One fixed $T$ cannot suit the bright areas and the shadowed areas. Shadowed object pixels fall below $T$ and are lost. The fix is adaptive thresholding, where $T(x,y)$ = local mean (or Gaussian-weighted mean) $- C$.
2. List the five global threshold types and state what each does to pixels above T.
BINARY: to 255 (else 0). BINARY_INV: to 0 (else 255). TRUNC: to $T$ (else unchanged). TOZERO: unchanged (else 0). TOZERO_INV: to 0 (else unchanged).
3. Apply THRESH_TRUNC and THRESH_TOZERO with T = 100 to the row 40, 100, 101, 180.
TRUNC: 40, 100, 100, 100. TOZERO: 0, 0, 101, 180 (100 is not strictly greater than 100).
4. Histogram: level 0 has 6 pixels, level 1 has 4, level 2 has 2, level 3 has 8. Compute the weighted within-class variance for the split {0,1} | {2,3}.
$\omega_0=\omega_1=0.5$; $\mu_0=0.4$, $\mu_1=2.8$; $\sigma_0^2=0.24$, $\sigma_1^2=0.16$; $\sigma_w^2 = 0.5(0.24)+0.5(0.16)=0.2$. This is the minimum over all splits (the others give 0.543 and 0.333), so Otsu picks it.
5. What does Otsu minimise, and what does it assume?
It minimises the weighted within-class variance $\omega_0\sigma_0^2+\omega_1\sigma_1^2$ (equivalently maximises between-class variance). It assumes a bimodal histogram and good contrast between object and background. For a noisy image, smooth it with a Gaussian first.
6. Define fit, hit and miss for a structuring element. Which one does erosion use, which one dilation?
Fit: all SE pixels lie on object pixels. Hit: at least one does. Miss: none does. Erosion outputs 1 on a fit; dilation outputs 1 on a hit.
7. Erode and dilate the row 0 1 1 1 0 1 0 with a $1\times3$ all-ones SE (outside = 0).
Erosion: 0 0 1 0 0 0 0 (only index 2 and both of its neighbours are 1). Dilation: 1 1 1 1 1 1 1.
8. Differentiate opening and closing. Which would you use to remove salt noise from a mask, and which to fill holes in an object?
Opening = erosion then dilation; closing = dilation then erosion. Use opening to remove small white specks (salt noise) and thin spurs. Use closing to fill small holes and narrow gaps.
9. Why are erosion and dilation not inverse operations?
Erosion deletes objects smaller than the SE for good. Dilation afterwards cannot bring them back. So $(A\ominus B)\oplus B \subseteq A$ in general (that is opening), not equal to $A$.
10. Adaptive mean threshold: window mean is 82, C = 6, pixel value is 78. What is the output?
$T = 82 - 6 = 76$. $78 \gt 76$ so output is 255 (white).
- A binary image has only foreground and background. Thresholding creates it: it splits the pixels by intensity.
- Global thresholding uses one fixed $T$ (needs bimodal histogram). There are five OpenCV types: BINARY, BINARY_INV, TRUNC, TOZERO, TOZERO_INV. Only the first two give a pure binary output.
- Adaptive thresholding uses a per-pixel $T$ = local mean (or Gaussian-weighted sum) $- C$. It handles uneven lighting. A small $C$ keeps more white pixels (and noise). A large $C$ is stricter.
- Otsu tries all thresholds and picks $t^*$ with the minimum $\sigma_w^2=\omega_0\sigma_0^2+\omega_1\sigma_1^2$. Smooth noisy images first. The slide example gives $\sigma_w^2=0.2$.
- Morphology is shape-based processing with a structuring element. The terms are fit, hit and miss.
- Erosion (output 1 on FIT) shrinks objects and removes small objects. Dilation (output 1 on HIT) grows objects and fills gaps.
- Opening = erode then dilate (remove specks). Closing = dilate then erode (fill holes).
- Connected components label the blobs. 4- or 8-connectivity changes the count.