Epipolar geometry & stereo
One image does not give depth, but two images do. This page explains why the match of a point must lie on a line (epipolar geometry). It shows how the matrices E and F encode that line and how to estimate F. It also shows how to turn matched pixels into depth with $Z = fB/d$.
Why one view is not enough: back-projection
A pixel $\mathbf{x}$ in an image gives only the direction of the scene point, not its distance. Every 3D point on the ray through the optic centre and the pixel lands on that same pixel. So an image point back-projects to a line (ray) in the scene, and an image line back-projects to a plane. A geometric entity always back-projects to an entity of higher dimension.
$\mathbf{X}$ is any scene point on the ray, $\mathbf{x}$ the image point (in camera coordinates), $\alpha$ the unknown depth scale.
A single view cannot solve geometric problems, so we need several views:
| Shape from stereo | Structure from motion | |
|---|---|---|
| Cameras | different cameras | same (moving) camera |
| Viewpoints | different | different |
| Time | same time | different times |
Two views: three sub-problems
A scene point is seen by cameras $C$ (left) and $C'$ (right). Back-project the two rays and intersect them. This needs three solutions:
- Epipolar geometry: find how the two views are geometrically related.
- Correspondence problem: find the right-image pixel that shows the same scene point as a given left pixel.
- Triangulation: intersect the rays to get the 3D point.
Warm-up: two identical parallel cameras
Take two cameras with the same focal length $f$. They are side by side, separated by $t_x$ along the x-axis. The same scene point has the same $Y, Z$ in both camera frames but a shifted $X$:
The reciprocal depth $1/Z$ is proportional to the horizontal disparity $(x'-x)$. Only the x-coordinate changes. The y-coordinate stays the same.
As $Z$ goes from $0$ to $\infty$, the shift $f t_x/Z$ goes from $\infty$ to $0$. So the possible match of $\mathbf{x}$ traces a straight line in the other image. The search for the match is therefore a 1D search, not a 2D search. This is the main benefit of epipolar geometry.
Hold a finger in front of your face and blink the left and right eye in turn. A near finger jumps a lot. The far wall hardly moves. The jump is the disparity: near means large disparity, far means small disparity.
Epipolar geometry: definitions
In the general case, the cameras are at arbitrary positions. The possible matches of $\mathbf{x}$ lie on the projection into $C'$ of the back-projected ray from $C$. A projected ray is always a straight line. It is the epipolar line $\mathbf{l}'$. When $\mathbf{X}$ slides along the ray, the second ray sweeps out a plane, the epipolar plane.
| Term | Meaning |
|---|---|
| $P$ | scene point |
| $p, p'$ | its images in the two views |
| $O, O'$ | optical centres (camera centres) |
| Baseline $OO'$ | line joining the two optical centres |
| $\Pi, \Pi'$ | image planes (retinas) |
| Epipolar plane | plane through $O$, $O'$ and $P$ |
| Epipole $e, e'$ | where the baseline meets each image plane; equivalently the image of the other camera's centre |
| Epipolar line $l, l'$ | intersection of the epipolar plane with an image plane |
Key facts to state in an exam:
- When $P$ moves, the whole epipolar plane moves with it and gives a new epipolar line. All epipolar planes hinge about the baseline (a "pencil of planes").
- So all epipolar lines in an image pass through the epipole of that image.
- All points on one epipolar line in $C$ share the same corresponding epipolar line in $C'$.
- The match of $p$ must lie on $l'$. This is the epipolar constraint. It reduces the correspondence search from 2D to 1D.
Special cases examiners like. (1) Parallel image planes (cameras side by side): the baseline is parallel to the image planes. The epipoles are at infinity and the epipolar lines are parallel to the $u$ axis (horizontal scanlines). (2) Converging cameras: the epipolar lines are slanted and all meet at the epipole. (3) One camera moves forward along its optical axis: the epipole is inside the image.
Preamble: homogeneous coordinates
The slides list three tools. We need them before we derive E and F:
- Points at infinity: homogeneous coordinates represent them. $(x, y, 0)^T$ is a point at infinity. This is how we represent epipoles at infinity (parallel cameras).
- Lines: a line $ax + by + c = 0$ is the vector $\mathbf{l} = (a,b,c)^T$. A point lies on it iff $\mathbf{l}^T\mathbf{x} = 0$.
- Cross product as a matrix: $\mathbf{a}\times\mathbf{b} = [\mathbf{a}]_\times \mathbf{b}$, where $[\mathbf{a}]_\times$ is skew-symmetric. The line through two points is $\mathbf{l}=\mathbf{x}_1\times\mathbf{x}_2$. The intersection of two lines is $\mathbf{x}=\mathbf{l}_1\times\mathbf{l}_2$.
Skew-symmetric means $[\mathbf{t}]_\times^T = -[\mathbf{t}]_\times$. The matrix has rank 2 and $[\mathbf{t}]_\times\mathbf{t}=\mathbf{0}$.
Essential matrix E
Assume the cameras are calibrated: we know the intrinsic matrices $K_L, K_R$. Then we can convert pixels into normalized camera coordinates, which are coordinates on an ideal image plane at $f=1$. In these coordinates, the matrix $E$ encodes only the relative pose (rotation and translation) of the two cameras. It gives the epipolar line for any point.
$x_L, x_R$: homogeneous normalized coordinates of the same scene point $P$ in the left and right images. $R$: rotation, $\mathbf{t}$: translation from the left to the right camera frame. $E$ is $3\times3$.
Why it holds (short derivation). In the right frame, $P_R = R P_L + \mathbf{t}$. The vectors $\mathbf{t}$, $P_R$ and $R P_L$ are coplanar (this is the epipolar plane). So $P_R\cdot(\mathbf{t}\times R P_L)=0$. Divide by the depths to get $x_R^T[\mathbf{t}]_\times R\,x_L=0$.
Properties of E (from the slides):
- Size $3\times3$, rank 2.
- 5 degrees of freedom: 3 for rotation + 2 for the direction of translation (the scale is lost).
- Two equal non-zero singular values and one zero: $\sigma_1=\sigma_2,\ \sigma_3=0$.
Worked example (E from R and t). Let $R$ be a $90^\circ$ rotation about $z$ and $\mathbf{t}=(0,0,1)^T$.
Step 1: $R=\begin{pmatrix}0&-1&0\\1&0&0\\0&0&1\end{pmatrix}$, $[\mathbf{t}]_\times=\begin{pmatrix}0&-1&0\\1&0&0\\0&0&0\end{pmatrix}$ (using $t_z=1$, others 0).
Step 2: $E=[\mathbf{t}]_\times R$. Row 1 of $[\mathbf t]_\times$ is $(0,-1,0)$, so row 1 of $E = -(\text{row 2 of }R) = (-1,0,0)$. Row 2 is $(1,0,0)$, so row 2 of $E$ = row 1 of $R$ = $(0,-1,0)$. Row 3 is zero.
| -1 | 0 | 0 |
| 0 | -1 | 0 |
| 0 | 0 | 0 |
Step 3: checks. The rank is 2. The singular values are $1,1,0$ (equal, as the property says).
Step 4: epipolar line for $x_L=(1,2,1)^T$: $\mathbf{l}_R = E x_L = (-1,-2,0)^T$, i.e. $-x-2y=0$. The point $x_R=(2,-1,1)^T$ gives $-2+2=0$, so it satisfies $x_R^TEx_L=0$.
Fundamental matrix F
Real images are in pixels, and often we do not know the intrinsics. $F$ is the same epipolar constraint, but on the pixel coordinates $p_L, p_R$. It needs no prior calibration.
Substitute $x_L = K_L^{-1}p_L$ and $x_R = K_R^{-1}p_R$ into $x_R^TEx_L=0$:
$K_L,K_R$: intrinsic matrices; $p_L,p_R$: homogeneous pixel coordinates $(u,v,1)^T$.
Properties of F:
- $3\times3$, rank 2, so $\det F = 0$.
- 7 degrees of freedom ($9$ entries $-1$ for overall scale $-1$ for $\det F=0$).
- It maps a point to an epipolar line: $\mathbf{l}_R = F p_L$ (line in the right image) and $\mathbf{l}_L = F^T p_R$ (line in the left image).
- Epipoles: $F e_L = 0$ and $F^T e_R = 0$. The epipoles are the null vectors of $F$ and $F^T$. All epipolar lines pass through the epipole, so $e_R^T\,F p_L = 0$ for every $p_L$.
| Property | Essential matrix E | Fundamental matrix F |
|---|---|---|
| Coordinates | normalized camera coordinates | pixel coordinates |
| Needs intrinsics? | yes (calibrated cameras) | no (uncalibrated) |
| Formula | $E=[\mathbf t]_\times R$ | $F=K_R^{-T}EK_L^{-1}$ |
| Constraint | $x_R^TEx_L=0$ | $p_R^TFp_L=0$ |
| Rank | 2 | 2 ($\det F=0$) |
| Degrees of freedom | 5 (3 R + 2 t-direction) | 7 |
| Singular values | $\sigma_1=\sigma_2,\ \sigma_3=0$ | only $\sigma_3=0$ |
| Contains | only relative pose | pose + both intrinsics |
Worked example (epipolar line from F). Let $F=\begin{pmatrix}0&-1&2\\1&0&-3\\-2&3&0\end{pmatrix}$ (skew-symmetric, so rank 2) and the left pixel $p_L=(1,1,1)^T$.
Step 1: $\mathbf{l}_R = Fp_L$. Row 1: $0-1+2=1$. Row 2: $1+0-3=-2$. Row 3: $-2+3+0=1$. So $\mathbf{l}_R=(1,-2,1)^T$, the line $x-2y+1=0$.
Step 2: test a candidate $p_R=(5,3,1)^T$: $5-6+1=0$, so it is on the line (valid match). Test $p_R=(6,4,1)^T$: $6-8+1=-1\neq0$, so reject it. Its distance from the line is $|{-1}|/\sqrt{1^2+2^2}=1/\sqrt5\approx0.45$ px.
Step 3: epipole of the right image. Solve $F e=0$: $-e_2+2e_3=0$, $e_1-3e_3=0$ gives $e=(3,2,1)^T$. Check row 3: $-6+6=0$. The line $\mathbf l_R$ passes through it: $3-4+1=0$. Every epipolar line must do this.
Worked example (parallel cameras). Pure translation along $x$, $R=I$, $K=I$, $\mathbf t=(1,0,0)^T$. Then $E=F=[\mathbf t]_\times=\begin{pmatrix}0&0&0\\0&0&-1\\0&1&0\end{pmatrix}$. For $p_L=(3,2,1)^T$: $\mathbf l_R=Fp_L=(0,-1,2)^T$, i.e. $-y+2=0$, so $y'=2$. The match lies in the same row. This is the rectified-stereo case. Epipole: $Fe=0\Rightarrow e=(1,0,0)^T$, a point at infinity along $x$.
Practice: pixel to normalized coordinates. $K=\begin{pmatrix}800&0&320\\0&800&240\\0&0&1\end{pmatrix}$, $p_L=(480,300,1)^T$. Then $x_L=K^{-1}p_L=\big(\tfrac{480-320}{800},\tfrac{300-240}{800},1\big)=(0.2,\ 0.075,\ 1)^T$. (Subtract the principal point, then divide by the focal length.)
Do not mix the direction of the maps. $Fp_L$ is a line in the right image. $F^Tp_R$ is a line in the left image. Also, $F$ constrains a point to a line. It does not give one single match point.
Computing F: eight-point algorithm
Every match $(x,y)\leftrightarrow(x',y')$ gives one linear equation in the 9 entries of $F$. Collect enough matches, stack the equations and solve them.
$(x,y)$: point in image 1, $(x',y')$: its match in image 2. One row of $A$ per correspondence.
$F$ is defined only up to scale, so there are 8 unknowns. Hence we need at least 8 correspondences.
Steps
- Collect $\ge 8$ matching points.
- Build the $n\times9$ matrix $A$.
- Solve $A\mathbf f=\mathbf 0$ with SVD. The vector $\mathbf f$ is the right singular vector for the smallest singular value (it minimises $\|A\mathbf f\|$ with $\|\mathbf f\|=1$).
- Reshape $\mathbf f$ into $F$ ($3\times3$).
- Enforce rank 2: do an SVD again, $F=U\Sigma V^T$. Set the smallest singular value to zero, $\Sigma=\mathrm{diag}(\sigma_1,\sigma_2,0)$, and rebuild $F=U\Sigma' V^T$.
Step 5 is necessary. Noise makes the solution of step 3 full rank (rank 3), but a true $F$ has rank 2. Without step 5, the epipolar lines do not meet at one epipole.
Plain 8-point: it is simple and fast. It is sensitive to noise and has poor numerical stability without normalization. Pixel coordinates like 1000 give entries of size $10^6$ next to entries of size 1.
Normalized eight-point algorithm
The fix is to normalize the coordinates first. The centroid goes to the origin and the average distance from the origin becomes $\sqrt2$.
- Normalize the points in both images with $T$ and $T'$ (translate to the centroid, then scale).
- Run the eight-point algorithm to get $F_n$ (include the rank-2 enforcement).
- Denormalize: $F = T'^{T} F_n T$.
Worked example (building T). Points $(100,200),(300,200),(100,400),(300,400)$.
Step 1: centroid $=(200,300)$. Centred points: $(\mp100,\mp100)$, i.e. $(-100,-100),(100,-100),(-100,100),(100,100)$.
Step 2: each point is at distance $\sqrt{100^2+100^2}=141.4$ from the origin, so the mean distance is $141.4$.
Step 3: scale $s=\sqrt2/141.4=0.01$. So $T=\begin{pmatrix}0.01&0&-2\\0&0.01&-3\\0&0&1\end{pmatrix}$ (apply $s(x-200)$, $s(y-300)$). The new points are $(\pm1,\pm1)$, each at distance $\sqrt2$.
The normalized algorithm is the most common linear estimator. It is much more stable and it is the standard in practice.
Seven-point algorithm
This algorithm uses exactly 7 correspondences. Seven equations leave a two-dimensional null space of $A\mathbf f=0$, spanned by $F_1, F_2$. So $F=\alpha F_1+(1-\alpha)F_2$. Impose $\det F=0$. This gives a cubic $a\alpha^3+b\alpha^2+c\alpha+d=0$ in $\alpha$, with up to three real solutions.
| 8-point | 7-point | |
|---|---|---|
| Correspondences | $\ge 8$ | exactly 7 |
| Method | linear SVD + rank-2 projection | 2D null space + cubic from $\det=0$ |
| Solutions | one | 1 to 3 |
| Pros / cons | simple, fast; noise-sensitive unless normalized | minimal, good inside RANSAC; multiple solutions, more complex |
Quick counting: unknowns in $F$ = 9, minus scale = 8 (8-point). The rank-2 constraint removes one more = 7 DOF (7-point). For $E$: 3 + 3 - 1 scale = 5 DOF (5-point algorithm, not in slides).
Practice (one row of A). Match $(x,y)=(2,3)\leftrightarrow(x',y')=(4,5)$. Row of $A$: $[x'x,\ x'y,\ x',\ y'x,\ y'y,\ y',\ x,\ y,\ 1]=[8,\ 12,\ 4,\ 10,\ 15,\ 5,\ 2,\ 3,\ 1]$.
Stereopsis
Stereopsis is how a visual system (human or computer) perceives depth from two slightly different views of the same scene. It has two components:
- Binocular fusion: combine the left and right images into a single perception.
- Reconstruction: estimate the 3D structure of the scene from the two images.
Binocular fusion
The eyes are about 6 to 7 cm apart, so each eye sees the scene from a slightly different viewpoint. The brain receives both images, aligns the corresponding points and merges them into one image.
- Motor fusion: the eyes align (vergence) so that the object falls on corresponding retinal locations.
- Sensory fusion: the brain combines the two retinal images into one visual perception.
- Corresponding retinal points: points in the two eyes that show the same spatial location. They have disparity $=0$.
Binocular disparity
$x_L, x_R$: coordinate of the same object in the left and right image. Large disparity means the object is near. Small disparity means it is far.
Correspondence: finding the match
Correlation (window-based) matching
Assume that a small neighbourhood around a left pixel looks like the neighbourhood of the true match. The match is in the right image, along the epipolar scanline. Slide a window across the candidate disparities and pick the best score.
$W$: window around $(x,y)$; $d$: candidate horizontal disparity; $I_L,I_R$: left/right intensities. Choose the $d$ with the smallest cost. SAD is fast but sensitive to gain/bias (brightness differences). SSD penalises large differences more.
Worked example (SAD/SSD, 3x3 window). Left window:
| 10 | 12 | 14 |
| 11 | 13 | 15 |
| 12 | 14 | 16 |
Right windows at three candidate disparities:
| 10 | 12 | 15 |
| 11 | 14 | 15 |
| 12 | 14 | 18 |
($d=0$)
| 12 | 14 | 20 |
| 13 | 15 | 22 |
| 14 | 16 | 25 |
($d=1$)
| 10 | 11 | 14 |
| 11 | 12 | 15 |
| 12 | 13 | 16 |
($d=2$)
$d=0$: absolute differences $0,0,1,0,1,0,0,0,2$, SAD $=4$, SSD $=0+0+1+0+1+0+0+0+4=6$.
$d=1$: differences $2,2,6,2,2,7,2,2,9$, SAD $=34$.
$d=2$: differences $0,1,0,0,1,0,0,1,0$, SAD $=3$, SSD $=3$.
The smallest cost is at $d=2$, so disparity $=2$ px. With $f=500$ px and $B=0.1$ m: $Z=fB/d=500\times0.1/2=25$ m.
Multi-scale edge matching (feature-based)
This method does not match every pixel. It matches prominent structures (edges, zero-crossings, Canny boundaries) across scale space.
- Scale-space decomposition: use Gaussian/Laplacian pyramids or filter banks such as the Marr-Hildreth filter $\nabla^2G$ (Laplacian of Gaussian).
- Coarse-level matching: at coarse scales the noise is smoothed out. Large disparities are matched reliably and the search does not get trapped in local minima.
- Fine-level propagation: the coarse disparity gives a narrow search window $[d_{\text{coarse}}-\Delta,\ d_{\text{coarse}}+\Delta]$ at the next finer level.
- Edge localization: at the finest scale, the edge positions give sub-pixel precision.
Coarse-to-fine is the standard answer to "how do you handle large disparities cheaply and robustly?".
Stereo reconstruction pipeline
Reconstruction recovers the 3D coordinates of scene points from a stereo pair. The pipeline has these stages:
Camera calibration then image rectification then disparity estimation then 3D reprojection then surface mesh / volumetric model.
Step 1: camera calibration
- Intrinsics: focal lengths $(f_x,f_y)$, principal point $(c_x,c_y)$, lens radial/tangential distortion coefficients. Find them with a calibration target (e.g. checkerboard).
- Extrinsics: rotation $R$ and translation $\mathbf t$ between the left and right camera centres $C_L, C_R$.
- Fundamental matrix: $F=K_R^{-T}[\mathbf t]_\times RK_L^{-1}$, which enforces $p_R^TFp_L=0$.
Step 2: stereo rectification
Warp both images so that conjugate epipolar lines become horizontal and aligned. This removes the vertical disparity.
- Homography warping: compute $H_L,H_R$ that rotate the image planes to be coplanar and parallel to the baseline $C_LC_R$.
- Scanline realignment: resample the images so that corresponding points lie on identical rows, $y_L=y_R$.
- Search reduction: the matching drops from a 2D search to a 1D search along row $y$.
Step 3: matching cost computation
Measure the dissimilarity between the left pixel $(x,y)$ and the right pixel $(x-d,y)$ for $d\in[0,d_{\max}]$.
- Census transform: encode each pixel as a binary string of comparisons with its neighbours. Compare candidates by Hamming distance (robust to brightness changes).
- SAD: $C(x,y,d)=\sum_{W}|I_L(x+u,y+v)-I_R(x+u-d,y+v)|$.
- Deep features: CNN backbones build a multi-dimensional matching cost volume.
Step 4: triangulation and reprojection
Take a rectified pair with baseline $B$ (distance between the camera centres) and focal length $f$ (in pixels). The geometry is two similar triangles:
$d$: disparity (pixels); $B$: baseline; $f$: focal length (pixels); $(c_x,c_y)$: principal point; $c_x'$: principal point of the right camera; $Q$: the $4\times4$ reprojection matrix (as in OpenCV). Depth is inversely proportional to disparity. In OpenCV the translation $T_x$ is signed (negative for the right camera). This makes $W$ positive. With magnitudes, $W=d/B$ when $c_x=c_x'$.
Worked example (disparity to depth). $f=700$ px, $B=0.12$ m, left pixel $x=400$, right pixel $x_R=386$, $y=260$, principal point $(320,240)$.
Step 1: $d=x_L-x_R=400-386=14$ px.
Step 2: $Z=fB/d=700\times0.12/14=6$ m.
Step 3: $X=(400-320)\times6/700=0.686$ m, $Y=(260-240)\times6/700=0.171$ m. The point is $(0.69,\ 0.17,\ 6)$ m.
Step 4 (via $Q$): $X'=400-320=80$, $Y'=260-240=20$, $Z'=f=700$, $W=d/B=14/0.12=116.67$. Divide: $X=80/116.67=0.686$, $Y=0.171$, $Z=700/116.67=6$ m. The answer is the same.
Sensitivity: if $d$ changes to $70$ px, the point is at $Z=1.2$ m (close). If $d=1$ px, then $Z=84$ m. The depth resolution is poor for far points. A 1 px error in $d$ is a huge relative error when $d$ is small.
Effects of the baseline: a larger $B$ gives larger disparities (better depth accuracy), but it also gives more occlusion and a harder correspondence problem. A smaller $B$ is easier to match, but the depth is noisier. The same formula also gives the range: the nearest measurable depth corresponds to $d_{\max}$.
Practice. A stereo rig has $f=1000$ px, $B=20$ cm. (a) A point has disparity 25 px: $Z=1000\times0.2/25=8$ m. (b) Which disparity corresponds to $Z=2$ m? $d=fB/Z=1000\times0.2/2=100$ px. (c) The disparity estimate at 8 m is off by 1 px (24 px): $Z=8.33$ m, an error of 0.33 m.
Quick self-test
1. Define epipole, epipolar plane and epipolar line. Why does the epipolar constraint reduce the search to 1D?
Epipole: where the baseline (line through the two camera centres) meets an image plane. It is the image of the other camera's centre. Epipolar plane: plane through the two centres and the scene point. Epipolar line: intersection of the epipolar plane with an image plane. The match of a point in image 1 must lie on the corresponding epipolar line in image 2. So we search along a line, not the whole image.
2. State the differences between the essential and fundamental matrices.
$E=[\mathbf t]_\times R$ works in normalized camera coordinates and needs calibrated cameras. It has 5 DOF, rank 2 and equal non-zero singular values. $F=K_R^{-T}EK_L^{-1}$ works in pixel coordinates and needs no calibration. It has 7 DOF and rank 2. Constraints: $x_R^TEx_L=0$ and $p_R^TFp_L=0$.
3. Given $F=\begin{pmatrix}0&-1&2\\1&0&-3\\-2&3&0\end{pmatrix}$ and $p_L=(2,1,1)^T$, find the epipolar line in the right image and check whether $(4,1,1)$ lies on it.
$\mathbf l_R=Fp_L$: row 1 $0\cdot2-1\cdot1+2\cdot1=1$; row 2 $2+0-3=-1$; row 3 $-4+3+0=-1$. Line $(1,-1,-1)$: $x-y-1=0$. Test $(4,1,1)$: $4-1-1=2\neq0$, so the point is not on the line. (e.g. $(3,2,1)$ gives $3-2-1=0$, which is on it.)
4. Why do E and F have rank 2? How is it enforced when estimating F?
$[\mathbf t]_\times$ has rank 2 (it annihilates $\mathbf t$), so $E$ and hence $F$ are singular. This shows that all epipolar lines pass through one epipole (the null vector of $F$). First solve $A\mathbf f=0$ by SVD. Then do an SVD of the resulting $F=U\Sigma V^T$, set the smallest singular value to 0 and rebuild the matrix.
5. Outline the normalized eight-point algorithm. Why normalize?
Translate each point set so that its centroid is at the origin. Scale it so that the mean distance is $\sqrt2$ ($T,T'$). Build $A$ from the normalized points. Solve $A\mathbf f=0$ by SVD. Enforce rank 2. Denormalize: $F=T'^TF_nT$. Normalization fixes the poor conditioning of $A$ (entries of very different magnitude). The result is then much less sensitive to noise.
6. How many correspondences are needed by the 8-point and 7-point methods, and why does the 7-point method give up to 3 solutions?
The methods need 8 and 7 correspondences. With 7 points the null space of $A$ is 2D, so $F=\alpha F_1+(1-\alpha)F_2$. The condition $\det F=0$ is a cubic in $\alpha$, which has 1 or 3 real roots.
7. A stereo pair has $f=800$ px, baseline 10 cm, disparity 16 px. Find the depth, and the depth if disparity were 8 px.
$Z=fB/d=800\times0.1/16=5$ m. For $d=8$: $Z=10$ m. Half the disparity gives double the depth. The relation is inverse.
8. Compute SAD for left window $[3,5;4,6]$ and right window $[2,5;6,6]$.
Differences $|3-2|,|5-5|,|4-6|,|6-6| = 1,0,2,0$. SAD $=3$. SSD $=1+0+4+0=5$.
9. Why rectify the images before stereo matching?
Rectification warps both images (homographies $H_L,H_R$) so that the epipolar lines become horizontal rows with $y_L=y_R$. Then the match search runs along one row only (1D). The vertical disparity vanishes and $Z=fB/d$ applies directly.
10. What is coarse-to-fine multi-scale matching and its benefit?
First match edges on blurred coarse images, where large disparities are found robustly. Then use $d_{\text{coarse}}\pm\Delta$ as a narrow search window at the finer levels. At the finest scale, localize the edges with sub-pixel precision. Benefits: it avoids local minima and saves computation.
- A pixel back-projects to a ray; two views intersect two rays (triangulation). Sub-problems: epipolar geometry, correspondence, triangulation.
- Epipolar plane = two centres + scene point. Epipolar lines = plane $\cap$ image plane. Epipoles = baseline $\cap$ image planes.
- All epipolar lines meet at the epipole. Parallel cameras: epipoles at infinity, horizontal epipolar lines.
- Parallel cameras: $1/Z=(x'-x)/(ft_x)$, so $Z=fB/d$, near means large disparity.
- $E=[\mathbf t]_\times R$, $x_R^TEx_L=0$, normalized coordinates, rank 2, 5 DOF.
- $F=K_R^{-T}EK_L^{-1}$, $p_R^TFp_L=0$, pixel coordinates, rank 2, 7 DOF; $\mathbf l_R=Fp_L$, $\mathbf l_L=F^Tp_R$, $Fe_L=0$, $F^Te_R=0$.
- 8-point: $A\mathbf f=0$ by SVD, then force rank 2 by zeroing the smallest singular value.
- Normalized 8-point: centroid 0, mean distance $\sqrt2$, $F=T'^TF_nT$.
- 7-point: 2D null space plus a cubic from $\det F=0$ (up to 3 solutions), used in RANSAC.
- Stereopsis = binocular fusion + reconstruction. Disparity $d=x_L-x_R$.
- Matching: SAD/SSD windows, census + Hamming, multi-scale edge matching (coarse to fine), deep cost volumes.
- Pipeline: calibration, rectification (homographies $H_L,H_R$, $y_L=y_R$), matching cost, triangulation with $Z=fB/d$, $X=(x-c_x)Z/f_x$, $Y=(y-c_y)Z/f_y$ or the $Q$ matrix.