Stereo vision basics

The vision unit measures distance the way your two eyes do: it compares what two cameras side by side see. This guide explains the principle and the words used everywhere else in this section, with diagrams and one worked example. The stages that implement it are in Stereo stages.

The worked example is two Raspberry Pi HQ Cameras with 8 mm lenses, mounted 8 cm apart, delivering 480 × 270 images. The same formulas apply to your own cameras with their own numbers.

#One camera: the lens and the focal length

A camera turns a point in the world into a pixel. The lens acts like a pinhole: light from a point goes straight through the lens centre to the sensor. The sensor sits behind the lens and gets an upside-down image; the diagrams here draw the image plane in front of the lens instead, at the same distance f, which gives the same geometry without the flip.

                   image plane
                        |                            * P
                        |                       .    |
                        |                  .         |
                        |             .              |  X  (P's offset
                        |        .                   |      from the axis)
                        * x .                        |
                   .    |                            |
     lens centre  o-----+----------------------------+----  optical axis
                  |< f >|                            |
                  |<------------------ Z ----------->|

The two triangles have the same shape, so:

x = f × X / Z

Focal length f is the distance between the lens and the sensor. A lens is sold by its focal length in millimetres; stereo works with it in pixels: millimetres divided by the size of one pixel.

Three things follow:

• The field of view comes from the lens and the sensor, not the image size. A longer lens sees a narrower cone, with more pixels per degree.
• f in pixels scales with the image width. Halve the image and f halves too: each pixel then covers twice the angle.
• Real lenses bend straight lines, mostly near the edges (distortion). Calibration measures that bending so it can be undone.

#Two cameras: the baseline

Put two identical cameras side by side, facing the same way. The distance between their lens centres is the baseline B: 8 cm in the example.

  seen from above
                                  * P
                                 /|\
                                / | \
                               /  |  \
                              /   |   \
                             /    |    \
    image planes  ----------*-----|-----*----------
    at distance f          / xL   |  xR  \
                          /       Z       \
                         /        |        \
               left lens o--------+--------o right lens
                         |<------ B ------>|

Each camera sees P at a different horizontal position in its image, measured from that image's centre: xL in the left one, xR in the right one. Their difference is the disparity:

d = xL − xR                    (in pixels)

From the triangles of both cameras:

Z = f × B / d

That one formula is stereo vision. The product f × B characterises a stereo pair; in the example it is 610.8 px × 0.08 m = 48.9 px·m.

Near objects shift a lot between the two images, far objects hardly at all. A point infinitely far away has a disparity of zero.

#Range: the nearest point

The matcher looks for each pixel's partner over a fixed number of positions, num_disparities. A shift larger than that cannot be found, so the nearest measurable distance is:

Z_min = f × B / num_disparities

At 64, an object at 0.5 m (97.7 px of disparity) is not measured, or worse, matched to a wrong position. More disparities cost proportionally more computing. There is also a blind band: the leftmost num_disparities columns of the left image have no partner in the right one, so they get no depth.

#Precision: why far is hard

Matching finds a disparity to a fraction of a pixel, never exactly. One pixel of disparity error moves the depth by:

ΔZ ≈ Z² / (f × B)              per pixel of disparity error

The error grows with the square of the distance. Matching refines disparities to a fraction of a pixel, so real errors are smaller, but the shape stays: a stereo pair is precise near and coarse far.

#Choosing lens and baseline

Range and precision both come from f × B:

For a vehicle that must stop for obstacles between 0.5 and 3 m, a short f × B and enough disparities matter more than far precision; for mapping farther away, the opposite.

#Rectification: making the search one row

To find P's partner, the matcher would have to search the whole right image. Two things make it a search along one row instead.

Epipolar geometry. When the cameras are perfectly side by side and parallel, a point lies on the same row in both images. Real cameras are never mounted that precisely, and their lenses bend the image.

Rectification fixes both. From the calibration, it computes for each camera an image transform that undoes the lens distortion and rotates the image as if the two cameras were perfectly parallel. After it, a point lies on the same row in both images:

  before rectification: P sits on different rows

     left image                 right image
     +------------------+       +------------------+
     |        P         |       |                  |
     |                  |       |     P            |
     +------------------+       +------------------+

  after rectification: P sits on the same row in both

     left image                 right image
     +------------------+       +------------------+
     |        P         |       |     P            |
     |                  |       |                  |
     +------------------+       +------------------+
              <---- search along this row only ---->

The dashboard's Row alignment figure checks this on every pair: if a camera or a lens moves after calibration, points drift off their rows and depth degrades.

#Calibration

Calibration measures each camera and their relation, usually by showing them a checkerboard of known size from many angles. It produces, per camera, in a camera_info YAML:

A calibration is valid for one lens setting: refocusing a lens changes its focal length and needs a new calibration. Changing the image size while keeping its aspect ratio does not: the pipeline scales the calibration. Changing the aspect ratio, for example from a 4:3 to a 16:9 sensor mode, needs a new one.

#Disparity map, depth image, point cloud

Matching every pixel gives three views of the same measurement:

  rectified pair ──► disparity map ──► depth image ──► point cloud
                     d per pixel       Z per pixel     (X, Y, Z) per pixel
                     (pixels)          (millimetres)   (metres)

Disparity map. An image whose pixels are disparities, in pixels: large values are near. Pixels without a reliable match are marked unmatched. It is what the matcher computes.

Depth image. The same image with each disparity turned into a distance, Z = f × B / d. The unit publishes it in millimetres as 16-bit integers, 0 meaning "no measurement": the ROS convention, which consumers such as costmaps read directly.

Point cloud. Each depth pixel (u, v) with depth Z becomes a 3D point, reversing the camera projection:

X = (u − cx) × Z / f           sideways, positive to the right
Y = (v − cy) × Z / f           vertical, positive downwards
Z                              forward, along the optical axis

The axes are the camera's optical frame: x right, y down, z forward, in metres, from the left camera's lens. ROS places this frame on the vehicle with a transform from the camera's mounting position. The unit's cloud is organized: it keeps the image's grid, one point every few pixels (decimation), with NaN where nothing was measured, so a consumer knows which pixel each point came from.

#What stereo needs from the scene

Stereo matches texture: small patterns of light and dark that can be told apart. Some scenes defeat it:

The median that ends the matching and the speckle and temporal filters after it remove isolated wrong pixels; Stereo stages describes each one.

#Words used in this section

#See also

• Stereo stages: how the unit computes each step.
• Vision performance: the cost of resolution and disparities.
• Vision unit

This page is docs/vision/basics.md in the ovcs repository; edit it there.