Where depth is computed

The stereo stages are written once and run one of two ways, depending on the board:

Both give the same results: the native stages are tested against the Nx ones, image for image.

To see where each stage of a running unit is computed, select it on the dashboard: its panel is tagged with where it runs (CPU or GPU), what it is written in (Rust, Elixir or Elixir Nx) and, for Nx, what compiles it (Vulkan, LLVM, EXLA or the Nx evaluator). GET /api/stages returns the same as each stage's compute attribute.

#Nx: the stages as tensor programs

Nx is Elixir's library for numerical computing: a function written with defn works on whole tensors, and a compiler turns it into code for a CPU or a GPU. Each stage implements VisionCore.Stereo.NxStage, whose step is such a function. The image algorithms come from nx_stereo:

The Pipeline joins consecutive Nx stages into one VisionCore.Stereo.NxSegment, compiled once when the pipeline starts. A pair then goes to the device once, every stage of the segment runs there, and the outputs come back once. A value that changes while running, such as a calibration or a live setting like p1, is an input of the compiled function; a build setting such as num_disparities is compiled in.

The compiler comes from the :defn_options of the :stereo component, which go to VisionCore.Stereo.NxSegment, the second of the component's default backends after VisionCore.Stereo.NativeSegment. A component can list its own :backends instead, each {module, opts}, with defn_options: among NxSegment's options:

nx_iree compiles the segment with IREE into a module that the IREE runtime on the board loads and runs. A board has neither EXLA nor the IREE compiler, so the modules are compiled on your workstation into a cache directory in your vehicle's priv/. Run this in your vehicle's project, on the host, where vision_firmware brings EXLA:

cd vehicles/<vehicle>
mix vision_core.nx_cache --composer <Vehicle>.Vision.Composer --out priv/nx_iree_cache

It compiles what the composer's stereo pipeline compiles when it starts. The cache keys each module on the stages, their build settings and the working size, so rebuild it after changing any of them; a unit whose cache lacks a module fails to start its stereo pipeline. A new calibration needs no rebuild. List the directory in required_files/0 so a build without it fails.

#On the C100's GPU

The Jetson Nano module of the C100 has 4 Cortex-A57 cores and a 128-core Maxwell GPU, about 236 GFLOPS in 32-bit floats, sharing 4 GB of LPDDR4 memory.

 v4l2_capture ──grey pair──► BEAM ──upload──► GPU: every stage as compute shaders ──download──► BEAM

1. v4l2_capture writes each grey pair into the BEAM through its pipe.
2. The segment's inputs are copied into GPU buffers. CPU and GPU share the same memory chips, so this is a copy within memory, not across a bus.
3. Every stage runs as Vulkan compute shaders, with libSGM's algorithm for the matching. The temporal filter's previous disparity stays on the GPU from one pair to the next.
4. The outputs are copied back once, and the stages after the segment run on the BEAM.

While one pair's outputs are collected, the next pair's GPU work already runs. The CPU's part is capture, the copies and publishing; the GPU does the arithmetic. The dashboard's Stereo load is the share of each second the segment takes, waits for the GPU included.

#On the Raspberry Pi 5's CPU

The Pi 5 has 4 Cortex-A76 cores at 2.4 GHz, a VideoCore VII GPU and LPDDR4X memory on a 32-bit bus. The stages run on the CPU cores, in native code:

• Not the GPU. The Raspberry Pi 5 system image has no Vulkan driver, and the VideoCore VII, about 77 GFLOPS, is slower than the four A76 cores together, about 150.
• Not Nx on the CPU. For a CPU target, NxStereo.sgm/3 runs its path aggregation as a loop of small tensor updates, which compiled code runs very slowly. At 480 × 360, the compiled matching took about 60 s a pair on the Pi 5, against 117 ms for the first native version.

#The stereo_match Port

With compute: :native on the :stereo component, every default stage runs in stereo_match, a Rust program the BEAM drives as a Port:

{:stereo,
 [
   compute: :native,
   stages: Pipeline.default_stages(disparity: [num_disparities: 64, paths: 3]),
   # driver, sizes, calibration and cameras
 ]}

The Port keeps each segment's state between pairs: the rectification sampling maps, the matching buffers and the temporal filter's previous disparity. A segment is configured on its first pair, and again after a calibration reload or a restart of the Port, which loses every segment's state. The configuration holds each stage's kind, its build settings and the constants its Nx step shares with the Port, such as the point cloud's depth range, so each is written once, in the stage. The Port refuses a configuration it cannot run, such as a kind it has no implementation of or a setting the stage does not take: the pipeline then stops at that first pair, and the log says why. Frames smaller than the working size are refused before, when the pipeline starts, by Rectification. The protocol has three requests, configure, run and release, each one length-prefixed record over the pipes; vision/rust/stereo_match/src/protocol.rs defines it.

The stages keep their names, settings and outputs, so the exporters, the dashboard and detection work the same. A stage can override the component's choice with compute: :native or compute: :nx among its options: Pipeline.default_stages(disparity: [compute: :native]) matches natively in a pipeline otherwise compiled with Nx, and compute: :nx compiles a stage in a native one. Disparity's matcher: option does the same for the matching, and must agree with a compute: given beside it. A stage asked to compute natively that no backend computes natively is refused at start.

Under compute: :native, a stage computes natively when it declares the kind of its native implementation with native_kind/0: every default stage does. A stage of your own that wraps a default one declares that stage's kind, and for TemporalFilter, PointCloud, RowAlignment and SpeckleFilter its native_constants/0 too, or the pipeline stops at its first pair; it cannot add a numeric build setting of its own, which the Port would refuse. A stage of your own without a native kind runs between the native segments around it, which hand it their outputs: an Nx stage becomes a compiled segment of its own, so the component also needs :defn_options, rpi5_defn_options/1 on the Pi 5, and a module cache. Without a compiler, the pipeline refuses to start rather than run that stage on Nx's evaluator. A stage that is not written in Nx runs on the BEAM. At start, the Pipeline logs each run of stages with where it computes, for example rectification+sharpness (cpu, nx, llvm_cpu) → disparity (cpu, rust).

#Threads

 reader thread    reads and parses the next pair from stdin
 compute pool     one thread per core: the stages of the current pair
 writer thread    writes the previous pair's outputs to stdout

The Pipeline keeps two pairs at the Port, so reading, computing and writing overlap (How a pair flows). The compute threads are started once and kept, blocked between pairs. Inside a stage, work is handed out in small items, usually image rows, as they become ready, rather than split into fixed shares; a thread waiting for a row another thread is finishing spins briefly, then yields its core, then sleeps. On a board where the BEAM, the camera Ports and jpeg_codec share the same four cores, a thread that loses its core then delays only the row it holds, not the whole stage. config :vision_core, :stereo_match_threads sets the pool size, one thread per core by default.

#Memory and the processor

The matching is the cost. For every pixel it keeps one cost per candidate disparity, and with paths: 8 those volumes do not fit in any cache:

Each A76 core has 512 KB of L2 cache, and the four share 2 MB of L3. With 8 paths, the matcher sweeps the image down and then up, so its volumes go through main memory several times per pair. With 3 paths it makes one pass down the image: the costs of the rows just computed sit in a ring that stays in cache, and each row is freed once the vertical path has gone through it. That is the main reason 3 paths are much faster on the Pi than the path count alone suggests.

The arithmetic is shaped for the A76's 128-bit NEON registers:

• Costs fit in bytes. A cost is at most p2 + 48, below 128, so a register holds 16 of them and two paths add without overflow.
• Census in bit planes. The 48-bit signatures are six byte planes, so a pixel's costs over all its disparities are byte popcounts over contiguous memory.
• Winner search. The search for the best disparity is written with NEON intrinsics.
• Tuned build. stereo_match is built with -C target-cpu=cortex-a76.

Memory bandwidth is still what limits the Pi. The cameras' image signal processor writes every frame to the same memory, so with the cameras running a plain memory copy goes from about 3.2 GB/s to about 1.1 to 1.5 GB/s. Together with the other processes on the cores, this makes the same stereo work take about three times longer while the unit runs than on an idle board.

#What crosses the pipes

At 480 × 270, each pair moves about 1.4 MB through the Port's pipes. Replies are written straight from the computed images, without first copying them into one message.

The depth in metres, which PointCloud uses, never leaves the Port. The Port enlarges its pipes to 4 MiB where the system allows it, so a whole pair goes through without the Port waiting on the BEAM.

#Rust Ports

The core runs its Rust programs as Ports: separate processes speaking length-prefixed binary records over stdin and stdout, not NIFs loaded into the BEAM, so a crash in one cannot take the BEAM down.

Compiling vision_core for a vision target cross-builds that target's Ports through vision/core/Makefile: camera_capture, hailo_detect, jpeg_codec and stereo_match for rpi5, v4l2_capture and jpeg_codec for okdo_c100. On a workstation, mix compile builds jpeg_codec and stereo_match for the host. The build host needs the aarch64-unknown-linux-gnu Rust target, which mise install provides, and libclang for bindgen. Each crate is described in vision/rust/README.md.

#Grey frames on the Pi 5

Give both cameras format: :grey in the :stereo component's :left and :right options. camera_capture then sends the luma plane the image signal processor already produced, one byte per pixel. Without it, each frame is encoded to JPEG in camera_capture and decoded again on the BEAM's side, two pieces of CPU work per frame for nothing. Only the frames that actually leave the unit are encoded: raw images at most :raw_image_max_hz a second, among the ROS 2 exporter's stereo: options, and the dashboard's previews while it is open.

#See also

• Stereo stages
• Vision performance
• vision/core/README.md: how the tests compare the native stages with Nx.

This page is docs/vision/compute.md in the ovcs repository; edit it there.