Chapter 9 β Depth Estimation
βDepth perception bridges the gap between 2D images and the 3D world β the foundation for spatial understanding.β
The Depth Estimation Problem
Planned content: Output a dense depth map
[B, 1, H, W]where each pixel value represents metric distance (meters) or relative disparity. Supervised (metric depth) vs. self-supervised (relative depth) approaches. Monocular vs. stereo vs. multi-view vs. sensor-guided (LiDAR).
π Planned diagram: Depth estimation output β RGB image alongside depth map rendered as heatmap, with near (warm) and far (cool) color coding.
Stereo Vision and Epipolar Geometry
Planned content: Two cameras separated by baseline. Epipolar constraint: a 3D point lies on a known epipolar line in the other image. Disparity
d = f * B / Z(f=focal length, B=baseline, Z=depth). Disparity maps from stereo matching. SGM (Semi-Global Matching). PSMNet, GwcNet.
\[Z = \frac{f \cdot B}{d}, \quad d = x_L - x_R\]π Planned diagram: Stereo geometry β two cameras, epipolar line, disparity vs. depth relationship. Formula labeled diagram.
Monocular Depth Estimation
Planned content: Ill-posed without additional cues. Supervised: use LiDAR ground truth during training. Self-supervised: pose network + photometric consistency loss (Monodepth2). Scale ambiguity in self-supervised methods. Foundation models for depth: DPT, Depth Anything, Marigold, UniDepth.
π Planned diagram: Monocular depth cues β size, occlusion, atmospheric haze, texture gradient, linear perspective β how models learn to exploit them.
π Planned diagram: Monodepth2 self-supervised training β source frame β depth net β depth + pose net β warp β photometric loss.
Depth Completion
Planned content: Sparse LiDAR points + RGB image β dense depth map. Sensor fusion problem. Guided upsampling. CSPN, DySPN. KITTI depth completion benchmark.
LiDAR and Sensor Fusion
Planned content: LiDAR gives sparse but highly accurate depth. Camera gives dense but unscaled color. Fusion: project LiDAR onto image plane, use as guide. Early fusion vs. late fusion vs. feature fusion. Surround-view setups (autonomous driving).
π Planned diagram: LiDAR-camera projection β 3D point cloud overlaid on image, showing sparse guidance points.
Evaluation Metrics
Planned content: AbsRel (absolute relative error). SqRel (squared relative error). RMSE (root mean squared error). RMSE_log. Threshold metrics Ξ΄1, Ξ΄2, Ξ΄3 (fraction of pixels where max(d_pred/d_gt, d_gt/d_pred) < 1.25^n). SILog (scale-invariant log error). When each metric matters.
π Planned table: Depth metrics summary β formula, interpretation, range, what failure modes each catches.
Next: Chapter 10 β Optical Flow β
Last updated: May 2026