← Back to Table of Contents

Chapter 9 β€” Depth Estimation

β€œDepth perception bridges the gap between 2D images and the 3D world β€” the foundation for spatial understanding.”

The Depth Estimation Problem

Planned content: Output a dense depth map [B, 1, H, W] where each pixel value represents metric distance (meters) or relative disparity. Supervised (metric depth) vs. self-supervised (relative depth) approaches. Monocular vs. stereo vs. multi-view vs. sensor-guided (LiDAR).

πŸ“Š Planned diagram: Depth estimation output β€” RGB image alongside depth map rendered as heatmap, with near (warm) and far (cool) color coding.

Stereo Vision and Epipolar Geometry

Planned content: Two cameras separated by baseline. Epipolar constraint: a 3D point lies on a known epipolar line in the other image. Disparity d = f * B / Z (f=focal length, B=baseline, Z=depth). Disparity maps from stereo matching. SGM (Semi-Global Matching). PSMNet, GwcNet.

πŸ“Š Planned diagram: Stereo geometry β€” two cameras, epipolar line, disparity vs. depth relationship. Formula labeled diagram.

\[Z = \frac{f \cdot B}{d}, \quad d = x_L - x_R\]

Monocular Depth Estimation

Planned content: Ill-posed without additional cues. Supervised: use LiDAR ground truth during training. Self-supervised: pose network + photometric consistency loss (Monodepth2). Scale ambiguity in self-supervised methods. Foundation models for depth: DPT, Depth Anything, Marigold, UniDepth.

πŸ“Š Planned diagram: Monocular depth cues β€” size, occlusion, atmospheric haze, texture gradient, linear perspective β€” how models learn to exploit them.

πŸ“Š Planned diagram: Monodepth2 self-supervised training β€” source frame β†’ depth net β†’ depth + pose net β†’ warp β†’ photometric loss.

Depth Completion

Planned content: Sparse LiDAR points + RGB image β†’ dense depth map. Sensor fusion problem. Guided upsampling. CSPN, DySPN. KITTI depth completion benchmark.

LiDAR and Sensor Fusion

Planned content: LiDAR gives sparse but highly accurate depth. Camera gives dense but unscaled color. Fusion: project LiDAR onto image plane, use as guide. Early fusion vs. late fusion vs. feature fusion. Surround-view setups (autonomous driving).

πŸ“Š Planned diagram: LiDAR-camera projection β€” 3D point cloud overlaid on image, showing sparse guidance points.

Evaluation Metrics

Planned content: AbsRel (absolute relative error). SqRel (squared relative error). RMSE (root mean squared error). RMSE_log. Threshold metrics Ξ΄1, Ξ΄2, Ξ΄3 (fraction of pixels where max(d_pred/d_gt, d_gt/d_pred) < 1.25^n). SILog (scale-invariant log error). When each metric matters.

πŸ“Š Planned table: Depth metrics summary β€” formula, interpretation, range, what failure modes each catches.

Next: Chapter 10 β€” Optical Flow β†’


Last updated: May 2026