← Back to Table of Contents

Chapter 12 β€” 3D Vision

β€œThe world is 3D. Computer vision that ignores this is perpetually incomplete.”

3D Representations

Planned content: Comparison of 3D representations: point clouds (unstructured, [N, 3+]), voxel grids ([X, Y, Z], memory-expensive), meshes (vertices + faces), implicit surfaces (occupancy networks, SDF), NeRF (neural radiance field), 3D Gaussian Splatting. When to use each.

πŸ“Š Planned diagram: Six 3D representations visualized side-by-side for the same 3D object (Stanford bunny) β€” point cloud, voxel grid, mesh, SDF slice, NeRF renders, 3DGS render.

Point Cloud Processing

Planned content: LiDAR point clouds: 360Β° scan, [N, 4] (x, y, z, intensity). Irregular, unordered. PointNet: per-point MLP + global max pooling β€” permutation invariant. PointNet++: hierarchical local grouping (ball query). Set abstraction layers.

πŸ“Š Planned diagram: PointNet architecture β€” input [B, N, 3] β†’ per-point MLP β†’ max pooling β†’ global feature [B, 1024] β†’ classification head.

3D Object Detection

Planned content: VoxelNet: voxelize point cloud β†’ 3D convolutions β†’ RPN. PointPillars: vertical pillars instead of voxels, faster. CenterPoint: center-based detection from BEV feature map. SECOND: sparse 3D convolutions. Evaluation: 3D IoU, BEV IoU, KITTI, Waymo Open Dataset.

πŸ“Š Planned diagram: PointPillars pipeline β€” scatter points into vertical pillars β†’ PointNet per pillar β†’ 2D BEV feature map β†’ 2D detection head.

Neural Radiance Fields (NeRF)

Planned content: Represent a scene as an implicit function: MLP maps (x, y, z, ΞΈ, Ο†) β†’ (RGB, Οƒ). Volume rendering: march rays, sample along ray, composite colors and densities. Training: minimize photometric loss against multi-view images. Limitations: slow training, slow rendering, requires many views. Improvements: Instant-NGP (hash encoding), TensoRF, Mip-NeRF.

πŸ“Š Planned diagram: NeRF rendering pipeline β€” ray casting, sampling N points per ray [N, 5] β†’ MLP β†’ density + color β†’ volume rendering integral β†’ pixel color.

πŸ“Š Planned diagram: Positional encoding β€” why raw coordinates fail and how sinusoidal encoding helps the MLP fit high-frequency details.

\[C(r) = \int_{t_n}^{t_f} T(t)\,\sigma(r(t))\,c(r(t),d)\,dt, \quad T(t) = \exp\!\left(-\int_{t_n}^{t}\sigma(r(s))ds\right)\]

3D Gaussian Splatting (3DGS)

Planned content: Represent scene as a set of 3D Gaussians with position, covariance, opacity, and SH color coefficients. Tile-based rasterizer: sort Gaussians by depth, project to 2D, alpha composite. Real-time rendering. Training from multi-view images with SfM initialization. Memory vs. NeRF.

πŸ“Š Planned diagram: 3DGS pipeline β€” SfM point cloud β†’ 3D Gaussians initialization β†’ differentiable rasterization β†’ photometric loss β†’ densification/pruning.

SLAM (Simultaneous Localization and Mapping)

Planned content: Estimate camera pose while building a map. Visual SLAM: camera only (ORB-SLAM). Visual-inertial SLAM. Dense vs. sparse maps. Neural SLAM (iMAP, NICE-SLAM, MonoGS). Frontend (odometry) + backend (loop closure, bundle adjustment).

πŸ“Š Planned diagram: SLAM system architecture β€” camera stream β†’ feature tracking β†’ pose estimation β†’ map update β†’ loop closure β†’ global optimization.

Next: Chapter 13 β€” Anomaly Detection β†’


Last updated: May 2026