Chapter 12 β 3D Vision
βThe world is 3D. Computer vision that ignores this is perpetually incomplete.β
3D Representations
Planned content: Comparison of 3D representations: point clouds (unstructured,
[N, 3+]), voxel grids ([X, Y, Z], memory-expensive), meshes (vertices + faces), implicit surfaces (occupancy networks, SDF), NeRF (neural radiance field), 3D Gaussian Splatting. When to use each.
π Planned diagram: Six 3D representations visualized side-by-side for the same 3D object (Stanford bunny) β point cloud, voxel grid, mesh, SDF slice, NeRF renders, 3DGS render.
Point Cloud Processing
Planned content: LiDAR point clouds: 360Β° scan,
[N, 4](x, y, z, intensity). Irregular, unordered. PointNet: per-point MLP + global max pooling β permutation invariant. PointNet++: hierarchical local grouping (ball query). Set abstraction layers.
π Planned diagram: PointNet architecture β input
[B, N, 3]β per-point MLP β max pooling β global feature[B, 1024]β classification head.
3D Object Detection
Planned content: VoxelNet: voxelize point cloud β 3D convolutions β RPN. PointPillars: vertical pillars instead of voxels, faster. CenterPoint: center-based detection from BEV feature map. SECOND: sparse 3D convolutions. Evaluation: 3D IoU, BEV IoU, KITTI, Waymo Open Dataset.
π Planned diagram: PointPillars pipeline β scatter points into vertical pillars β PointNet per pillar β 2D BEV feature map β 2D detection head.
Neural Radiance Fields (NeRF)
Planned content: Represent a scene as an implicit function: MLP maps
(x, y, z, ΞΈ, Ο) β (RGB, Ο). Volume rendering: march rays, sample along ray, composite colors and densities. Training: minimize photometric loss against multi-view images. Limitations: slow training, slow rendering, requires many views. Improvements: Instant-NGP (hash encoding), TensoRF, Mip-NeRF.
π Planned diagram: NeRF rendering pipeline β ray casting, sampling
Npoints per ray[N, 5]β MLP β density + color β volume rendering integral β pixel color.
\[C(r) = \int_{t_n}^{t_f} T(t)\,\sigma(r(t))\,c(r(t),d)\,dt, \quad T(t) = \exp\!\left(-\int_{t_n}^{t}\sigma(r(s))ds\right)\]π Planned diagram: Positional encoding β why raw coordinates fail and how sinusoidal encoding helps the MLP fit high-frequency details.
3D Gaussian Splatting (3DGS)
Planned content: Represent scene as a set of 3D Gaussians with position, covariance, opacity, and SH color coefficients. Tile-based rasterizer: sort Gaussians by depth, project to 2D, alpha composite. Real-time rendering. Training from multi-view images with SfM initialization. Memory vs. NeRF.
π Planned diagram: 3DGS pipeline β SfM point cloud β 3D Gaussians initialization β differentiable rasterization β photometric loss β densification/pruning.
SLAM (Simultaneous Localization and Mapping)
Planned content: Estimate camera pose while building a map. Visual SLAM: camera only (ORB-SLAM). Visual-inertial SLAM. Dense vs. sparse maps. Neural SLAM (iMAP, NICE-SLAM, MonoGS). Frontend (odometry) + backend (loop closure, bundle adjustment).
π Planned diagram: SLAM system architecture β camera stream β feature tracking β pose estimation β map update β loop closure β global optimization.
Next: Chapter 13 β Anomaly Detection β
Last updated: May 2026