← Back to Table of Contents

Chapter 42 β€” The Frontier

β€œThe frontier is where assumptions break down, benchmarks saturate, and the most interesting questions live.”

World Models and Video Prediction

Planned content: World models: internal representations that can simulate future states. Genie (DeepMind): interactive world from video. JEPA (Joint Embedding Predictive Architecture) β€” Yann LeCun’s framework. Latent world models for RL (DreamerV3). The case against pixel-level prediction. RSSM (Recurrent State Space Model). Open challenge: building accurate, efficient world models.

πŸ“Š Planned diagram: World model loop β€” current state β†’ predict next state in latent space β†’ compare with actual next state (JEPA architecture).

3D Gaussian Splatting at Scale

Planned content: 3DGS limitations: memory (millions of Gaussians), dynamic scenes, in-the-wild capture. Mip-Splatting: alias-free rendering. 4DGS for dynamic scenes. COLMAP-free training. Street-level and city-scale 3DGS. Foundation model for 3DGS (MASt3R, DUSt3R for pose-free reconstruction).

Embodied AI and Robotics Vision

Planned content: Vision in the context of action: perceive β†’ plan β†’ act. RT-2 (Robotics Transformer 2): VLM as robot policy. OpenVLA: open-source robot policy. Affordance detection. Grasp prediction. Navigation (VLN: Vision-Language Navigation). Active perception. PointNet++ for robot manipulation. The sim-to-real gap. Data collection at scale (DROID, Open-X-Embodiment).

πŸ“Š Planned diagram: Embodied AI pipeline β€” RGB camera β†’ VLM β†’ language instruction + visual context β†’ robot policy β†’ action tokens β†’ robot arm.

Neural Rendering and Novel View Synthesis

Planned content: NeRF β†’ Instant-NGP β†’ TensoRF β†’ 3DGS evolution. Generalizable NeRFs (pixelNeRF, IBRNet): predict from a single image. Zero-shot 3D from 2D: Zero123, Zero123-XL. Feed-forward 3D (Large Reconstruction Model). The future: any image β†’ 3D object in seconds.

Neuromorphic Vision

Planned content: Event cameras (DVS): microsecond latency, high dynamic range, no motion blur. Events as sparse spikes (x, y, t, polarity). Processing event streams: SNN (Spiking Neural Networks), point cloud methods, time-surface methods. Applications: high-speed robotics, autonomous driving. Prophesee Metavision cameras.

οΏ½οΏ½ Planned diagram: Event camera output vs. frame camera β€” traditional frame (30fps) vs. event stream showing per-pixel microsecond events for fast motion.

Scaling Laws for Vision

Planned content: Do vision models follow scaling laws? Evidence from ViT scaling (Zhai et al.). FLOP-optimal training for vision. Emergent capabilities at scale in VLMs. When does scale help vs. architectural innovations?

Interpretability in Vision Models

Planned content: GradCAM and its variants (GradCAM++, EigenCAM, ScoreCAM). Attention map visualization. DINO self-attention as segmentation. Probing classifiers: what does each layer encode? Mechanistic interpretability for vision. Superposition in CNNs. Circuit analysis.

πŸ“Š Planned diagram: GradCAM visualization β€” gradient-weighted activation maps highlighting discriminative regions for classification.

Open Problems and Research Directions

Planned content: Long-tail and out-of-distribution generalization. Compositional generalization (understanding β€œa red cube on a blue sphere” never seen in training). Continual learning without catastrophic forgetting. Few-shot and zero-shot learning for dense prediction. Privacy-preserving vision (federated learning, differential privacy). Robustness to adversarial attacks. AI safety in vision systems. Closing the gap to human visual intelligence.

πŸ“Š Planned diagram: Open challenges map β€” current SOTA performance vs. human performance vs. 2030 projection for each major task.

← Back to Table of Contents


Last updated: May 2026