Chapter 42 β The Frontier
βThe frontier is where assumptions break down, benchmarks saturate, and the most interesting questions live.β
World Models and Video Prediction
Planned content: World models: internal representations that can simulate future states. Genie (DeepMind): interactive world from video. JEPA (Joint Embedding Predictive Architecture) β Yann LeCunβs framework. Latent world models for RL (DreamerV3). The case against pixel-level prediction. RSSM (Recurrent State Space Model). Open challenge: building accurate, efficient world models.
π Planned diagram: World model loop β current state β predict next state in latent space β compare with actual next state (JEPA architecture).
3D Gaussian Splatting at Scale
Planned content: 3DGS limitations: memory (millions of Gaussians), dynamic scenes, in-the-wild capture. Mip-Splatting: alias-free rendering. 4DGS for dynamic scenes. COLMAP-free training. Street-level and city-scale 3DGS. Foundation model for 3DGS (MASt3R, DUSt3R for pose-free reconstruction).
Embodied AI and Robotics Vision
Planned content: Vision in the context of action: perceive β plan β act. RT-2 (Robotics Transformer 2): VLM as robot policy. OpenVLA: open-source robot policy. Affordance detection. Grasp prediction. Navigation (VLN: Vision-Language Navigation). Active perception. PointNet++ for robot manipulation. The sim-to-real gap. Data collection at scale (DROID, Open-X-Embodiment).
π Planned diagram: Embodied AI pipeline β RGB camera β VLM β language instruction + visual context β robot policy β action tokens β robot arm.
Neural Rendering and Novel View Synthesis
Planned content: NeRF β Instant-NGP β TensoRF β 3DGS evolution. Generalizable NeRFs (pixelNeRF, IBRNet): predict from a single image. Zero-shot 3D from 2D: Zero123, Zero123-XL. Feed-forward 3D (Large Reconstruction Model). The future: any image β 3D object in seconds.
Neuromorphic Vision
Planned content: Event cameras (DVS): microsecond latency, high dynamic range, no motion blur. Events as sparse spikes
(x, y, t, polarity). Processing event streams: SNN (Spiking Neural Networks), point cloud methods, time-surface methods. Applications: high-speed robotics, autonomous driving. Prophesee Metavision cameras.
οΏ½οΏ½ Planned diagram: Event camera output vs. frame camera β traditional frame (30fps) vs. event stream showing per-pixel microsecond events for fast motion.
Scaling Laws for Vision
Planned content: Do vision models follow scaling laws? Evidence from ViT scaling (Zhai et al.). FLOP-optimal training for vision. Emergent capabilities at scale in VLMs. When does scale help vs. architectural innovations?
Interpretability in Vision Models
Planned content: GradCAM and its variants (GradCAM++, EigenCAM, ScoreCAM). Attention map visualization. DINO self-attention as segmentation. Probing classifiers: what does each layer encode? Mechanistic interpretability for vision. Superposition in CNNs. Circuit analysis.
π Planned diagram: GradCAM visualization β gradient-weighted activation maps highlighting discriminative regions for classification.
Open Problems and Research Directions
Planned content: Long-tail and out-of-distribution generalization. Compositional generalization (understanding βa red cube on a blue sphereβ never seen in training). Continual learning without catastrophic forgetting. Few-shot and zero-shot learning for dense prediction. Privacy-preserving vision (federated learning, differential privacy). Robustness to adversarial attacks. AI safety in vision systems. Closing the gap to human visual intelligence.
π Planned diagram: Open challenges map β current SOTA performance vs. human performance vs. 2030 projection for each major task.
Last updated: May 2026