Chapter 10 β Optical Flow
βOptical flow is the apparent motion of brightness patterns β the visual velocity field of the world.β
The Optical Flow Problem
Planned content: Dense motion estimation: for every pixel (x, y) at time t, estimate displacement (u, v) to time t+1. Output:
[B, 2, H, W]β two channels for horizontal and vertical displacement. Applications: video compression, action recognition, video stabilization, autonomous driving.
π Planned diagram: Optical flow visualization β arrow field overlaid on a video frame, color-wheel encoding of magnitude and direction.
Classical Methods
Planned content: Brightness constancy assumption. Horn-Schunck: global regularization (smooth flow field). Lucas-Kanade: local patch-based solution, pyramidal extension for large motions. FlowFields, EpicFlow. Limitations: occlusion, large displacements, texture-less regions.
π Planned diagram: Lucas-Kanade optical flow β patch around a pixel, gradient matrix, iterative refinement.
Deep Learning Approaches
Planned content: FlowNet: first CNN for optical flow β correlation layer for matching. FlowNet2. SPyNet: spatial pyramid + warping. PWC-Net: pyramid + warping + cost volume. RAFT: recurrent all-pairs field transforms β the current state-of-the-art approach.
π Planned diagram: RAFT architecture β feature encoder, context encoder, all-pairs correlation volume
[B, H*W, H*W], GRU update operator for iterative refinement.
Evaluation: Endpoint Error (EPE)
Planned content: Average endpoint error (AEE/EPE) = mean Euclidean distance between predicted and ground truth flow vectors. Fl-all: fraction of pixels with EPE > 3px AND > 5% of GT magnitude. Sintel and KITTI benchmarks.
Applications
Planned content: Video stabilization (inverse flow warping). Video super-resolution. Action recognition (two-stream networks with flow). Motion segmentation. Event cameras and flow.
Next: Chapter 11 β Video Understanding β
Last updated: May 2026