Chapter 11 β Video Understanding
βA video is a 5D tensor β and the extra dimension of time is where all the interesting semantics live.β
Video as a Tensor
Planned content: Video tensor:
[B, T, C, H, W]or[B, C, T, H, W]. Storage formats (mp4, avi, frames on disk). Temporal sampling strategies (uniform, dense, stride). Frame rate and duration. Memory constraints: a 10-second 1080p clip at 30fps is ~1.5GB raw.
π Planned diagram: Video tensor decomposition β temporal slices, showing
[T, C, H, W]structure with time as the 4th axis.
Action Recognition
Planned content: Classify a video clip into one of K action classes. Trimmed vs. untrimmed video. Two-stream networks: RGB stream + optical flow stream β late fusion. 3D CNNs (C3D, I3D): convolutions across time and space. SlowFast: slow pathway (low frame rate, full spatial) + fast pathway (high frame rate, small spatial).
π Planned diagram: Two-stream network architecture β RGB stream and flow stream with late fusion.
π Planned diagram: SlowFast network β slow pathway
[B, C, T/8, H, W]and fast pathway[B, C/8, T, H, W], lateral connections.
Temporal Modeling
Planned content: 3D convolutions: extend spatial kernel to 3D
[t, h, w]. Pseudo-3D (P3D): factorize into 2D spatial + 1D temporal. TSM (Temporal Shift Module): shift feature maps along time axis with zero extra parameters. Transformers for video: TimeSformer (divided space-time attention), Video Swin Transformer.
π Planned diagram: TSM β feature shift illustration showing how temporal information is exchanged between frames at zero computational cost.
Temporal Action Localization
Planned content: Given an untrimmed video, predict when (start/end time) and what (class) each action occurs. Two-stage (propose + classify) vs. one-stage. BSN, BMN, AFSD. Temporal IoU (tIoU). ActivityNet benchmark.
Video Object Segmentation (VOS)
Planned content: Semi-supervised VOS: given first-frame mask, propagate to all frames. Unsupervised VOS: find salient moving objects. DAVIS benchmark. Space-time memory networks. SAM 2 for video.
Key Datasets
Planned content: Kinetics-400/600/700, Something-Something v2 (temporal reasoning), UCF-101, HMDB-51, AVA (spatio-temporal action detection), ActivityNet (temporal localization), DAVIS (VOS).
Next: Chapter 12 β 3D Vision β
Last updated: May 2026