← Back to Table of Contents

Chapter 24 β€” Vision Transformers

β€œThe transformer’s success in language asks a natural question: can attention replace convolution? The answer β€” yes, given enough data β€” reshaped computer vision.”

Adapting Transformers to Images

Planned content: Why naive application of self-attention to images fails: for a 224Γ—224 image, N = 50,176 tokens β†’ O(NΒ²) attention is 2.5B operations. The patch trick: divide image into non-overlapping patches, treat each as a token. Reduces N to 196 for 16Γ—16 patches.

πŸ“Š Planned diagram: Image β†’ patch grid decomposition β€” [3, 224, 224] divided into 196 patches of [3, 16, 16] each, flattened and projected to [196, 768].

ViT Deep Dive

Planned content: Patch embedding (linear projection of flattened patches). CLS token. Learnable position embeddings (1D, how they work better than 2D in practice). Multi-head self-attention across all patch tokens. MLP block. Layer normalization (pre-norm for stability). Classification via CLS token. ViT-S/B/L/H variants.

πŸ“Š Planned diagram: ViT block β€” LayerNorm β†’ MHSA β†’ residual β†’ LayerNorm β†’ MLP β†’ residual. Tensor shapes: [B, N+1, D] throughout (N=196 patches + 1 CLS token).

πŸ“Š Planned diagram: Attention maps in ViT β€” visualizing attention from CLS token to patches, showing semantic segmentation-like behavior in late layers.

1
2
3
4
5
# ViT attention visualization
model = timm.create_model('vit_base_patch16_224', pretrained=True)
# Last layer attention: [B, num_heads, N+1, N+1]
# Attention from CLS to patches: [B, num_heads, 1, N]
# Average over heads β†’ [B, N] β†’ reshape β†’ [B, 14, 14]

DeiT: Training ViTs Efficiently (2020)

Planned content: Data-Efficient Image Transformer: train on ImageNet-1K only (no JFT). Knowledge distillation from a CNN teacher via a special distillation token. Hard distillation label = teacher’s argmax. DeiT-S and DeiT-B competitive with ResNet. Key training tricks: strong augmentation (RandAugment, CutMix, Mixup), stochastic depth.

πŸ“Š Planned diagram: DeiT β€” CLS token + distillation token + N patch tokens, showing separate classification and distillation prediction heads.

Swin Transformer Deep Dive (2021)

Planned content: Four-stage hierarchical architecture. Stage 1: [B, H/4, W/4, 96] (patch partition + linear embed). Stages 2-4: patch merging halves resolution, doubles channels. Window multi-head self-attention (W-MSA): each window 7Γ—7 patches, attention within window only. Shifted window (SW-MSA): cyclic shift + masking for cross-window communication in odd layers. Relative position bias.

πŸ“Š Planned diagram: Swin window partition and shift β€” regular windows (W-MSA) vs. shifted windows (SW-MSA), showing cyclic shift and masking for valid attention.

πŸ“Š Planned diagram: Swin feature map sizes β€” [H/4, W/4] β†’ [H/8, W/8] β†’ [H/16, W/16] β†’ [H/32, W/32] like a 4-stage CNN.

MAE: Masked Autoencoding (2021)

Planned content: Self-supervised pre-training: mask 75% of patches, reconstruct masked pixels with decoder. Sparse ViT encoder (only 25% visible tokens). Lightweight decoder. Pre-training is fast. Fine-tuning reaches ViT performance with far less labeled data. Why high masking ratio works (images are redundant, unlike text).

πŸ“Š Planned diagram: MAE pipeline β€” mask 75% of patches β†’ encoder on visible 25% [B, 0.25*N, D] β†’ add mask tokens β†’ decoder β†’ reconstruct masked pixels β†’ MSE loss.

DINO: Self-Distillation with No Labels (2021)

Planned content: Self-supervised training via self-distillation. Student network: sees distorted crops. Teacher network: EMA of student. Loss: cross-entropy between student and teacher output (soft labels over prototypes). No labels. Centering to prevent collapse. Local-to-global correspondence.

πŸ“Š Planned diagram: DINO training β€” student (random augment) vs. teacher (large crop EMA) with centering and sharpening.

DINOv2: Better Self-Supervised Features (2023)

Planned content: Large-scale curated data (LVD-142M). DINO + iBOT (masked image modeling) combined. Registers for global/background understanding. Strong zero-shot features: k-NN classification, segmentation, depth, without task-specific fine-tuning. The β€œvisual GPT-3” analogy.

πŸ“Š Planned diagram: DINOv2 feature quality β€” PCA of patch features showing object part segmentation emerging from self-supervised training alone.

EVA, InternImage, and Beyond

Planned content: EVA-CLIP: large-scale CLIP pre-training for ViT-18B. InternImage: deformable convolution as a vision foundation model. Scaling laws for vision transformers. SoViT. FlexiViT. NaViT (variable resolution). PatchDropout.

Next: Chapter 25 β€” Multimodal Architectures β†’


Last updated: May 2026