Chapter 24 β Vision Transformers
βThe transformerβs success in language asks a natural question: can attention replace convolution? The answer β yes, given enough data β reshaped computer vision.β
Adapting Transformers to Images
Planned content: Why naive application of self-attention to images fails: for a 224Γ224 image, N = 50,176 tokens β O(NΒ²) attention is 2.5B operations. The patch trick: divide image into non-overlapping patches, treat each as a token. Reduces N to 196 for 16Γ16 patches.
π Planned diagram: Image β patch grid decomposition β
[3, 224, 224]divided into 196 patches of[3, 16, 16]each, flattened and projected to[196, 768].
ViT Deep Dive
Planned content: Patch embedding (linear projection of flattened patches). CLS token. Learnable position embeddings (1D, how they work better than 2D in practice). Multi-head self-attention across all patch tokens. MLP block. Layer normalization (pre-norm for stability). Classification via CLS token. ViT-S/B/L/H variants.
π Planned diagram: ViT block β LayerNorm β MHSA β residual β LayerNorm β MLP β residual. Tensor shapes:
[B, N+1, D]throughout (N=196 patches + 1 CLS token).
π Planned diagram: Attention maps in ViT β visualizing attention from CLS token to patches, showing semantic segmentation-like behavior in late layers.
1
2
3
4
5
# ViT attention visualization
model = timm.create_model('vit_base_patch16_224', pretrained=True)
# Last layer attention: [B, num_heads, N+1, N+1]
# Attention from CLS to patches: [B, num_heads, 1, N]
# Average over heads β [B, N] β reshape β [B, 14, 14]
DeiT: Training ViTs Efficiently (2020)
Planned content: Data-Efficient Image Transformer: train on ImageNet-1K only (no JFT). Knowledge distillation from a CNN teacher via a special distillation token. Hard distillation label = teacherβs argmax. DeiT-S and DeiT-B competitive with ResNet. Key training tricks: strong augmentation (RandAugment, CutMix, Mixup), stochastic depth.
π Planned diagram: DeiT β CLS token + distillation token + N patch tokens, showing separate classification and distillation prediction heads.
Swin Transformer Deep Dive (2021)
Planned content: Four-stage hierarchical architecture. Stage 1:
[B, H/4, W/4, 96](patch partition + linear embed). Stages 2-4: patch merging halves resolution, doubles channels. Window multi-head self-attention (W-MSA): each window7Γ7patches, attention within window only. Shifted window (SW-MSA): cyclic shift + masking for cross-window communication in odd layers. Relative position bias.
π Planned diagram: Swin window partition and shift β regular windows (W-MSA) vs. shifted windows (SW-MSA), showing cyclic shift and masking for valid attention.
π Planned diagram: Swin feature map sizes β
[H/4, W/4]β[H/8, W/8]β[H/16, W/16]β[H/32, W/32]like a 4-stage CNN.
MAE: Masked Autoencoding (2021)
Planned content: Self-supervised pre-training: mask 75% of patches, reconstruct masked pixels with decoder. Sparse ViT encoder (only 25% visible tokens). Lightweight decoder. Pre-training is fast. Fine-tuning reaches ViT performance with far less labeled data. Why high masking ratio works (images are redundant, unlike text).
π Planned diagram: MAE pipeline β mask 75% of patches β encoder on visible 25%
[B, 0.25*N, D]β add mask tokens β decoder β reconstruct masked pixels β MSE loss.
DINO: Self-Distillation with No Labels (2021)
Planned content: Self-supervised training via self-distillation. Student network: sees distorted crops. Teacher network: EMA of student. Loss: cross-entropy between student and teacher output (soft labels over prototypes). No labels. Centering to prevent collapse. Local-to-global correspondence.
π Planned diagram: DINO training β student (random augment) vs. teacher (large crop EMA) with centering and sharpening.
DINOv2: Better Self-Supervised Features (2023)
Planned content: Large-scale curated data (LVD-142M). DINO + iBOT (masked image modeling) combined. Registers for global/background understanding. Strong zero-shot features: k-NN classification, segmentation, depth, without task-specific fine-tuning. The βvisual GPT-3β analogy.
π Planned diagram: DINOv2 feature quality β PCA of patch features showing object part segmentation emerging from self-supervised training alone.
EVA, InternImage, and Beyond
Planned content: EVA-CLIP: large-scale CLIP pre-training for ViT-18B. InternImage: deformable convolution as a vision foundation model. Scaling laws for vision transformers. SoViT. FlexiViT. NaViT (variable resolution). PatchDropout.
Next: Chapter 25 β Multimodal Architectures β
Last updated: May 2026