Chapter 35 β Self-Supervised Learning
βThe most valuable commodity in CV is labels. Self-supervised learning is the art of learning without them.β
Why Self-Supervised Learning?
Planned content: Labeled data bottleneck: ImageNet has 1M labels, but the internet has 5 billion images. SSL: design pretext tasks where the supervisory signal comes from the data itself. Two families: contrastive (pull together similar views, push apart dissimilar) and non-contrastive (no negative pairs). Downstream transfer: SSL features often outperform supervised when fine-tuned.
π Planned diagram: SSL philosophy β two augmented views of the same image should produce similar representations; SSL framework overview.
Contrastive Learning: SimCLR (2020)
Planned content: Two augmented views of each image. Encoder
f(Β·)+ projection headg(Β·). NT-Xent loss: attract same-image pairs, repel different-image pairs within a mini-batch. Large batch (4096-8192) essential (more negatives). Strong augmentation: crop + color jitter + grayscale + Gaussian blur. Two-layer MLP head (discard after pre-training).
\[\mathcal{L}_{NT-Xent} = -\log \frac{\exp(\text{sim}(z_i, z_j)/\tau)}{\sum_{k \neq i} \exp(\text{sim}(z_i, z_k)/\tau)}\]π Planned diagram: SimCLR framework β image β two augmented views β encoder β projection head β NT-Xent loss. Batch of N β 2N views β N positive pairs + 2(N-1) negative pairs.
MoCo: Momentum Contrast (2020)
Planned content: Key idea: maintain a large queue of negative keys from past batches. Momentum encoder: slowly updated copy of encoder (like DINO teacher) to keep key representations consistent. Decouples batch size from number of negatives (queue size = 65536). MoCo v2: add MLP head, strong augmentation. MoCo v3: ViT backbone, no queue needed with large batch.
π Planned diagram: MoCo queue mechanism β current mini-batch + large queue of past negatives β InfoNCE loss. Momentum encoder updated slowly (Ξ±=0.999).
BYOL: Bootstrap Your Own Latent (2020)
Planned content: No negative pairs! Student predicts teacherβs representations of augmented views. Teacher = EMA of student. Predictor MLP on student side only. How does it avoid collapse? The asymmetry (predictor + EMA) prevents trivial solutions. Requires careful hyperparameter tuning.
π Planned diagram: BYOL architecture β online network (encoder + projector + predictor) vs. target network (EMA encoder + projector). Loss = cosine similarity between online prediction and target projection.
SimSiam: Exploring Collapse Prevention (2021)
Planned content: No negative pairs, no momentum encoder. Stop gradient on one branch only. Stop-gradient is the key mechanism preventing collapse. VICReg: variance-invariance-covariance regularization. BarlowTwins: off-diagonal cross-correlation minimization.
MAE: Masked Autoencoding (2021)
Planned content: (Also covered in Ch. 24 for ViT context) β SSL framework: mask 75% of image patches, reconstruct masked patches with a lightweight decoder. Encoder only sees visible tokens (25%) β efficient. Decoder: 8-layer transformer, discarded after pre-training. Pixel-level MSE reconstruction loss. MAE pre-training on ImageNet-1K gives better fine-tuning than supervised on ImageNet-21K.
π Planned diagram: MAE pre-training β 75% masked patches, encoder on 25% visible
[B, 49, 768]β add mask tokens β decoder[B, 196, 768]β reconstruct[B, 196, 768]pixel values β MSE.
DINO: Self-Distillation (2021)
Planned content: (Also covered in Ch. 24) β SSL framework details: global and local crops. Student sees small crops (local), teacher sees large crops (global). This forces student to understand local crops using global context from teacher. Centering prevents collapse (subtract running mean from teacher output). Sharpening (low temperature for teacher).
DINOv2: Large-Scale SSL (2023)
Planned content: Scale DINO with: curated dataset (LVD-142M), iBOT (masked image modeling + DINO), SwAV multi-crop, KoLeo regularizer (entropy of pairwise distances). Registers: dedicated register tokens absorb global background information. DINOv2 features are directly usable as embeddings for segmentation, depth, classification without fine-tuning.
π Planned diagram: DINOv2 training objectives β DINO global/local crop loss + iBOT masked patch reconstruction + KoLeo entropy regularizer combined.
Transfer Learning from SSL Features
Planned content: Linear probing: freeze backbone, train only linear head. Few-shot probing: k-NN classification. Full fine-tuning. SSL β supervised comparison: SSL often wins with >10M pre-training images. Task-specific benefits: DINOv2 for segmentation, MAE for detection, SimCLR for metric learning.
Next: Chapter 36 β Deployment β
Last updated: May 2026