Chapter 34 β Training Recipes
βThe difference between a 75% and an 80% model is often not the architecture β itβs the training recipe.β
Loss Functions in Computer Vision
Planned content: Cross-entropy (classification). BCE with logits (multi-label, detection objectness). Focal loss (imbalanced detection). Smooth L1 / Huber (bounding box regression). IoU-based losses (GIoU, DIoU, CIoU for detection). Dice + LovΓ‘sz (segmentation). Wing loss (landmark regression). OKS loss (pose estimation heatmaps: MSE on heatmaps). Triplet loss, ArcFace (metric learning).
π Planned table: Loss function reference β task, formula, hyperparameters, when to use.
Optimizers
Planned content: SGD + momentum: classic, requires careful LR tuning. AdamW: adaptive LR, weight decay decoupled (Loshchilov & Hutter). Best for ViT/transformer training. LARS (Layer-Adaptive Rate Scaling): scales LR per layer based on weight norm, enables large-batch training. LAMB: LARS for transformers. SAM (Sharpness-Aware Minimization): seeks flatter minima, improved generalization. Lion (EvoLved Sign Momentum): sign-based update, memory efficient.
π Planned diagram: SGD vs. AdamW loss landscape trajectory β SGD with sharp minimum vs. SAMβs flat minimum, and why flat = better generalization.
Learning Rate Schedules
Planned content: Constant LR: simple but suboptimal. Step decay (every 30 epochs Γ·10). Multi-step. Cosine annealing: smooth decay to 0. Cosine with restarts (SGDR). OneCycleLR: 1-cycle policy (warm up β peak β anneal). Warmup + cosine: standard for ViT training (5-20 epoch warmup). Linear warmup formula. Effect of warmup for batch-normalized models.
π Planned diagram: LR schedule comparison β step decay vs. cosine annealing vs. OneCycle vs. warmup+cosine, all on same axes (epoch vs. LR).
1
2
3
4
5
6
7
8
9
import torch.optim as optim
from torch.optim.lr_scheduler import CosineAnnealingLR, LinearLR, SequentialLR
optimizer = optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.05)
# Warmup 5 epochs + cosine 295 epochs
warmup = LinearLR(optimizer, start_factor=1e-4, end_factor=1.0, total_iters=5)
cosine = CosineAnnealingLR(optimizer, T_max=295, eta_min=1e-5)
scheduler = SequentialLR(optimizer, schedulers=[warmup, cosine], milestones=[5])
Exponential Moving Average (EMA)
Planned content: Maintain a shadow copy of model weights as EMA:
ΞΈ_ema = Ξ± * ΞΈ_ema + (1-Ξ±) * ΞΈ. EMA model used for validation/inference. Smoother loss surface. Used in YOLO, DINO, SAM. Typical Ξ± = 0.999. Why EMA helps: averages out gradient noise, approximates ensemble.
1
2
3
4
5
6
7
8
9
10
class EMA:
def __init__(self, model, decay=0.999):
self.model = copy.deepcopy(model)
self.decay = decay
@torch.no_grad()
def update(self, model):
for ema_p, p in zip(self.model.parameters(), model.parameters()):
ema_p.data = self.decay * ema_p.data + (1 - self.decay) * p.data
# ema_p: exponential moving average of parameter
Bag of Tricks for ResNet Training
Planned content: He et al. βBag of Tricks for Image Classificationβ: Linear LR scaling rule (LR β batch size / 256). Warmup. No bias decay. Label smoothing. MixUp. Cosine LR. ResNet-D: anti-aliased downsampling. Zero-initialization of last BN in residual blocks.
Large-Batch Training
Planned content: Linear scaling rule. LARS for very large batches (64K+). Gradient accumulation as a substitute. Synchronous vs. asynchronous data-parallel training. Batch size vs. accuracy at ImageNet.
Gradient Clipping
Planned content: Why gradients explode (especially in transformers, RNNs). Gradient norm clipping: rescale gradient if norm exceeds threshold. Gradient value clipping. PyTorch
nn.utils.clip_grad_norm_. Typical threshold: 1.0 for transformers, higher for CNNs.
Mixed Precision Training
Planned content: Train with FP16/BF16, master weights in FP32.
torch.autocast+GradScaler. Loss scaling to prevent underflow in FP16. BF16 advantages (larger range, no scaling needed). Memory saving: ~2Γ vs. FP32.
Next: Chapter 35 β Self-Supervised Learning β
Last updated: May 2026