Chapter 29 β Knowledge Distillation
βDistillation is mentorship in machine learning β a small student learns from a large teacherβs internal reasoning, not just its final answers.β
Hintonβs Knowledge Distillation (2014)
Planned content: Train a small student network using soft labels from a large teacher. Soft labels: the teacherβs full probability distribution (not just argmax). Temperature T smooths the distribution, revealing inter-class relationships. KD loss = CE with soft labels + CE with hard labels (weighted sum). Why soft labels help: they encode βdark knowledgeβ β how similar classes relate.
\[\mathcal{L}_{KD} = \alpha \cdot \mathcal{L}_{CE}(p_{student}, p_{teacher}^T) + (1-\alpha) \cdot \mathcal{L}_{CE}(p_{student}, y_{hard})\]π Planned diagram: KD training β teacher produces soft probabilities
[B, C], student is trained with both teacher soft targets and ground-truth one-hot labels. Temperature effect on softmax distribution.
Feature Mimicking (FitNets)
Planned content: Match intermediate feature maps, not just final predictions. Student feature
[B, C_s, H, W]β adapter conv β[B, C_t, H, W]β teacher feature. Norm-based loss, attention map transfer. Why feature-level matching provides richer supervisory signal than just logits.
π Planned diagram: Feature mimicking β teacher intermediate features vs. student features with adaptation layer, L2/cosine loss alignment.
Attention Transfer
Planned content: AT: transfer attention maps (spatial activation statistics) from teacher to student. Attention map
A = ||F||_p(p-norm across channels). The spatial locations the teacher βlooks atβ guide student learning. Zagoruyko & Komodakis (2017).
DKD: Decoupled Knowledge Distillation (2022)
Planned content: Decompose KD loss into: TCKD (Target Class KD) β information about the target class, and NCKD (Non-Target Class KD) β inter-class relationships. Show that TCKD and NCKD have contradictory effects and can be decoupled and balanced separately. Consistently outperforms vanilla KD.
Task-Specific KD in Vision
Planned content: Detection KD: Mimic proposal features, FPN features, classification + box distributions. LD (Localization Distillation): transfer box distribution from teacher. Segmentation KD: Structured feature alignment, affinity graph transfer. Pose KD: Heatmap soft label distillation.
π Planned diagram: Detection distillation β teacher detector features at FPN level P3-P7 vs. student, with weighted feature alignment loss.
Self-Distillation
Planned content: Deep supervision: shallower exits learn from the final output. Born-Again Networks: a sequence of students each trained on the previous model. DeiT: use CNN teacher to train ViT student. DINO: self-distillation without labels.
Online Distillation (Mutual Learning)
Planned content: No pre-trained teacher. Multiple students train simultaneously, each learning from the others. DML (Deep Mutual Learning). Collaborative learning. More robust than offline KD in some settings.
Next: Chapter 30 β Neural Architecture Search β
Last updated: May 2026