← Back to Table of Contents

Chapter 27 β€” Quantization

β€œQuantization is the art of storing 32 bits of knowledge in 8 bits β€” or 4 bits, or fewer.”

Quantization Fundamentals

Planned content: Why quantize: 4Γ— memory reduction (FP32 β†’ INT8), 2–4Γ— throughput on INT8 hardware (Tensor Cores, VNNI, NEON). Affine (asymmetric) quantization: x_q = round((x / s) + z), x_deq = s * (x_q - z). Symmetric: z = 0, simpler. Scale and zero-point. Per-tensor vs. per-channel quantization.

πŸ“Š Planned diagram: Quantization pipeline β€” FP32 value range β†’ clip to [Ξ±, Ξ²] β†’ scale + round β†’ INT8 integer β†’ dequantize back β†’ approximate FP32.

\[x_q = \text{clamp}\!\left(\text{round}\!\left(\frac{x}{s}\right) + z,\; q_{min},\, q_{max}\right), \qquad s = \frac{\beta - \alpha}{q_{max} - q_{min}}\]

Post-Training Quantization (PTQ)

Planned content: No retraining required. Calibration: run representative data through model to measure activation ranges (min/max, percentile, entropy/KLD). Weight quantization: per-channel symmetric. Activation quantization: per-tensor. SmoothQuant for challenging activations. Cross-layer equalization (CLE). Bias correction.

πŸ“Š Planned diagram: PTQ pipeline β€” trained FP32 model β†’ calibration dataset β†’ compute scale/zero-point per layer β†’ INT8 model β†’ evaluate accuracy.

Quantization-Aware Training (QAT)

Planned content: Simulate quantization during training: insert fake quantization ops (FQ) in forward pass (round β†’ gradient via STE). Model learns to tolerate quantization noise. Typically recovers 1-3% accuracy lost in PTQ. Straight-Through Estimator (STE): βˆ‚L/βˆ‚x β‰ˆ βˆ‚L/βˆ‚x_q (pass gradient through round op). Training cost: 10% of original training.

πŸ“Š Planned diagram: QAT forward pass β€” FP32 weights β†’ fake quantize β†’ INT8 simulation β†’ forward pass β†’ loss β†’ STE backward β†’ update FP32 weights.

πŸ“Š Planned diagram: STE gradient flow β€” showing where gradient would be zero (round op) but STE approximates it as 1 in the range.

1
2
3
4
5
6
7
8
9
10
11
12
import torch.quantization as tq

# QAT preparation: insert FakeQuantize observers
model.train()
model.qconfig = tq.get_default_qat_qconfig('fbgemm')  # x86 INT8
model_prepared = tq.prepare_qat(model)

# Train for a few epochs with fake quantization
# ...

# Convert to real INT8 model
model_int8 = tq.convert(model_prepared.eval())

Mixed Precision and Bit Widths

Planned content: Not all layers need the same precision. First and last layers: typically stay FP32 (high sensitivity). INT8 for most layers. INT4 or W4A8 for extreme compression. HAWQv2, APQ: hardware-aware mixed precision search. FP8 training (H100+).

πŸ“Š Planned diagram: Sensitivity analysis β€” accuracy drop vs. bit width for each layer, guiding mixed precision assignment.

W8A8, W4A8, W4A16 Schemes

Planned content: W8A8: INT8 weights + INT8 activations (GPU INT8 Tensor Cores). W4A16: INT4 weights + FP16 activations (memory bandwidth bound). W4A8: INT4 weights + INT8 activations (hardware-specific). GPTQ, AWQ style for W4A16.

INT8 in Computer Vision

Planned content: TensorRT INT8: post-training INT8 calibration with ICalibrator. OpenVINO POT (Post-Training Optimization Tool). PyTorch FX Graph Mode quantization. Accuracy comparison: ResNet-50 INT8 vs. FP32 (typically < 0.5% accuracy drop).

Next: Chapter 28 β€” Pruning & Sparsification β†’


Last updated: May 2026