← Back to Table of Contents

Chapter 22 — Quantization Benchmarks & Selection Guide

“The best quantization method depends on your constraints — there is no single winner. This chapter helps you choose.”

Perplexity vs Bit-Width

Perplexity (PPL) on WikiText-2 is the standard measure of quantization quality loss. Lower is better.

LLaMA-3-8B Quantization Comparison

Scaling with Model Size

Larger models tolerate quantization much better:

Rule of thumb: INT4 quantization is nearly lossless for 70B+ models, and acceptably lossy for 7–13B models.

Speed Benchmarks by GPU

Tokens per second (batch_size=1, seq_len=512, generating 128 tokens):

For llama.cpp on CPU (Apple M2 Ultra, 192 GB):

Decision Flowchart

GPU Deployment

CPU / Local Deployment

Practical Tips

1. Start with AWQ INT4 or GPTQ INT4 for GPU inference — they have the best ecosystem support (vLLM, TGI, transformers)
2. Use Q4_K_M for llama.cpp/Ollama — best quality-per-bit for local deployment
3. NF4 + bitsandbytes for QLoRA — it’s the standard for fine-tuning on consumer GPUs
4. FP8 on H100 if available — simplest with near-zero quality loss
5. INT4 is the sweet spot — below 4 bits, quality degrades noticeably; above 4 bits, the size savings don’t justify the cost
6. Benchmark on YOUR tasks — perplexity doesn’t perfectly predict downstream task performance. Always evaluate on your actual use case.

Benchmark Script

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
"""Minimal script to compare quantization methods on perplexity."""
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

def compute_perplexity(model, tokenizer, dataset, max_samples=100, max_length=2048):
    total_loss = 0
    total_tokens = 0

    for i, sample in enumerate(dataset):
        if i >= max_samples:
            break
        inputs = tokenizer(
            sample["text"], return_tensors="pt",
            truncation=True, max_length=max_length
        ).to(model.device)

        with torch.no_grad():
            outputs = model(**inputs, labels=inputs["input_ids"])
            total_loss += outputs.loss.item() * inputs["input_ids"].shape[1]
            total_tokens += inputs["input_ids"].shape[1]

    avg_loss = total_loss / total_tokens
    return torch.exp(torch.tensor(avg_loss)).item()

# Usage
dataset = load_dataset("wikitext", "wikitext-2-raw-v1", split="test")
model = AutoModelForCausalLM.from_pretrained("your-quantized-model", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("your-quantized-model")
ppl = compute_perplexity(model, tokenizer, dataset)
print(f"Perplexity: {ppl:.2f}")

What’s Next

With quantization covered comprehensively, the next part of the guide shifts to the hardware that runs these models. The next chapter covers GPU architecture — SMs, Tensor Cores, memory hierarchy, and the roofline model.

← Previous: Chapter 21 — Quantization Techniques · Next: Chapter 23 — Knowledge Distillation & QAD →

Last updated: April 2026