← Back to Table of Contents

Chapter 21 — Quantization Techniques: The Full Landscape

“Every quantization method is a different answer to the same question: how do you preserve accuracy while dramatically reducing precision?”

Overview

This chapter covers every major quantization technique for LLMs, organized by what they quantize.

Weight-Only Quantization

GPTQ (Frantar et al., 2022)

GPTQ applies Optimal Brain Quantization (OBQ) layer by layer — it quantizes each weight column while compensating the error in remaining columns using the inverse Hessian of the layer’s output.

Key ideas:

• Quantize weights one column at a time using second-order (Hessian) information
• After quantizing column j, update remaining columns to minimize total output error
• Uses a calibration dataset (~128 samples) to estimate the Hessian

Typical config: INT4, group_size=128, symmetric quantization.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig

quantization_config = GPTQConfig(
    bits=4,
    group_size=128,
    dataset="c4",            # calibration dataset
    desc_act=True,           # order columns by activation magnitude
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=quantization_config,
    device_map="auto",
)
model.save_pretrained("Llama-3.1-8B-GPTQ-INT4")

AWQ (Lin et al., 2023)

Activation-Aware Weight Quantization: not all weight channels are equally important. Channels that correspond to large activations (salient channels) should be preserved with higher precision.

Key ideas:

• Identify salient weight channels by observing activation magnitudes on calibration data
• Scale salient channels up before quantization, scale down after (equivalent to multiplying activations by inverse)
• This moves quantization difficulty from sensitive channels to insensitive ones
• No Hessian computation — faster than GPTQ

1
2
3
4
5
6
7
8
9
10
from awq import AutoAWQForCausalLM

model = AutoAWQForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")

model.quantize(
    tokenizer,
    quant_config={"w_bit": 4, "q_group_size": 128, "version": "gemm"},
)
model.save_quantized("Llama-3.1-8B-AWQ")

GGUF / llama.cpp Quantization

GGUF (GPT-Generated Unified Format) is the format used by llama.cpp and Ollama. It supports many quantization variants with different quality/size tradeoffs:

1
2
3
# Convert and quantize with llama.cpp
python convert_hf_to_gguf.py meta-llama/Llama-3.1-8B --outtype f16
./llama-quantize Llama-3.1-8B-F16.gguf Llama-3.1-8B-Q4_K_M.gguf Q4_K_M

bitsandbytes

The library behind QLoRA. Two main modes:

1
2
3
4
5
6
7
8
9
10
11
12
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

# 4-bit NF4 with double quantization (QLoRA config)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B", quantization_config=bnb_config, device_map="auto"
)

Other Weight-Only Methods

Weight + Activation Quantization

SmoothQuant (Xiao et al., 2022)

The key insight: weights are easy to quantize; activations are hard (due to outliers). SmoothQuant migrates difficulty from activations to weights by applying per-channel scaling:

The scaling factor $s$ balances the quantization difficulty between X and W per channel.

QServe / QoQ (W4A8KV4)

QServe (Lin et al., 2024) achieves the holy grail: 4-bit weights, 8-bit activations, AND 4-bit KV-cache in a single framework. Key technique: QoQ (Quattuor-Octo-Quattuor) progressive quantization.

FP8 Inference

On H100+ GPUs, FP8 inference is native and requires minimal changes:

1
2
3
4
5
6
7
8
9
10
11
12
from transformers import AutoModelForCausalLM

# FP8 inference with transformers
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    torch_dtype=torch.float8_e4m3fn,  # FP8 weights
    device_map="auto",
)

# Or with vLLM
from vllm import LLM
llm = LLM(model="meta-llama/Llama-3.1-8B", dtype="float16", quantization="fp8")

Quantization-Aware Training (QAT)

BitNet (Wang et al., 2023)

Trains 1-bit (ternary: -1, 0, +1) transformers from scratch. No floating-point weights at all.

BitNet can match FP16 quality at 3B+ params while being dramatically more efficient. The challenge: requires training from scratch.

Master Comparison Table

Dequantization at Runtime

When a layer performs its computation, INT4 weights are dequantized on the fly:

Efficient implementations (GPTQ, AWQ kernels) fuse unpacking + dequantization + matmul into a single GPU kernel, avoiding materializing the full FP16 weight matrix.

What’s Next

With all techniques covered, the next chapter provides benchmarks and a selection guide — empirical comparisons of quality, speed, and memory across methods, plus a decision flowchart for choosing the right approach.

← Previous: Chapter 20 — Quantization Fundamentals · Next: Chapter 22 — Quantization Benchmarks →

Last updated: April 2026