← Back to Table of Contents

Chapter 18 — KV-Cache Optimization Strategies

“The art of LLM serving is the art of KV-cache management — fitting more requests, longer contexts, and higher throughput into the same GPU memory.”

The Problem Space

The KV-cache is the dominant memory consumer in LLM inference (see Chapter 17). Optimization strategies fall into several categories:

KV-Cache Optimization Landscape
Memory Management
PagedAttention, continuous batching, prefix caching. How we allocate and reuse KV-cache memory.
Compression
Quantized KV-cache (INT4/INT8), token eviction, low-rank compression. Reduce bits per cached token.
Architectural
GQA/MQA (fewer heads), MLA (latent compression), Sliding Window (bounded cache). Design-time choices.
Offloading
CPU/disk offload of older KV blocks. Trade latency for capacity.

PagedAttention (vLLM)

PagedAttention (Kwon et al., 2023) applies virtual memory concepts to KV-cache management, eliminating fragmentation and enabling efficient memory sharing.

The Fragmentation Problem

Without PagedAttention, each request pre-allocates a contiguous KV-cache for its maximum possible length. This wastes memory:

Contiguous vs Paged KV-Cache
Contiguous (Naive)
  • Pre-allocate max_seq_len per request
  • Request uses 2K tokens but allocated 8K
  • 75% memory wasted (internal fragmentation)
  • Can't reclaim until request finishes
Paged (vLLM)
  • Allocate in fixed-size blocks (e.g., 16 tokens)
  • Blocks can be non-contiguous in physical memory
  • Allocate new blocks on demand
  • Near-zero waste, ~4× more requests per GPU

How PagedAttention Works

PagedAttention — Block-Level KV-Cache
KV-cache divided into fixed blocks: [block_size, n_kv_heads, d_k] e.g., block_size=16
Page table maps logical positions → physical block addresses like OS virtual memory
Attention kernel reads blocks via page table non-contiguous memory access
New tokens fill current block; allocate new block when full on-demand allocation

Each physical block stores KV entries for block_size tokens:

Physical KV Block
K block
[ block_size, G, d_k ]
e.g., [16, 8, 128]
V block
[ block_size, G, d_k ]
Block memory
[ 2 × 16 × 8 × 128 × 2B = 64 KB ]
per block per layer

Continuous Batching

Traditional static batching waits until a batch is full, processes all requests, and waits until all finish. Continuous batching inserts new requests as soon as any slot frees up:

Static vs Continuous Batching
Static Batching
  • Batch of 8 requests processed together
  • Request 3 finishes early → its slot is idle
  • Entire batch waits for the longest request
  • Poor GPU utilization
Continuous Batching
  • Request 3 finishes → immediately replaced
  • No idle slots, maximum GPU utilization
  • Iteration-level scheduling (per-token)
  • 2–3× higher throughput

Prefix Caching

Many requests share the same system prompt (e.g., “You are a helpful assistant…”). Prefix caching computes and stores this shared KV-cache once, then reuses it for all requests with the same prefix:

Prefix Caching with PagedAttention
Shared prefix: "You are a helpful assistant..." KV blocks cached once
Request 1: [shared prefix blocks] → [unique blocks for "What is ML?"]
Request 2: [shared prefix blocks] → [unique blocks for "Write a poem"]
Request 3: [shared prefix blocks] → [unique blocks for "Explain RoPE"]

In vLLM, shared prefix blocks use copy-on-write: all requests point to the same physical blocks. Only the unique suffix blocks are allocated per request.

KV-Cache Quantization

Storing the KV-cache in lower precision (INT8, INT4) provides direct memory savings:

KV-Cache Quantization Impact
FP16 KV-cache
[ B, G, T, d_k ] @ 2 bytes
INT8 KV-cache
[ B, G, T, d_k ] @ 1 byte → 2× savings
INT4 KV-cache
[ B, G, T, d_k ] @ 0.5 byte → 4× savings

Methods

Method Approach Key Insight
KIVI Per-channel K, per-token V quantization to 2-bit Keys have per-channel outliers; values have per-token outliers
KVQuant Non-uniform quantization with sensitivity weighting Different heads have different quantization sensitivity
QServe KV4 Joint W4A8KV4 optimization Co-design kernel for weights + activations + KV
Gear Low-rank + sparse residual KV compression Separate low-rank approximation from sparse outliers

Token Eviction and Pruning

Not all cached tokens are equally important. Token eviction strategies selectively remove less-important cached entries:

Token Eviction Strategies
H2O (Heavy-Hitter Oracle)
Keep tokens that accumulate the most attention weight over time. "Heavy hitter" tokens are consistently attended to. Budget: keep top-k important + recent window.
StreamingLLM (Attention Sinks)
Keep initial "attention sink" tokens (first 4) + recent sliding window. The first tokens absorb disproportionate attention mass regardless of content.

StreamingLLM insight: attention scores for the first few tokens are always high (attention sinks). Keeping these + a recent window enables infinite-length generation with fixed memory:

1
2

Cache layout: [sink tokens (4)] + [recent window (4092)] = fixed 4096 entries
             ↑ always kept        ↑ sliding window

Comparison of Optimization Strategies

Strategy Memory Savings Quality Impact Latency Impact Complexity
GQA (architectural) 4–8× Minimal None Design-time
MLA (architectural) 8–16× Minimal Slight overhead Design-time
Sliding Window Bounded at W Good for local tasks None Design-time
PagedAttention ~0% (reduces waste) None Slight overhead Framework
Continuous batching Higher utilization None Better Framework
Prefix caching Proportional to sharing None Faster prefill Framework
KV INT8 2× Negligible (<0.1 PPL) Slight overhead Kernel
KV INT4 4× Small (~0.2 PPL) Slight overhead Kernel
H2O eviction ~2–4× (fixed budget) Small None Runtime
StreamingLLM Bounded at window Good for streaming None Runtime
CPU offload Extends to system RAM None Higher latency System

Using These Optimizations

vLLM with PagedAttention + KV Quantization

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    tensor_parallel_size=1,
    dtype="bfloat16",
    kv_cache_dtype="fp8_e5m2",   # FP8 KV-cache — 2× savings
    max_model_len=32768,
    enable_prefix_caching=True,   # Prefix caching enabled
    gpu_memory_utilization=0.90,  # Use 90% of GPU for KV blocks
)

# PagedAttention and continuous batching are automatic
outputs = llm.generate(
    ["What is PagedAttention?", "Explain KV-cache quantization."],
    SamplingParams(temperature=0.7, max_tokens=512),
)

TGI with KV-Cache Options

1
2
3
4
5
6
7

docker run --gpus all \
  -e MODEL_ID=meta-llama/Llama-3.1-8B-Instruct \
  -e QUANTIZE=bitsandbytes-fp4 \
  -e MAX_INPUT_LENGTH=4096 \
  -e MAX_TOTAL_TOKENS=8192 \
  -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference

What’s Next

KV-cache precision is just one aspect of numerical representation. The next chapter provides a comprehensive guide to data types and numerical precision — the foundation for understanding quantization.

← Previous: Chapter 17 — KV-Cache — Mechanics & Memory · Next: Chapter 19 — Data Types & Numerical Precision →

Last updated: April 2026

GitHub · Language Modelling — The Complete Guide · Last updated April 2026