Chapter 6 — Decoder-Only Models

“The decoder-only transformer, trained with a simple next-token prediction objective on internet-scale data, turned out to be the most powerful architecture humanity has ever built for language understanding and generation.”

The Causal Language Modelling Objective

Decoder-only models are trained to predict the next token given all previous tokens. The training objective is straightforward — minimize the cross-entropy loss:

\[\mathcal{L} = -\sum_{t=1}^{T} \log P(x_t \mid x_1, \ldots, x_{t-1})\]

This simple objective, scaled to trillions of tokens, produces models capable of reasoning, code generation, and multilingual understanding.

Causal Language Modelling — Training vs Inference

Training (Teacher Forcing)

All positions processed in parallel
Causal mask ensures each position only sees past
Loss computed at every position simultaneously
One forward pass for the entire sequence

Inference (Autoregressive)

Tokens generated one at a time
Each new token appended to context
KV-cache avoids recomputation
Prefill (parallel) → Decode (sequential)

Full Decoder Pass — Tensor Shapes

End-to-End Tensor Flow Through a Decoder Model

Token IDs: [B, T] integer indices

Token Embedding: [B, T, d_model] + positional encoding

Layer 1: RMSNorm → GQA → residual → RMSNorm → SwiGLU FFN → residual [B, T, d_model]

Layers 2 … N [B, T, d_model] — shape preserved through all layers

Final RMSNorm: [B, T, d_model]

LM Head (unembedding): [B, T, vocab_size] logits over vocabulary

Softmax → next-token probability distribution

The GPT Family

GPT (Generative Pre-trained Transformer) established the decoder-only paradigm:

GPT Family Evolution

2018

GPT-1

117M params. 12 layers. Showed unsupervised pre-training + supervised fine-tuning works.

2019

GPT-2

1.5B params. 48 layers. Zero-shot task transfer. "Too dangerous to release" (at the time).

2020

GPT-3

175B params. 96 layers, d=12288. In-context learning. Few-shot prompting.

2023–24

GPT-4 / GPT-4o

Rumored MoE architecture. Multimodal. RLHF-aligned. State-of-the-art on most benchmarks.

The LLaMA Architecture

LLaMA (Meta, 2023–2024) is the most influential open-source decoder model family. Its architectural choices have become the de facto standard:

LLaMA Architecture — Key Design Choices

Pre-RMSNorm (normalize before attention, not after)

GQA Attention with RoPE positional encoding

SwiGLU Feed-Forward Network (3 projection matrices)

No bias terms anywhere (QKV, FFN, RMSNorm)

Tied/untied embedding (varies by version)

Model Size Comparison

Model	Params	Layers	d_model	Heads (Q/KV)	d_k	FFN dim	Context	Vocab
LLaMA-1 7B	6.7B	32	4096	32/32	128	11008	2048	32000
LLaMA-2 7B	6.7B	32	4096	32/32	128	11008	4096	32000
LLaMA-2 70B	70B	80	8192	64/8	128	28672	4096	32000
LLaMA-3 8B	8B	32	4096	32/8	128	14336	8192	128256
LLaMA-3 70B	70B	80	8192	64/8	128	28672	8192	128256
LLaMA-3.1 405B	405B	126	16384	128/8	128	53248	128K	128256

Key evolution: MHA → GQA (LLaMA-2 70B onwards), 32K → 128K vocab (LLaMA-3), 2K → 128K context (LLaMA-3.1).

The Open-Source Landscape

Major Open-Weight Decoder-Only Model Families

LLaMA (Meta)

RMSNorm, SwiGLU, RoPE, GQA. 1B–405B. Most forked architecture.

Mistral / Mixtral

Sliding window attention, GQA, MoE (Mixtral). Strong at 7B scale.

Qwen 2.5 (Alibaba)

GQA, SwiGLU, RoPE. Strong multilingual. 0.5B–72B.

Gemma 2 (Google)

Alternating local/global attention. Logit soft-capping. 2B–27B.

Phi-3/4 (Microsoft)

High-quality small models. Curated training data. 3.8B–14B.

DeepSeek (V2/V3)

MLA attention, MoE. 236B total / 21B active. Cost-efficient.

Using Decoder Models with Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "meta-llama/Llama-3.1-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [{"role": "user", "content": "Explain attention in one paragraph."}]
input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)

# Inspect intermediate shapes
with torch.no_grad():
    outputs = model(input_ids, output_hidden_states=True)

print(f"Input:   {input_ids.shape}")                     # [1, T]
print(f"Hidden:  {outputs.hidden_states[-1].shape}")      # [1, T, 4096]
print(f"Logits:  {outputs.logits.shape}")                 # [1, T, 128256]

# Generate
output_ids = model.generate(input_ids, max_new_tokens=256, temperature=0.7, do_sample=True)
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

Autoregressive Generation Loop

Under the hood, model.generate() runs an autoregressive loop. Here’s what happens step by step:

def generate_simple(model, input_ids, max_new_tokens, temperature=1.0):
    """Simplified autoregressive generation loop."""
    past_key_values = None  # KV-cache

    for _ in range(max_new_tokens):
        outputs = model(
            input_ids=input_ids if past_key_values is None else input_ids[:, -1:],
            past_key_values=past_key_values,
            use_cache=True,
        )
        past_key_values = outputs.past_key_values  # cache K, V

        # Sample from the logits of the last position
        logits = outputs.logits[:, -1, :] / temperature  # [B, vocab_size]
        probs = torch.softmax(logits, dim=-1)
        next_token = torch.multinomial(probs, num_samples=1)  # [B, 1]

        input_ids = torch.cat([input_ids, next_token], dim=-1)

        if next_token.item() == model.config.eos_token_id:
            break

    return input_ids

Notice: on the first pass (prefill), the entire prompt is processed at once. On subsequent passes (decode), only the last token is fed in — the KV-cache stores everything else. See Chapter 17 for full details.

What’s Next

Decoder-only models dominate generation, but they’re not the only game in town. The next chapter covers encoder and encoder-decoder architectures that excel at understanding tasks like classification, NER, and translation.

← Previous: Chapter 5 — GQA, MQA & MLA · Next: Chapter 7 — Encoder & Seq2Seq Models →

Last updated: April 2026