← Back to Table of Contents

Chapter 9 — Data for LLMs & VLMs

“A model is only as good as its data. Every capability you see in a frontier model traces back to some text, image, or interaction in the training set.”

The Data Hierarchy

Modern LLM/VLM training unfolds in distinct phases — each with its own data format, objective, and tensor shapes:

Pre-Training Data

Pre-training builds world knowledge and language understanding. Models learn by predicting the next token over massive, diverse corpora.

What Pre-Training Data Looks Like

Objective: Next-token prediction (causal language modelling)

Data source examples: Common Crawl, GitHub, Wikipedia, ArXiv, Books, StackExchange

Raw example (one document, packed into a context window):

1
2
The transformer architecture was introduced in 2017 by Vaswani et al.
It replaced recurrent networks with self-attention, enabling parallelism...

After tokenization and packing:

1
2
tokens = [464, 47058, 10478, 373, 5495, 287, 2177, 416, 569, 853, 648, ...]
# shape: [seq_len]  e.g. [4096]

Multiple documents are concatenated with a special separator token <|endoftext|> and packed to fill max_seq_len, minimising padding waste:

1
doc1_tokens + [EOS] + doc2_tokens + [EOS] + doc3_tokens... → packed to [4096]

Tensor Shapes — Pre-Training Batch

Where B = batch size (e.g., 1024–4096 with gradient accumulation), T = sequence length (e.g., 4096–8192).

Concrete example (LLaMA-3 8B training):

• B = 1024, T = 4096 → 4.2M tokens per step
• Global batch: often 4M–16M tokens
• input_ids: [1024, 4096] of dtype int32
• Logits output: [1024, 4096, 128256] (vocab size 128k)
• Loss: scalar (mean cross-entropy over all B×T positions)

Pre-Training Data Mix

Key pre-training datasets:

Mid-Training / Continued Pre-Training Data

Mid-training extends or adapts a pre-trained model. The data format is identical to pre-training (next-token prediction), but the source mix changes to achieve specific goals.

Goals and Data Strategies

Tensor shapes: identical to pre-training but T is larger:

• Long-context: input_ids shape [B, 32768] or [B, 131072]
• Batch size B decreases as T increases to maintain memory budget

Post-Training Data: SFT (Supervised Fine-Tuning)

SFT teaches the model to follow instructions. Data is formatted as conversations.

Chat Format

Modern models use a structured chat template. Example (Llama-3 chat format):

1
2
3
4
5
6
7
8
<|begin_of_text|>
<|start_header_id|>system<|end_header_id|>
You are a helpful assistant.<|eot_id|>
<|start_header_id|>user<|end_header_id|>
Explain gradient descent in simple terms.<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>
Gradient descent is an optimisation algorithm...
<|eot_id|>

Loss masking: loss is computed only on assistant tokens (instruction/system tokens are masked):

1
2
3
4
# labels shape: [B, T]
# -100 is the ignore index for cross-entropy loss
labels = [-100, -100, ..., -100,   # system + user tokens (masked)
           tok1, tok2, ..., tokN]   # assistant tokens (loss computed here)

SFT Tensor Shapes

Typical dimensions: B=4–32, T=2048–8192

SFT Data Sample

1
2
3
4
5
6
7
{
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is backpropagation?"},
    {"role": "assistant", "content": "Backpropagation is the algorithm used to train neural networks. It computes gradients of the loss with respect to each parameter by applying the chain rule of calculus backwards through the network..."}
  ]
}

Key SFT datasets:

Post-Training Data: RLHF & Preference Learning

After SFT, alignment training uses comparison data — pairs of responses where humans (or models) indicate which is better.

Reward Model Data

Used to train a reward model that scores responses:

1
2
3
4
5
{
  "prompt": "Explain quantum entanglement.",
  "chosen": "Quantum entanglement is a phenomenon where two particles...",
  "rejected": "Quantum entanglement is when particles are quantum."
}

Tensor shapes for reward model training:

The reward model outputs a scalar from the last token’s hidden state:

1
hidden_states: [B, T, d_model]  →  last token: [B, d_model]  →  reward: [B, 1]

DPO / Direct Preference Optimisation Data

DPO skips the reward model and trains directly on preference pairs:

1
(prompt, chosen_response, rejected_response)

The DPO loss uses log-probabilities of both responses under the current and reference model.

Key preference datasets:

Post-Training Data: Reinforcement Learning (RL / GRPO / RLVR)

The latest frontier in post-training uses verifiable rewards — the model generates answers and receives a binary or graded reward signal based on correctness.

RL with Verifiable Rewards (RLVR)

Used for math, code, and reasoning tasks where answers can be checked programmatically.

Data format: problems with ground-truth answers

1
2
3
4
5
{
  "problem": "Solve: 3x + 7 = 22. Find x.",
  "answer": "5",
  "domain": "algebra"
}

Training loop:

1. Model generates multiple completions (rollouts): [B × G, T] where G = group size (e.g., 8)
2. Each completion is scored: correct answer → reward +1, wrong → 0 or -1
3. GRPO computes advantage by comparing against group mean reward
4. Policy gradient update improves high-reward completions

Tensor shapes for GRPO:

Key RL datasets:

Process Reward Models (PRMs)

Instead of just rewarding the final answer, PRMs give step-level rewards for each reasoning step in a chain-of-thought.

Data format (step-annotated):

1
2
3
4
5
6
7
8
{
  "problem": "If a train travels at 60 mph for 2.5 hours, how far does it go?",
  "steps": [
    {"text": "Distance = speed × time", "correct": true},
    {"text": "Distance = 60 × 2.5", "correct": true},
    {"text": "Distance = 150 miles", "correct": true}
  ]
}

Reward tensor shapes:

VLM Data — Vision-Language Models

VLMs require paired image-text data across all training phases.

Pre-Training VLM Data

Large-scale image-text pairs (alt-text, web captions):

1
[image_tokens (256–2048 tokens)] + [caption_tokens]

Tensor shapes (vision encoder output fused with language model):

Key VLM pre-training datasets:

SFT for VLMs

Structured conversations with image inputs:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "image", "image": "<base64_or_path>"},
        {"type": "text", "text": "Describe what you see in this image."}
      ]
    },
    {
      "role": "assistant",
      "content": "The image shows a sunset over the ocean..."
    }
  ]
}

Key VLM instruction datasets:

Data Quality vs. Quantity

A common insight from scaling: quality beats raw token count at a fixed compute budget.

Key findings from research:

• Phi-1 (2023): 1.3B model trained on “textbook quality” synthetic data matched models 10× larger
• LIMA (2023): 1000 carefully curated SFT examples matched 52K example models on alignment
• MiniCPM (2024): Annealing on high-quality data in final training stage boosts benchmarks significantly

Data Contamination & Deduplication

A critical but often overlooked issue: benchmark contamination — when test data leaks into training data.

Deduplication methods:

Contamination detection: n-gram overlap between training data and benchmark test sets. Most responsible training runs report contamination analysis.

Summary: Data Shapes Across Phases

What’s Next

With a solid understanding of data across all training phases, we’re ready to look at pre-training at scale — how you orchestrate a trillion-token training run.

← Previous: Chapter 8 — Multimodal Models · Next: Chapter 10 — Pre-Training at Scale →

Last updated: April 2026