← Back to Table of Contents

Chapter 30 — Evaluation & Benchmarks

“If you can’t measure it, you can’t improve it. But if you measure the wrong thing, you optimize for the wrong outcome.”

Perplexity

The most fundamental language model metric — how surprised is the model by the test data?

• Lower is better — a perfect model has PPL = 1
• Typical values: GPT-2 ~29 on WikiText-103, LLaMA-3-8B ~6
• Only meaningful for comparing models on the same tokenizer + dataset

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

def compute_perplexity(model, tokenizer, text, max_length=2048, stride=512):
    """Compute perplexity with sliding window."""
    encodings = tokenizer(text, return_tensors="pt")
    input_ids = encodings.input_ids.to(model.device)
    seq_len = input_ids.size(1)
    
    nlls = []
    for begin in range(0, seq_len, stride):
        end = min(begin + max_length, seq_len)
        target_len = end - (begin if begin == 0 else begin + max_length - stride)
        
        input_chunk = input_ids[:, begin:end]
        with torch.no_grad():
            outputs = model(input_chunk, labels=input_chunk)
        nlls.append(outputs.loss * target_len)
        
        if end == seq_len:
            break
    
    ppl = torch.exp(torch.stack(nlls).sum() / seq_len)
    return ppl.item()

Major Benchmarks

Knowledge & Reasoning Benchmarks

Math & Code Benchmarks

Chat & Human Evaluation

The Open LLM Leaderboard

HuggingFace’s Open LLM Leaderboard evaluates models using lm-eval-harness:

Leaderboard v2 suite (as of 2024):

• MMLU-Pro (knowledge)
• GPQA (expert knowledge)
• MuSR (multi-step reasoning)
• MATH (mathematical reasoning)
• BBH (Big-Bench Hard — diverse reasoning)
• IFEval (instruction following)

Running Evaluations with lm-eval-harness

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# Install
pip install lm-eval

# Evaluate a model on MMLU
lm_eval --model hf \
    --model_args pretrained=meta-llama/Llama-3.1-8B-Instruct \
    --tasks mmlu \
    --batch_size auto \
    --num_fewshot 5 \
    --output_path results/

# Multiple benchmarks
lm_eval --model hf \
    --model_args pretrained=meta-llama/Llama-3.1-8B-Instruct,dtype=bfloat16 \
    --tasks mmlu,hellaswag,arc_challenge,winogrande,gsm8k \
    --batch_size auto \
    --output_path results/

# Evaluate a GPTQ quantized model
lm_eval --model hf \
    --model_args pretrained=TheBloke/Llama-3-8B-GPTQ,autogptq=True \
    --tasks mmlu \
    --batch_size auto

Benchmark Limitations

Other limitations:

• Sensitivity to prompting: Few-shot count, prompt format, and chat template all affect scores significantly
• Saturation: Some benchmarks become too easy — HellaSwag is >95% for most modern models
• Narrow scope: Benchmarks don’t test creativity, nuance, or long-form coherence
• Static datasets: Real-world capability evolves, benchmarks don’t

Practical Evaluation Strategy

For your own fine-tuned models:

1. Perplexity on a held-out set — did training reduce loss?
2. Task-specific metrics — accuracy, F1, exact match on your use case
3. lm-eval-harness on standard benchmarks — to compare with baselines
4. Qualitative inspection — read 50 outputs manually. There is no substitute
5. Chatbot Arena (if applicable) — compare against known models

What’s Next

Evaluating models is one thing — serving them at scale is another. The next chapter covers serving and deployment with vLLM, TGI, and other inference frameworks.

← Previous: Chapter 29 — Transformers Library Deep Dive · Next: Chapter 31 — Serving & Deployment →

Last updated: April 2026