← Back to Table of Contents

Chapter 35 — Reasoning Models

“Chain-of-thought doesn’t teach the model new facts — it teaches the model to use the facts it already knows, step by step.”

The Reasoning Gap

Standard LLMs generate token by token, choosing the most likely next token. This works well for fluent text but struggles with tasks that require multi-step reasoning:

• “What is 347 × 892?” — requires carrying digits across steps
• “If all roses are flowers and some flowers fade quickly, can we conclude some roses fade quickly?” — requires logical chaining
• “Write a function that finds the longest increasing subsequence” — requires algorithm design

Reasoning models address this by generating intermediate reasoning steps before the final answer.

Chain-of-Thought (CoT) Prompting

Wei et al. (2022) showed that adding “Let’s think step by step” dramatically improves math and logic:

1
2
3
4
5
6
7
8
9
10
11
12
13
# Zero-shot CoT — just append the magic phrase
prompt = """Q: A store had 45 apples. They sold 12 on Monday and received 
a shipment of 30 on Tuesday. Then sold 25 on Wednesday. How many remain?

Let's think step by step:"""

# Few-shot CoT — provide worked examples in the prompt
few_shot_prompt = """Q: If there are 3 cars in the parking lot and 2 more arrive, 
how many cars are there?
A: There are originally 3 cars. 2 more arrive. 3 + 2 = 5. The answer is 5.

Q: {actual_question}
A:"""

CoT works because it:

1. Decomposes complex problems into simpler sub-problems
2. Allocates more compute: more tokens = more FLOPs per problem
3. Creates intermediate representations the model can condition on
4. Only effective at ≥ ~60B parameters (smaller models generate incoherent chains)

Self-Consistency

Sample multiple chain-of-thought reasoning paths and take the majority vote:

1
2
3
4
5
6
7
8
9
10
11
12
13
import collections

def self_consistency(model, prompt, n_samples=10, temperature=0.7):
    """Generate multiple reasoning paths and majority vote."""
    answers = []
    for _ in range(n_samples):
        response = model.generate(prompt, temperature=temperature, max_tokens=512)
        answer = extract_final_answer(response)  # parse the numeric/categorical answer
        answers.append(answer)
    
    # Majority vote
    counter = collections.Counter(answers)
    return counter.most_common(1)[0][0]

Self-consistency improves GSM8K accuracy by 5–15% over single-sample CoT.

Test-Time Compute Scaling (o1-style)

The breakthrough insight: scaling compute at inference time (more thinking tokens) can be as effective as scaling model size:

How o1 Works (Conceptual)

OpenAI’s o1 family generates long internal reasoning chains before answering:

Key capabilities trained into reasoning models:

• Backtracking: “Wait, that’s wrong. Let me try a different approach.”
• Verification: “Let me check: 347 × 892 = 309,524. Verify: 892 × 300 = 267,600, 892 × 47 = 41,924. Sum: 309,524. ✓”
• Self-reflection: “This approach is getting too complicated. There might be a simpler way.”

Process Reward Models vs Outcome Reward Models

Training a PRM requires step-level annotations (expensive) or automated verification (for math/code where answers can be checked programmatically).

Open-Source Reasoning Models

DeepSeek-R1

DeepSeek-R1 demonstrated that reasoning can emerge from pure RL without supervised CoT data:

1. Start with DeepSeek-V3 base model
2. Apply RL (GRPO) with only outcome verification (correct/incorrect)
3. The model spontaneously learns to reason, self-verify, and backtrack
4. Distill the reasoning ability into smaller models (1.5B, 7B, 14B, 32B, 70B)

Budget Forcing: Controlling Reasoning Depth

Not every question needs deep reasoning. Budget forcing lets you control how much the model “thinks”:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# Conceptual: control reasoning depth
def generate_with_budget(model, prompt, thinking_budget="medium"):
    """
    thinking_budget: "low" (fast), "medium", "high" (thorough)
    """
    budgets = {"low": 256, "medium": 1024, "high": 4096}
    max_thinking_tokens = budgets[thinking_budget]
    
    response = model.generate(
        prompt,
        max_tokens=max_thinking_tokens + 512,  # thinking + answer
        stop=["</answer>"],
    )
    return extract_answer(response)

The Reasoning Landscape

What’s Next

Reasoning helps models think better with their internal knowledge. But what if the model’s knowledge is outdated or insufficient? The next chapter covers Retrieval-Augmented Generation (RAG) — grounding model outputs in external, up-to-date information.

← Previous: Chapter 34 — SSMs & Beyond Transformers · Next: Chapter 36 — Retrieval-Augmented Generation →

Last updated: April 2026