🧠 Language Modelling — The Complete Guide
From Tokens to Frontier: Understanding, building, and deploying large language models. A comprehensive learning path for ML practitioners navigating the generative AI landscape.
Who This Is For
You’re comfortable with Python and basic machine learning. You’ve heard about transformers, GPT, and LLMs — maybe even used them — but you want to truly understand how they work under the hood: from tokenization and attention mechanisms to quantization, CUDA kernels, and production deployment. This guide takes you from first principles all the way to the research frontier.
📋 Table of Contents
| # | Chapter | What You’ll Learn |
|---|---|---|
| Foundations | ||
| 1 | Introduction to Language Modelling | The full pipeline — text to tokens to tensors to predictions. Brief history, tokenization essentials, and how everything connects |
| 2 | Embeddings & Representations | From one-hot to dense vectors, positional encodings (sinusoidal, RoPE, ALiBi), and the geometry of meaning |
| 3 | The Transformer | Complete architecture walkthrough with tensor shapes at every layer — residuals, norms, feed-forward networks |
| 4 | Attention — SDPA & Multi-Head | Q/K/V math, scaled dot-product attention, multi-head mechanism, causal masking, Flash Attention |
| Attention & Architecture | ||
| 5 | Attention Variants — GQA, MQA & MLA | The KV bottleneck, grouped-query attention, multi-latent attention, sliding window — with tensor shape comparisons |
| 6 | Decoder-Only Models | GPT family, LLaMA architecture, model size tables, open-source landscape |
| 7 | Encoder & Seq2Seq Models | BERT, T5, BART — when to use encoder vs decoder vs encoder-decoder |
| 8 | Multimodal Models | Vision-language models, ViT, Whisper, modality fusion, image-to-token pipelines |
| Training | ||
| 9 | Data for LLMs & VLMs | Pre-training, mid-training, SFT, RLHF, RL data — formats, tensor shapes, and representative datasets for every phase |
| 10 | Pre-Training at Scale | Data curation, training objectives, learning rate schedules, loss curves, datasets |
| 11 | Optimizers & Loss Functions | Cross-entropy, KD loss, DPO loss; AdamW, Adam-mini, Muon, SOAP — the machinery that drives LLM training |
| 12 | Mid-Training & Continued Pre-Training | Long-context extension, domain adaptation, data mixing, annealing — the bridge between pre-training and fine-tuning |
| 13 | Fine-Tuning & Adaptation | SFT, LoRA, QLoRA, DoRA — parameter-efficient methods with rank decomposition math |
| 14 | PEFT: LoRA, QLoRA & Variants | Deep dive into LoRA math, QLoRA NF4 quantisation, DoRA, rsLoRA, VeRA, Adapters, Prefix Tuning, IA³ — with tensor shapes and selection guide |
| 15 | Alignment — RLHF & Beyond | Reward models, PPO, DPO, RLAIF, Constitutional AI — making models helpful and safe |
| Inference & Generation | ||
| 16 | Inference & Sampling Strategies | Temperature, top-k/p, beam search, speculative decoding, structured output |
| 17 | KV-Cache — Mechanics & Memory | How KV-cache works, tensor shapes through generation, memory calculations, worked examples for real models |
| 18 | KV-Cache — Optimization Strategies | PagedAttention, continuous batching, eviction policies, KV-cache quantization, prefix caching |
| Numerical Precision & Quantization | ||
| 19 | Data Types & Numerical Precision | FP32 to FP4 — bit layouts, range vs precision, mixed precision training, BF16 vs FP16 |
| 20 | Quantization Fundamentals | Affine math, symmetric vs asymmetric, calibration, weight-only vs weight-activation vs KV-cache quantization |
| 21 | Quantization Techniques — Full Landscape | Every major method: GPTQ, AWQ, SmoothQuant, KIVI, BitNet + master comparison table |
| 22 | Quantization Benchmarks & Selection | Perplexity vs bits, throughput benchmarks, decision flowchart, GGUF guide |
| 23 | Knowledge Distillation & QAD | Response/feature/relation distillation, sequence-level distillation, quantisation-aware distillation, fake quantisation with STE |
| Hardware & Kernels | ||
| 24 | GPU Architecture for ML | SMs, Tensor Cores, memory hierarchy, roofline model, A100/H100/B200 comparison |
| 25 | CUDA & Kernel Development | CUDA programming model, Triton kernels, Flash Attention design, fused operations |
| 26 | Distributed Training | DDP, tensor/pipeline parallelism, FSDP, DeepSpeed ZeRO, 3D parallelism |
| 27 | ASICs & Specialized Accelerators | TPUs, Groq, Trainium, Apple Silicon — when GPUs aren’t the answer |
| Ecosystem & Tooling | ||
| 28 | The Hugging Face Ecosystem | Hub, datasets, tokenizers, Spaces — the open-source ML platform |
| 29 | Transformers Library Deep Dive | AutoModel, config, Trainer, Pipeline — internals and patterns |
| 30 | Evaluation & Benchmarks | Perplexity, MMLU, HumanEval, lm-eval-harness, Chatbot Arena |
| 31 | Serving & Deployment | vLLM, TGI, TensorRT-LLM, Ollama, llama.cpp — from prototype to production |
| Advanced Topics | ||
| 32 | Scaling Laws & Emergent Abilities | Kaplan, Chinchilla, emergence debate — the science of “how big” |
| 33 | Mixture of Experts | Sparse routing, load balancing, Mixtral, DeepSeek-MoE — more params, same compute |
| 34 | SSMs & Beyond Transformers | Mamba, RWKV, Jamba — O(T) alternatives to quadratic attention |
| 35 | Reasoning Models | Chain-of-Thought, o1-style reasoning, process rewards, test-time compute scaling |
| Applications & Frontier | ||
| 36 | Retrieval-Augmented Generation | RAG pipeline, embedding models, vector databases, advanced retrieval |
| 37 | Agents & Tool Use | Function calling, coding agents, computer use, MCP protocol |
| 38 | The Frontier | Long context, world models, safety, interpretability — open problems |
| Appendices | ||
| A | History of Language Modelling | Full timeline from Shannon to GPT-4 — every milestone, paper, and breakthrough |
| B | Tokenization Deep Dive | BPE step-by-step, WordPiece, SentencePiece, training tokenizers from scratch |
| C | HuggingFace Model Config Field Reference | Every config.json parameter explained — values, formulae, caveats, VLM/VLA configs, and kernel pitfall checklist |
🗺️ Learning Path
Recommended Learning Path
📖 Ch 1–4: Foundations & Transformer
🧩 Ch 5–8: Attention Variants & Architectures
🏋️ Ch 9–15: Training Pipeline
⚡ Ch 16–18: Inference & KV-Cache
🔢 Ch 19–23: Precision, Quantization & Distillation
🖥️ Ch 24–27: Hardware & Kernels
🛠️ Ch 28–31: Ecosystem & Deployment
🚀 Ch 32–38: Advanced & Frontier
⚡ Quick Start Paths
Path A: “I want to understand transformers” (5 chapters)
- 01 — Introduction — the big picture
- 03 — The Transformer — architecture deep dive
- 04 — SDPA & Multi-Head Attention — attention mechanics
- 06 — Decoder-Only Models — GPT & LLaMA
- 16 — Inference & Sampling — how generation works
Path B: “I want to train / fine-tune models” (7 chapters)
- 09 — Data for LLMs & VLMs — data across all training phases
- 10 — Pre-Training at Scale — data and objectives
- 11 — Optimizers & Loss Functions — AdamW, schedules, losses
- 12 — Mid-Training — continued pre-training
- 13 — Fine-Tuning & Adaptation — SFT & LoRA
- 14 — PEFT: LoRA, QLoRA & Variants — parameter-efficient methods
- 26 — Distributed Training — scaling to multi-GPU
Path C: “I want to deploy and optimize” (7 chapters)
- 17 — KV-Cache Mechanics — memory bottleneck
- 18 — KV-Cache Optimization — PagedAttention & more
- 19 — Data Types — precision tradeoffs
- 21 — Quantization Techniques — the full landscape
- 23 — Knowledge Distillation & QAD — compress with distillation
- 29 — Transformers Library — practical tooling
- 31 — Serving & Deployment — vLLM, TGI, Ollama
Path D: “I want deep understanding” (full guide)
Read chapters 1 through 38 in order. Each builds on the previous. Appendices A and B provide additional depth on history and tokenization.
📚 Prerequisites
Before diving in, you should be comfortable with:
- Python — NumPy, basic PyTorch tensor operations
- Linear algebra — matrix multiplication, dot products, transposes
- ML basics — loss functions, gradient descent, backpropagation
- Probability — softmax, cross-entropy, sampling from distributions
- Command line — terminal, pip/conda, environment variables
📝 Changelog
| Date | Changes |
|---|---|
| April 2026 | Added Appendix C — HuggingFace Model Config Field Reference (LLM, VLM, VLA configs, kernel pitfall checklist) |
| April 2026 | Added Ch 9 (Data), Ch 11 (Optimizers & Loss Functions), Ch 14 (PEFT Deep Dive), Ch 23 (Knowledge Distillation & QAD) — renumbered all chapters accordingly |
| April 2026 | Initial release — 34 chapters + 2 appendices |
Last updated: April 2026