🦙 Llama Model Family — A Comprehensive Technical Reference

From a fully open-weight decoder-only Transformer to a native multimodal Mixture-of-Experts colossus — the complete evolutionary arc of Meta's Llama series.

📑 Table of Contents

• Executive Summary
• Version Release Timeline
• Cross-Version Benchmark Comparison
• Master Architecture Diagram
• Llama 1 — February 2023
• Llama 2 — July 2023
• Llama 3 — April 2024
• Llama 3.1 — July 2024
• Llama 3.2 — September 2024
• Llama 3.3 — December 2024
• Llama 4 — April 2025
• References

📋 Executive Summary

The Llama (Large Language Model Meta AI) series is Meta’s flagship family of open-weight language models, spanning from a pure research release in February 2023 to a production-scale multimodal Mixture-of-Experts system in April 2025.

• Llama 1 (Feb 2023): Decoder-only Transformer with RoPE, RMSNorm, and SwiGLU; 4 sizes (7B–65B); sparked the open-source LLM revolution.
• Llama 2 (Jul 2023): Expanded context to 4K, added GQA for larger models, introduced RLHF-tuned Chat variants with Ghost Attention; commercial license.
• Llama 3 (Apr 2024): GQA universally applied, 128K vocabulary (tiktoken), 15T training tokens; 8B and 70B flagship sizes.
• Llama 3.1 (Jul 2024): 128K context via RoPE scaling (θ=500,000), new 405B flagship, multilingual capability, tool use, multi-stage post-training pipeline.
• Llama 3.2 (Sep 2024): Introduced vision models (11B, 90B with cross-attention adapters) and small edge models (1B, 3B via pruning/distillation).
• Llama 3.3 (Dec 2024): Single drop-in 70B improvement via enhanced post-training — better math, reasoning, and coding at lower compute cost.
• Llama 4 (Apr 2025): Native early-fusion multimodal MoE (Scout 17B active / Maverick 17B active / Behemoth 288B active); iRoPE positional encoding; up to 10M context.

📅 Version Release Timeline

| Version | Release | Key Milestone | Parameters | Context | |:-------:|:-------:|:-------------|:----------:|:-------:| | | Feb 2023 | First fully open-weight competitive LLM | 7B – 65B | 2K | | | Jul 2023 | Commercial license, RLHF Chat, GQA (large models) | 7B – 70B | 4K | | | Apr 2024 | 128K vocab, GQA universal, 15T tokens | 8B, 70B | 8K | | | Jul 2024 | 128K context, 405B flagship, tool use | 8B – 405B | 128K | | | Sep 2024 | Vision models + edge models (1B/3B) | 1B – 90B | 128K | | | Dec 2024 | Improved post-training; 70B only | 70B | 128K | | | Apr 2025 | Native multimodal MoE, iRoPE, 10M ctx | 109B–~2T total | 10M |

📊 Cross-Version Benchmark Comparison

🏗️ Master Architecture Diagram

🟤 Llama 1 — February 2023

Summary

• Architecture: Pure decoder-only Transformer; no encoder cross-attention.
• Positional Encoding: Rotary Position Embeddings (RoPE) with θ=10,000 applied to every attention layer.
• Normalization: Pre-norm RMSNorm (replaces LayerNorm entirely); eliminates mean centering for efficiency.
• Activation: SwiGLU feed-forward with three weight matrices (W1, W2, W3); intermediate dim ≈ 2/3 × 4d.
• Attention: Multi-Head Attention (MHA) — no grouped-query; full Q/K/V projections per head.
• Vocabulary: 32,000 tokens via SentencePiece BPE (byte-level fallback for unknown characters).
• Context Window: 2,048 tokens (causal attention mask; no sliding window).
• Training Data: Publicly available corpora — CommonCrawl, C4, GitHub, Wikipedia, Books, ArXiv, StackExchange; totaling ~1T (7B/13B) and 1.4T (33B/65B) tokens.
• Optimizer: AdamW, β₁=0.9, β₂=0.95, ε=10⁻⁵; cosine LR schedule; weight decay 0.1; gradient clipping 1.0.
• Training Infrastructure: 2,048 A100 80GB GPUs (65B model); efficient memory via FlashAttention v1 and activation checkpointing.
• Open Release: Weights released under a non-commercial research license; spawned hundreds of community fine-tunes (Alpaca, Vicuna, WizardLM, etc.).

Architecture Diagram

Community Perspective

Llama 1 fundamentally democratised large language model research. Within weeks of its release (initially leaked, then officially distributed), the community produced Alpaca (Stanford, Stanford CRFM fine-tune using GPT-4-generated instruction data), Vicuna (LMSYS, ShareGPT conversations), WizardLM, and dozens of quantised variants via llama.cpp and GGUF format. The model demonstrated that publicly available data alone could match or surpass contemporaneous commercial APIs at a fraction of the compute cost, validating scaling law predictions and opening a new era of community-driven alignment research.

Model Variants

🟠 Llama 2 — July 2023

Summary

• Context Expansion: Window doubled to 4,096 tokens; trained with a longer document mix.
• Grouped-Query Attention (GQA): Introduced for 34B and 70B models only; 7B and 13B retain full MHA.
• GQA Configuration (70B): 40 query heads, 8 key-value heads (5:1 ratio); dramatically reduces KV-cache memory.
• GQA Configuration (34B): 64 query heads, 8 key-value heads (8:1 ratio).
• Training Data: ~2 trillion tokens; updated pretraining mix with additional helpfulness-oriented data.
• Chat Variants: First official instruction-tuned release; SFT on >27,500 human-annotated examples.
• RLHF Pipeline: Two separate reward models trained — one for helpfulness, one for safety; Proximal Policy Optimisation (PPO) applied iteratively.
• Ghost Attention (GAtt): Synthetic technique to preserve system-prompt adherence across multi-turn dialogue; system instruction replicated at each user turn during training.
• Safety Measures: Red-teaming, safety reward model, context distillation, human evaluations; “responsible use guide” released alongside weights.
• Commercial License: Llama 2 Community License allows commercial use for organisations with fewer than 700M monthly active users (separate license for larger entities).
• Code Model: Code Llama released separately (Aug 2023) on top of Llama 2 base; 7B/13B/34B with 100K context infill.

Architecture Diagram

Community Perspective

Llama 2’s commercial license transformed the ecosystem: enterprises could now legally deploy and fine-tune these weights in production. The 70B Chat model quickly became the reference open-weight instruction-tuned LLM, benchmarked extensively against GPT-3.5. The Ghost Attention mechanism addressed a key weakness in multi-turn instruction following that plagued early RLHF systems. Code Llama (built on Llama 2) became one of the most widely adopted open code generation models, directly influencing Copilot-alternative tooling.

Model Variants

🟡 Llama 3 — April 2024

Summary

• Architecture: Same decoder-only Transformer skeleton; GQA now applied universally across all model sizes.
• Vocabulary Jump: 128,256 tokens using tiktoken BPE (previously 32K SentencePiece); much better multilingual and code tokenisation.
• Model Sizes: Two flagship sizes — 8B and 70B; 400B+ in pre-release preview.
• Context Window: 8,192 tokens (2× Llama 2); RoPE θ unchanged at 500,000 in the 3.1 release but at 8K here.
• Training Scale: 15T tokens (7.5× Llama 2); quality-filtered CommonCrawl + code + multilingual sources; 95% English.
• 8B Config: 32 layers, 32 Q heads, 8 KV heads (4:1 GQA), hidden dim 4096.
• 70B Config: 80 layers, 64 Q heads, 8 KV heads (8:1 GQA), hidden dim 8192.
• Post-Training: SFT (high-quality human instructions) → Rejection Sampling → PPO → DPO; four distinct stages.
• Safety Tooling: Llama Guard 2 and Code Shield released alongside; Meta Prompt Guard for injection detection.
• FlashAttention 2: Used throughout training; significantly improves memory efficiency vs v1 used in Llama 1.
• Instruction-tuned Variants: Llama 3 Instruct models show substantial jump on HumanEval (81.7 for 70B-Instruct) and MMLU (82.0) over Llama 2 70B.

Architecture Diagram

Community Perspective

Llama 3 represented a watershed moment in open-weight capabilities. The 70B Instruct model surpassed GPT-3.5 on several established benchmarks and competed seriously with early GPT-4 variants on coding tasks. The tiktoken vocabulary change, while breaking compatibility with prior Llama tokenisers, dramatically improved multilingual efficiency. The Llama Guard safety tooling suite became a widely referenced framework for responsible deployment of open models, with organisations like Hugging Face and AI safety researchers publishing extensive evaluations.

Model Variants

🔶 Llama 3.1 — July 2024

Summary

• Context Explosion: Long-context variant extends window to 128,192 tokens via RoPE θ scaling to 500,000; trained with sequences up to 128K.
• New Flagship — 405B: 126 layers, 128 Q heads, 8 KV heads, hidden dim 16,384; largest open-weight dense model at time of release.
• Multilingual: Official support for 8 languages — English, German, French, Italian, Portuguese, Hindi, Spanish, Thai.
• Tool Use & Function Calling: First Llama to support structured JSON tool calls natively in the instruct model; enables agentic pipelines.
• Multi-Stage Post-Training: Supervised Fine-Tuning → Rejection Sampling (per-language) → DPO → PPO; each stage builds on the last.
• Synthetic Data at Scale: Meta generated large volumes of synthetic instruction and reasoning data (similar to Nemotron-4 approach); key to 405B performance.
• Distillation from 405B: The 8B and 70B Instruct models in 3.1 were explicitly improved via knowledge distillation from the 405B teacher.
• VRAM Requirements: 405B in BF16 requires ~8 × H100 80GB GPUs; 70B in BF16 requires ~2 × H100.
• International Safety Standard: First open model released with a model card addressing EU AI Act risk categories.
• Benchmark: 405B scores 88.6 on MMLU, 61.6 on GPQA Diamond, competing with GPT-4o at time of release.

Architecture Diagram

Community Perspective

Llama 3.1 405B became a reference point for the entire open-source LLM ecosystem — arguably the first model to make GPT-4-level performance accessible without API dependence for well-resourced organisations. The 128K context window unlocked document-level reasoning tasks that were previously exclusive to commercial APIs. The officially sanctioned distillation policy (using 405B outputs to improve smaller Llama variants) enabled a new class of small-but-capable fine-tunes; community projects like Hermes 3, Nous-Capybara, and dozens of GGUF quantisations proliferated rapidly.

Model Variants

🔷 Llama 3.2 — September 2024

Summary

• Two Product Lines: Vision models (11B, 90B) and edge/mobile text models (1B, 3B).
• Vision Architecture: Cross-attention adapter (Flamingo-style) attaching a frozen ViT image encoder to Llama 3.1 text backbone; cross-attention inserted at every fourth transformer layer.
• Image Encoder: ViT-H/14 backbone, image resolution 560 × 560 px; tiled to handle higher-resolution inputs.
• Vision–Language Alignment: Linear projection of ViT patch embeddings followed by cross-attention; text decoder treats image features as external memory.
• 11B Vision Model: Llama 3.1 8B text backbone + vision adapter layers; strong single-image understanding, OCR, charts.
• 90B Vision Model: Llama 3.1 70B text backbone + vision adapter; multi-image reasoning, document analysis, visual QA.
• 1B Edge Model: Obtained via structured pruning of Llama 3.1 8B followed by knowledge distillation; targets mobile NPUs and CPUs.
• 3B Edge Model: Pruned/distilled from Llama 3.1 70B with 8B as intermediate teacher; on-device performance near 7B-class.
• Quantisation: 1B and 3B released in INT4 SpinQuant and QAT quantised variants for mobile deployment (Apple silicon, Arm, Qualcomm).
• On-Device Context: Edge models support 128K context with quantised KV cache; designed for real-time streaming inference.
• Meta AI Connect: Announced at Meta Connect 2024 with live demo of on-device inference on Ray-Ban Meta glasses.

Architecture Diagram

Community Perspective

Llama 3.2 marked Meta’s entry into the on-device AI race. The 1B and 3B models demonstrated that a well-distilled small model could punch significantly above its weight class, reviving interest in model compression research. The vision models provided the first truly open-weight multimodal alternative to GPT-4V, with the 90B competing on several VQA and document benchmarks. The Flamingo-style cross-attention adapter architecture became a popular blueprint for community vision extensions to other text-only models.

Model Variants

🟫 Llama 3.3 — December 2024

Summary

• Single Model Release: Only one model size — 70B; same architectural specification as Llama 3.1 70B.
• Enhanced Post-Training: Entirely revised SFT dataset, new DPO preference pairs, and extended PPO reward modelling; no architectural changes.
• Math & Reasoning Gains: MATH benchmark improvement to ~77 (vs Llama 3.1 70B ~65); structured chain-of-thought data incorporated.
• Coding Improvements: HumanEval rises to 88.4 (vs 80.5 for 3.1 70B Instruct); new code-specific SFT and DPO data.
• Instruction Following: Significantly improved IFEval scores; better adherence to format constraints (JSON, markdown, lists).
• Multilingual Retention: All 8 languages from 3.1 maintained; additional data for Hindi and Thai in post-training.
• Cost–Performance Target: Designed to offer Llama 3.1 405B-class performance at 70B inference cost; Meta claims parity on many benchmarks.
• Drop-in Replacement: Compatible with Llama 3.1 70B serving infrastructure; same tokeniser, same context length, same tool-call schema.
• Safety Updates: Updated Llama Guard 3 and Prompt Guard released simultaneously; improved refusal on newer jailbreak patterns.
• Research Insight: Demonstrates that high-quality post-training data can yield substantial gains without any pretraining compute.

Architecture Diagram

Community Perspective

Llama 3.3 illustrated a key principle increasingly understood in the field: post-training data quality can deliver disproportionate gains relative to compute. By releasing a single improved 70B model, Meta offered organisations a free upgrade path from 3.1 70B with no infrastructure changes. The model quickly became the default recommendation for cost-sensitive deployments, with benchmarks showing it frequently matching or exceeding Llama 3.1 405B on reasoning and coding tasks at a fraction of the serving cost. It also sparked renewed interest in post-training research as an alternative to scaling pretraining compute.

Model Variants

🔴 Llama 4 — April 2025

Summary

• Architecture Paradigm Shift: Native early-fusion multimodal Mixture-of-Experts (MoE); abandons the cross-attention adapter approach of Llama 3.2.
• Early Fusion: Images and text tokens processed together from the first layer; no separate vision encoder; unified tokenisation of visual and language tokens.
• MoE FFN: Sparse expert routing replaces the dense SwiGLU FFN; only a subset of experts activated per token, dramatically reducing active parameters vs total parameters.
• iRoPE Positional Encoding: “interleaved RoPE” — alternating attention layers have no positional encoding (NoPE / infinite context) and layers with standard RoPE; enables extrapolation to very long sequences.
• Scout (109B total, 17B active): 16 experts; 10M token context window; single H100 deployment target; vision + text.
• Maverick (400B total, 17B active): 128 experts per token routing; 1M context; full multimodal; strong benchmark performance; requires ~8× H100.
• Behemoth (~2T total, 288B active): 16 experts; training/teacher model; used for distillation; 16K+ context; not released in April 2025.
• MetaP Hyperparameter Tuning: New technique for systematic hyperparameter transfer across model scales; stabilises training of very large MoE models.
• Training Scale: 30T+ tokens on multimodal data; largest pretraining budget in the Llama series.
• Omni Capability: Scout and Maverick handle image, video, and text inputs; text-only output at release; voice planned.
• Benchmark: Maverick achieves 80.5 on MMLU-Pro and competitive results on GPQA Diamond; Scout’s 10M context enables retrieval over entire codebases.

Architecture Diagram

Community Perspective

Llama 4 represents the most architecturally ambitious Llama release to date. The shift from cross-attention vision adapters to native early-fusion MoE aligns Meta’s approach with GPT-4o and Gemini Ultra in treating multimodality as a first-class concern rather than an add-on. Scout’s 10M-token context window is unprecedented in an open-weight model and opens entirely new application categories — whole-repository code analysis, book-length summarisation, and long-horizon agentic tasks. The iRoPE position encoding scheme attracted significant academic interest as a practical solution to the context-length extrapolation problem. The MetaP hyperparameter transfer technique addresses one of the key engineering pain points in training very large MoE models reliably.

Model Variants

📚 References

Technical Papers

Official Blog Posts

GitHub Repositories

Cited Techniques

_{This document covers the Llama model family from Llama 1 (Feb 2023) through Llama 4 (Apr 2025).
All benchmark figures are reported as published; instruct vs base distinctions apply where noted.
Maintained for educational and research reference purposes.}

↑ Back to top