🟣 Qwen — Model Architecture Across Generations

From a competitive Chinese LLM to a global open-weight powerhouse — tracing 5 generations of architecture evolution.

📑 Table of Contents

• Executive Summary
• Version Release Timeline
• Cross-Version Benchmark Comparison
• Master Architecture Diagram
• Qwen 1 (September 2023)
• Qwen 1.5 (February 2024)
• Qwen 2 (July 2024)
• Qwen 2.5 (September 2024)
• Qwen 3 (April 2025)
• References

📋 Executive Summary

This document covers five generations of the Qwen large language model family developed by Alibaba Cloud’s Qwen Team:

• Qwen 1 — The foundation: SwiGLU, RoPE, RMSNorm, 151K BPE vocab, 3T tokens
• Qwen 1.5 — Scale-out: 6 dense sizes (0.5B–72B) + first MoE model, 32K context, HF transformers native
• Qwen 2 — Architecture leap: GQA, DCA+YARN for 128K context, MoE 57B-A14B, 7T tokens, 30 languages
• Qwen 2.5 — Data scaling: 18T tokens, new 3B/14B/32B sizes, structured output, 8K generation
• Qwen 3 — Reasoning era: Hybrid think/non-think modes, 36T tokens, 119 languages, 4-stage RL

📝 Note: The Qwen family also includes specialized variants — Qwen-VL (vision-language), Qwen-Audio, Qwen-Coder, and Qwen-Math — which are documented separately. This document focuses on the core LLM architecture.

📅 Version Release Timeline

| Version | Release Date | Paper / Blog | Flagship Size | Training Tokens | Context Length | Headline Feature | |:-------:|:----------:|:------------|:------------:|:--------------:|:------------:|:----------------| | | Sep 28, 2023 | [arXiv:2309.16609](https://arxiv.org/abs/2309.16609) | 72B | 3T | 8K (32K ext.) | First competitive Alibaba LLM | | | Feb 4, 2024 | [Blog](https://qwenlm.github.io/blog/qwen1.5/) | 72B + MoE-A2.7B | ~3T | 32K | HF-native + first MoE | | | Jul 15, 2024 | [arXiv:2407.10671](https://arxiv.org/abs/2407.10671) | 72B + MoE-57B-A14B | 7T | 128K | GQA + DCA/YARN + MoE | | | Sep 19, 2024 | [arXiv:2412.15115](https://arxiv.org/abs/2412.15115) | 72B | 18T | 128K | Data scaling + structured output | | | Apr 29, 2025 | [arXiv:2505.09388](https://arxiv.org/abs/2505.09388) | 235B-A22B | 36T | 128K | Hybrid thinking + 119 languages |

📊 Cross-Version Benchmark Comparison

All numbers are for the flagship base model of each generation (largest dense model). Sources: official technical papers.

_{*Qwen 3 base model benchmarks are for the largest dense model (Qwen3-32B-Base) from the Qwen 3 technical report pre-training performance curves. Exact numbers may vary.}

🏗️ Master Architecture Diagram

This diagram shows the core Transformer decoder architecture shared across all Qwen versions, with color-coded annotations indicating which generation introduced each component.

🟣 Qwen 1 — September 2023

Summary

• First large-scale open-weight LLM from Alibaba Cloud, establishing the Qwen brand in the competitive Chinese LLM landscape alongside Yi, Baichuan, and ChatGLM
• Built on a standard decoder-only Transformer architecture with causal attention masks, following the paradigm set by GPT and LLaMA
• Adopted SwiGLU (Swish-Gated Linear Unit) activation in FFN layers — borrowed from PaLM/LLaMA, replacing the conventional ReLU/GELU, providing smoother gradients and better performance
• Used Rotary Positional Embeddings (RoPE) for encoding position information, enabling better length generalization than absolute positional embeddings
• Applied RMSNorm with pre-normalization for improved training stability — a technique popularized by LLaMA and now standard practice
• Introduced QKV bias in attention layers — an uncommon design choice at the time that later papers showed helps with RoPE-based length extrapolation
• Trained on 3 trillion tokens of multilingual data (primarily English and Chinese) using a byte-level BPE tokenizer with 151,646 vocabulary — one of the largest vocabs at the time, designed for strong multilingual compression
• Released in two sizes: 7B and 72B parameters, using standard Multi-Head Attention (MHA) across all layers
• Included specialized variants: Qwen-Chat (aligned via SFT + RLHF), Code-Qwen, and Math-Qwen — demonstrating a full-stack approach from day one
• Supported 8K context natively with NTK-aware interpolation for extending to 32K — an industry technique for RoPE-based dynamic length extrapolation

Architecture Diagram — Qwen 1

Community Perspective

• Received strong reception in the Chinese AI community as a capable alternative to LLaMA for Chinese-English tasks
• The 72B model demonstrated that Chinese AI labs could produce GPT-3.5-competitive models
• The large 151K vocabulary was praised for efficient multilingual tokenization — many competitors used smaller vocabs
• Tool-use and code interpreter capabilities in Qwen-Chat were ahead of most open-source alternatives at launch
• Some concern about training data transparency compared to fully open models like LLaMA

Model Variants

🟪 Qwen 1.5 — February 2024

Summary

• Incremental refinement rather than architecture overhaul — focused on improving base model quality and massively expanding the developer experience
• Expanded to 8 dense model sizes: 0.5B, 1.8B, 4B, 7B, 14B, 32B, 72B, and 110B — the 110B was the first 100B+ model in the Qwen family
• Introduced the first Qwen MoE model: Qwen1.5-MoE-A2.7B with 14.3B total parameters, 2.7B activated — achieving 7B-class performance at 1/3 the compute
• Architecture identical to Qwen 1 (MHA, SwiGLU, RoPE, RMSNorm, QKV bias) — improvements came from better data, longer training, and alignment techniques
• Uniformly 32K context across all model sizes — up from the 8K default of Qwen 1 — achieved through RoPE frequency adjustments
• Native Hugging Face transformers integration — no more trust_remote_code=True, making deployment frictionless with transformers>=4.37.0
• Alignment enhanced with DPO (Direct Preference Optimization) and PPO (Proximal Policy Optimization) — producing significantly better chat models
• Multilingual capabilities expanded to ~12 languages with structured evaluation on Arabic, Spanish, French, Japanese, Korean, Thai, Vietnamese, and more
• MoE architecture used 64 fine-grained experts with 4 shared + 60 routed (4 activated per token) — inspired by DeepSeek-MoE’s fine-grained expert design
• Upcycling initialization for MoE: started from Qwen-1.8B weights, transformed into MoE structure with randomized initialization for diversity — reduced training cost by 75% vs. training from scratch

Architecture Diagram — Qwen 1.5 MoE

Community Perspective

• Widely praised for the developer experience overhaul — HF-native support was a game-changer for adoption
• The MoE model (A2.7B) surprised many by matching Mistral-7B and Qwen1.5-7B while being 1/3 the activated size
• The 110B model was seen as a statement of scale ambition, though it didn’t get as much adoption as the 72B
• Strong reception for the expanded size lineup — the 0.5B and 1.8B models enabled edge/mobile deployment
• Criticism: the architecture was largely unchanged from Qwen 1, so improvements felt incremental

Model Variants

🔵 Qwen 2 — July 2024

Summary

• Major architecture upgrade — the most significant changes since Qwen’s inception, introducing multiple new attention and positional mechanisms
• Grouped Query Attention (GQA) replaced MHA across all models — dramatically reducing KV cache memory during inference while maintaining quality
• Dual Chunk Attention (DCA) + YARN enabled 128K context by segmenting long sequences into manageable chunks with rescaled attention weights
• Expanded to 5 model sizes: 0.5B, 1.5B, 7B, 72B (dense) + 57B-A14B (MoE) — the MoE model had 57B total parameters with 14B active per token
• MoE architecture advanced significantly: fine-grained experts with smaller expert size, shared + routing experts (8 shared + 64 routed, 8 activated), and upcycled from Qwen2-7B
• Training data scaled to 7 trillion tokens (from 3T) with dramatically expanded code, math, and multilingual content — supporting ~30 languages
• Smaller models (0.5B, 1.5B) used embedding tying and were trained on 12T and 7T tokens respectively — more tokens per parameter than larger models
• Post-training involved SFT with 500K+ examples followed by both offline DPO and online RLHF with a reward model — the most sophisticated alignment pipeline in the Qwen family at the time
• Online Merging Optimizer was used to mitigate alignment tax — reducing performance degradation from RLHF
• RoPE base frequency increased from 10,000 to 1,000,000 in the long-context training phase — enabling much longer effective sequence lengths

Architecture Diagram — Qwen 2

Official Paper Figure

Needle in a Haystack test results for Qwen2 instruction-tuned models showing capability across 128K context:

_{Source: Qwen2 Technical Report (arXiv:2407.10671), Figure 1}

Community Perspective

• GQA adoption was welcomed as overdue — competitors like LLaMA 2 (70B) had already adopted it for KV cache efficiency
• The 128K context via DCA+YARN was a major selling point, though real-world performance degraded at extreme lengths
• 57B-A14B MoE model showcased that Qwen’s MoE expertise had matured — fine-grained experts were more efficient than Mixtral’s coarse approach
• Qwen2-72B’s competitiveness with LLaMA-3-70B established Qwen as a top-tier global open-weight model — not just a Chinese alternative
• The 7T token dataset with 30 language support marked Qwen’s transition from a bilingual to a truly multilingual model family

Model Variants

🔷 Qwen 2.5 — September 2024

Summary

• Data scaling landmark — pre-training dataset expanded from 7T to 18 trillion tokens, representing one of the largest known training runs for open-weight models
• Architecture identical to Qwen 2 at the model level — same GQA, DCA+YARN, SwiGLU, RoPE, 151K vocab — the improvements were entirely from data quality and scale
• Introduced three new model sizes: 3B (for mobile), 14B and 32B (for production) — filling gaps that the community had been requesting
• Knowledge dramatically improved: MMLU jumped from 84.2 (Qwen 2) to 86.1 (Qwen 2.5) for the 72B base model — a significant gain at the top of the benchmark
• Long text generation breakthrough: models could now generate up to 8K tokens per response (vs. ~1K in Qwen 2) — enabled by post-training on long-form data
• Structured output support added — models reliably produce JSON, tables, and formatted data — a critical feature for production agentic applications
• Post-training evolved to over 1 million SFT samples plus multi-stage RL — incorporating techniques from Qwen2.5-Math and Qwen2.5-Coder specialist models
• Code performance surged thanks to Qwen2.5-Coder integration: LiveCodeBench jumped from 32.2 (Qwen 2) to 55.5 (Qwen 2.5) for the 72B instruct model
• Math equally improved via Qwen2.5-Math technology: MATH benchmark went from 69.0 to 83.1 for the 72B instruct model
• Qwen2.5-72B demonstrated competitive with or superior to LLaMA-3.1-405B on many benchmarks despite being ~5x smaller

Architecture Diagram — Qwen 2.5

Official Paper Figures

_{Source: Qwen2.5 Blog — 72B-Instruct benchmark comparison}

_{Source: Qwen2.5 Blog — Model specifications overview}

Community Perspective

• The 18T token dataset was a headline number — more than Llama 3’s 15T and signaling massive investment in data curation
• Qwen2.5-32B outperforming Qwen2-72B demonstrated that data quality matters more than model size at this scale
• The structured output capabilities made Qwen 2.5 the go-to choice for many agentic/tool-use applications
• The code and math improvements were directly attributable to specialist model techniques — showing the value of the Qwen ecosystem approach
• Community noted that same-architecture improvements have diminishing returns — expectations built for an architecture refresh in Qwen 3

Model Variants

API-only models: Qwen2.5-Turbo (MoE) and Qwen2.5-Plus (MoE) were also released through Alibaba Cloud Model Studio.

🟢 Qwen 3 — April 2025

Summary

• Paradigm shift: introduced hybrid thinking modes — models can seamlessly switch between “Thinking” mode (step-by-step reasoning, like o1/QwQ) and “Non-Thinking” mode (fast direct responses) within a single model
• Massive scale-up: flagship Qwen3-235B-A22B has 235B total parameters with 22B activated — the largest Qwen MoE to date, plus Qwen3-30B-A3B as an efficient smaller MoE
• Released 8 models total: 6 dense (0.6B, 1.7B, 4B, 8B, 14B, 32B) + 2 MoE (30B-A3B, 235B-A22B) — all open-weighted under Apache 2.0
• Training data nearly doubled to 36 trillion tokens covering 119 languages and dialects — a dramatic jump from Qwen 2.5’s 29 languages
• Used Qwen2.5-VL to extract text from PDF-like documents and Qwen2.5-Math/Coder to generate synthetic training data — the “models training models” paradigm
• 4-stage post-training pipeline: (1) Long CoT cold start, (2) Reasoning-based RL with rule-based rewards, (3) Thinking mode fusion — blending thinking and non-thinking data, (4) General RL across 20+ domains
• Thinking budget mechanism allows users to control how much reasoning compute to allocate per query — enabling smooth latency vs. quality tradeoffs
• Three-stage pre-training: S1 (30T+ tokens, 4K context) → S2 (5T tokens, knowledge-intensive STEM/code/reasoning) → S3 (high-quality long-context data, extend to 32K)
• Dense models match performance of Qwen 2.5 models 2× their size: e.g., Qwen3-8B ≈ Qwen2.5-14B, Qwen3-4B ≈ Qwen2.5-7B
• MoE models achieve similar performance to Qwen 2.5 dense models at only ~10% of active parameters — Qwen3-30B-A3B outperforms QwQ-32B with 10× fewer active params

Architecture Diagram — Qwen 3

Official Paper Figures

_{Source: Qwen3 Blog — Qwen3-235B-A22B benchmark comparison against DeepSeek-R1, o1, o3-mini, Grok-3, Gemini-2.5-Pro}

_{Source: Qwen3 Blog — Qwen3-30B-A3B outperforming QwQ-32B with 10× fewer active parameters}

_{Source: Qwen3 Blog — Thinking budget mechanism showing smooth performance scaling with compute}

_{Source: Qwen3 Blog — 4-stage post-training pipeline overview}

Community Perspective

• The hybrid thinking mode was seen as a direct answer to OpenAI’s o1/o3 and DeepSeek-R1 — but more elegant because it’s a single model rather than separate chat vs. reasoning models
• Qwen3-30B-A3B outperforming QwQ-32B was a landmark result — demonstrating extreme MoE efficiency
• 119 language support (from 29) was a massive expansion — making Qwen 3 one of the most multilingual open-weight models available
• The 4-stage post-training pipeline was praised as a well-engineered approach to combining reasoning and general capabilities
• Open-source community quickly adopted the /think and /no_think toggles as an intuitive user interface for controlling reasoning depth

Model Variants — Dense

Model Variants — MoE

📚 References

Technical Papers

Official Blog Posts

GitHub & Model Repositories

Cited Techniques

_{Built with data from official Qwen technical papers and blog posts. All benchmark numbers sourced directly from the referenced publications.}

← Back to Index