The Big LLM Architecture Comparison

TL;DR

Long-form architecture walkthrough of the 2024-2026 generation of open-weight decoder LLMs, organized model-by-model from DeepSeek V3 / R1 (Dec 2024) through GLM-5 and Gemma 4 (Apr 2026). Each section isolates one or two architectural deltas relative to a GPT-2 / Llama-3 baseline — MLA vs GQA, MoE topology choices (shared vs no shared expert, fine- vs coarse-grained), normalization placement (Pre/Post/sandwich, QK-Norm), sliding-window attention ratios, NoPE — and walks through the reasoning the model authors gave (or didn’t give) for each choice. The post is continuously updated (last revision Apr 2026 to add Gemma 4); a companion video and Raschka’s LLM Architecture Gallery reference site share material.

Key claims

Despite seven years of LLM development, the structural skeleton from GPT-2 (2019) to DeepSeek V3 / Llama 4 (2024-2025) is largely unchanged — positional embeddings shifted from absolute to RoPE, MHA largely gave way to GQA, GELU was replaced by SwiGLU, but the block-level architecture is recognizably the same [§Intro].
DeepSeek V3’s two distinguishing architectural choices over Llama-style baselines are Multi-Head Latent Attention (MLA) — which compresses K/V into a lower-dim latent before caching and re-expands at use time — and a fine-grained MoE with 256 experts per layer of which 9 (1 shared + 8 routed) activate per token, yielding 37B active out of 671B total [§1.1, §1.2, §1.3].
The DeepSeek-V2 ablation reproduced in the post claims MLA outperforms MHA on modeling quality and beats GQA, whereas GQA underperforms MHA — likely why DeepSeek chose MLA over the more common GQA [§1.1, Fig. 4].
OLMo 2’s two architectural deltas relative to Llama 3 are (a) a Post-Norm-flavored RMSNorm placement (inside the residual but after attention/FFN, not before) and (b) QK-Norm — an RMSNorm applied to queries and keys before RoPE; both are claimed to improve training stability, with the ablation that disentangles their individual contributions not shown [§2.1, §2.2, §2.3].
Gemma 3 uses a 5:1 ratio of sliding-window (1024-token) to full-attention layers — up from Gemma 2’s 1:1 with a 4096 window — and pairs RMSNorm both before and after attention/FFN modules (sandwich norm). The team reports minimal modeling-quality impact from sliding-window at this ratio [§3.1, §3.2, §3.3].
Mistral Small 3.1 24B abandons sliding-window attention ("sliding_window": null in the official config) in favor of vanilla GQA — Raschka’s speculation is that sliding-window saves memory but doesn’t reduce inference latency the way standard GQA + FlashAttention does, and Mistral was optimizing for latency [§4].
Llama 4 Maverick (17B active / 400B total) adopts MoE but with a different topology from DeepSeek V3 — fewer, larger experts (2 active × 8,192 hidden) vs DeepSeek’s many, smaller experts (9 active × 2,048 hidden), and MoE alternated with dense FFN every other block vs DeepSeek’s MoE-in-every-block-after-the-first-three [§5].
Qwen3 ships both dense (0.6B–32B) and MoE (30B-A3B, 235B-A22B) variants of the same family. The Qwen3 MoE variants removed the shared expert that DeepSeek V3 and earlier Qwen2.5-MoE used; Junyang Lin’s quoted explanation is that they did not see significant enough improvement to justify the inference-side optimization cost when going from 2 to 8 experts [§6.1, §6.2].
SmolLM3 (3B) is notable for adopting NoPE (No Positional Embeddings) in a subset of its layers and for sharing extensive training process documentation alongside the weights — a HuggingFace pattern the post highlights as rare [§7].
The post is structured to deliberately exclude training data, training algorithms, and benchmark performance, focusing only on the structural deltas of the decoders themselves — an explicit choice on Raschka’s part to control article scope, made at the start of the post [§Intro].

Method

Not a research paper — a curated long-form walkthrough. Each model section follows the same template: scale (total/active params, context, vocab), the one or two architectural deltas worth highlighting, a side-by-side comparison diagram against a prior baseline (often DeepSeek V3, Llama 3, or the model’s own predecessor), and a “Summary” sub-section that compresses the deltas into 2-3 bullets. Where relevant, Raschka cross-references his own from-scratch implementations of the GQA / QK-Norm / RoPE blocks and the model-by-model fact cards in the companion LLM Architecture Gallery (which carries the per-card KV-cache-bytes-per-token and layer-mix numbers the prose article does not).

The post is maintained as a living document — the Apr 2026 revision adds a Gemma 4 section (section 23), and the section numbering grows as new releases land. The substack version carries a narrated abridged video.

Results

No metrics — the article is taxonomic. Notable comparative observations the post pulls from primary-source ablations:

DeepSeek V2 MLA ablation: MLA > MHA > GQA on modeling quality at matched compute [Fig. 4, citing arXiv:2405.04434].
Gemma 3 sliding-window ablation: 5:1 SWA:global with 1024-token window has minimal impact on modeling performance vs. all-global [Fig., citing arXiv:2503.19786].
OLMo 2 normalization ablation: Post-Norm + QK-Norm improves training-loss stability vs Pre-Norm baseline, but the two effects are not separately disentangled [§2.3].
Qwen3 0.6B (the smallest currently-released open-weight model) outperforms Llama-3 1B on throughput and memory at smaller hidden dims and fewer heads, but is slower in tokens/sec generation because it uses more transformer blocks [§6.1].

Why it’s interesting

This is the prose companion to LLM Architecture Gallery — where the Gallery gives uniform-schema fact cards over ~40 open models, this article gives the narrative on why each architecture team made each choice (and where the public reasoning is silent). For the wiki’s Open foundation-model releases cluster it’s a useful reference for the structural side of model releases — counterpart to the data-and-training-process documentation thread covered by Smol Training Playbook (GPU MODE talk on SmolLM3). The MoE-design discussions (shared expert in DeepSeek V3 vs removed in Qwen3, fine- vs coarse-grained in DeepSeek V3 vs Llama 4 Maverick) connect directly to the MoE Routing Design cluster’s “expert size and count dominate” datapoint from Slicing and Dicing MoEs — Margaret Li announcing a >2000-MoE-LM design-space study and to ERNIE 4.5’s modality-isolated routing in ERNIE 4.5 Technical Report. The attention-variant discussion (MLA / GQA / sliding-window) pairs naturally with How Attention Got So Efficient [GQA / MLA / DSA]‘s explainer.