Skip to content
Bud's Wiki

LLM Architecture Gallery

TL;DR

Section titled “TL;DR”

A continuously-updated, browsable reference of modern LLM decoder architectures, maintained as a side companion to Raschka’s “From GPT-2 to GPT-OSS” and “Big LLM Architecture Comparison” posts. Each model gets a fact card with scale (total/active params), context length, license, release date, decoder type (dense / sparse MoE / hybrid / recurrent), attention variant (MHA / GQA / MQA / MLA / DeltaNet / Mamba-2 / Lightning / sliding-window mixes), explicit layer mix counts, KV-cache bytes per token, and a one-line “key detail” summary. The gallery covers everything from GPT-2 XL (1.5B, 2019) through GLM-5 (744B, 2026-02), Gemma 4 (2026-04), and Nemotron 3 Super (2026-03), and includes a side-by-side architecture diff tool. The original published date is 2024-01-03 but the catalog has been kept current through late-2025 / early-2026 releases.

Key claims

Section titled “Key claims”
  • The gallery tabulates decoder type for each model across four buckets: Dense, Sparse MoE, Sparse hybrid (DeltaNet/Lightning-attention + MLA/GQA), and Hybrid MoE (Mamba-2 + GQA + MoE); each card lists the explicit layer mix (e.g. “61 MLA”, “12 sliding-window + 12 global”, “6 GQA + 23 Mamba-2 + 23 MoE”) [card grid].
  • KV-cache bytes per token at bf16 is reported per card with a qualitative rating (Very low / Low / Moderate / High / Very high), making cross-architecture KV-pressure directly comparable — e.g. DeepSeek V3 671B = 68.6 KiB/token (Low) vs OLMo 2 7B = 512 KiB/token (Very high) vs Kimi Linear 48B = 7.9 KiB/token (Very low) vs Nemotron 3 Nano 30B = 6 KiB/token (Very low) [card grid].
  • Most large open MoE models released in 2025-2026 inherit the DeepSeek V3 template (61 layers, MLA, MTP-1 path, dense prefix, sometimes shared expert) — explicitly noted for DeepSeek V3 (2024-12), DeepSeek R1 (2025-01), Kimi K2 (2025-07, “essentially scales the DeepSeek V3 recipe upward”), GLM-4.5 (2025-07, “DeepSeek’s dense-prefix MoE layout”), Mistral Large 3 (2025-12, “near-clone of DeepSeek V3”), DeepSeek V3.2 (2025-12, adds DSA), Kimi K2.5 (2026-01), and GLM-5 (2026-02, “MLA with DeepSeek Sparse Attention”) [card grid].
  • The “linear-attention hybrid” line is a distinct family: Qwen3 Next (2025-09, 3:1 Gated DeltaNet : Gated Attention), Kimi Linear (2025-10, 3:1 Kimi Delta Attention : MLA, NoPE in MLA layers), Ling 2.5 (2026-02, 7:1 Lightning Attention : MLA), Qwen3.5 (2026-02, Next-style hybrid now mainline, 3:1 Gated DeltaNet : Gated Attention), Nemotron 3 Nano/Super (Mamba-2-dominant with sparse GQA layers) — all in the “Very low” KV-cache bracket (6–30 KiB/token) [card grid].
  • Sliding-window-dominant designs cluster separately from linear-attention hybrids: Gemma 3 (5:1 SWA:global), GPT-OSS 20B/120B (1:1 alternating SWA/global), Step 3.5 Flash (3:1), Mistral Large 3 (no SWA), OLMo 3 (3:1), Xiaomi MiMo-V2-Flash (5:1, unusually small 128-token local window), Gemma 4 (5:1, unified K/V on global layers, p-RoPE on global) [card grid].
  • QK-Norm is now near-ubiquitous in 2025-2026 dense/MoE GQA stacks: Qwen3 family (4B/8B/32B and 235B-A22B), Gemma 3, Gemma 4, OLMo 2, OLMo 3, GLM-4.5/4.7, MiniMax M2/M2.5, Sarvam, Arcee Trinity Large, Step 3.5 Flash — listed as GQA with QK-Norm on each card [card grid].
  • The catalog reports MTP (multi-token prediction) as a standard ingredient in the DeepSeek-lineage MoE training recipe — DeepSeek V3 / R1 / V3.2 use MTP-1; GLM-4.5 uses MTP during training; GLM-4.7 uses “MTP-3-style setup”; Mistral Large 3 inherits MTP; Step 3.5 Flash uses MTP-3 during both training and inference; MiniMax-M2.5 uses three MTP modules; Nemotron 3 Super uses shared-weight MTP for fast inference [card grid].
  • The xLSTM 7B card is the only non-attention, non-SSM entry: “No self-attention; mLSTM recurrent layers with matrix memory”, 32 mLSTM layers, “No cache” (0 B KV cache), “No explicit limit” context — a recurrent baseline included for contrast [xLSTM card].
  • Smaller / on-device models with conservative recipes (Llama 3.2, Phi-4 14B, Nanbeige 4.1, Tiny Aya 3.35B) keep plain GQA + RoPE + RMSNorm with no QK-Norm, no SWA, no MLA — making them useful “Llama-3-style” baselines for ablation comparisons [card grid].

Method

Section titled “Method”

The artifact is a single-page reference site with model cards rendered in a grid plus a side-by-side architecture-diff tool (“Model A / Model B” selectors on each card). Each card carries: model name + size, scale (total / active params, with active-percentage for MoE), context (tokens) at deploy time, vocabulary size where notable, license, release date (ISO), decoder type, attention spec, explicit layer-mix counts, KV-cache bytes per token at bf16 with a low/moderate/high/very-high band, and a one-line “key detail” callout. There is no per-card prose narrative — the value is in the uniform schema across ~40 models, which makes cross-model architectural choices directly comparable. The page is maintained as an issue-tracked living document with corrections via GitHub. Distinct from Raschka’s longer essay-form posts (e.g. Big LLM Architecture Comparison, From GPT-2 to GPT-OSS), which the gallery cross-references but does not duplicate.

Results

Section titled “Results”

Not a research artifact — no metrics. The value is taxonomic. As a snapshot of what’s in the gallery at filing time:

  • Dense models: GPT-2 XL (1.5B / 2019), Llama 3 8B, Llama 3.2 1B/3B, OLMo 2 7B, Gemma 3 27B / 270M, Mistral Small 3.1 24B, Qwen3 0.6B/4B/8B/32B, SmolLM3 3B, Phi-4 14B, Nanbeige 4.1 3B, Tiny Aya 3.35B, Gemma 4 31B, OLMo 3 7B/32B.
  • Sparse MoE: DeepSeek V3 / R1 / V3.2 (671B), Llama 4 Maverick (400B), Qwen3 30B-A3B / 235B-A22B / Coder Flash 30B-A3B, Kimi K2 (1T) / K2.5 (1T), GLM-4.5 (355B), GLM-4.5-Air (106B), GLM-4.7 (355B), GLM-5 (744B), GPT-OSS 20B / 120B, Grok 2.5 (270B), MiniMax M2 (230B) / M2.5 (230B), Mistral Large 3 (673B), Sarvam 30B / 105B, Arcee Trinity Large (400B), Step 3.5 Flash (196B), Xiaomi MiMo-V2-Flash (309B), Gemma 4 26B-A4B, INTELLECT-3 (106B).
  • Sparse hybrid (linear attention + full attention): Qwen3 Next (80B-A3B), Kimi Linear (48B-A3B), Ling 2.5 (1T), Qwen3.5 (397B).
  • Hybrid MoE (Mamba-2 + attention + MoE): Nemotron 3 Nano (30B-A3B), Nemotron 3 Super (120B-A12B).
  • Recurrent: xLSTM (7B).

Why it’s interesting

Section titled “Why it’s interesting”

This is the single most useful index page for the wiki’s open-foundation-releases cluster — many of the models the wiki already files individually (Step 3.5 Flash, Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters, MiniMax-M2.5, Trinity Large: An Open 400B Sparse MoE Model, Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning, GLM-5: from Vibe Coding to Agentic Engineering) get cross-referenced into one table with comparable attention/layer-mix/KV-cache columns. It also concretizes the “DeepSeek architecture lineage” candidate concept that the wiki has been tracking (mHC: Manifold-Constrained Hyper-Connections is the lone filed entry against deepseek-architecture (1/3)) — the gallery makes explicit that DeepSeek V3 / R1 / V3.2 / Kimi K2 / K2.5 / GLM-4.5 / Mistral Large 3 / GLM-5 all share the 61-MLA / MTP / dense-prefix / shared-expert template, with concrete numerical deltas on each card. And it pairs naturally with How Attention Got So Efficient [GQA / MLA / DSA] (GQA / MLA / DSA explainer) and Beyond Softmax: The Future of Attention Mechanisms (linear attention) as the reference catalog those videos describe in principle.

See also

Section titled “See also”