The MoE 101 Guide: From Theory to Production (Daria Soboleva, Cerebras)

TL;DR

A five-part practitioner’s guide to Mixture-of-Experts pretraining and inference by Daria Soboleva (Head Research Scientist, Cerebras) and Quentin Anthony. The series spans the MoE design space end-to-end: why sparse models scale better than dense at fixed compute (Fundamentals), what routing strategies actually work in production (Router Wars), how to debug a dead MoE training run from scratch on top of modded-nanogpt (Debugging Dead MoE Models), wafer-scale and multi-GPU scaling tradeoffs (MoE at Scale), and the FLOP / memory / KV-cache arithmetic behind names like “8x7B” (MoE Math Demystified). It’s the most explicitly engineering-flavored MoE resource the research team has filed — every section names a failure mode and the fix.

Key claims

More experts give roughly logarithmic loss improvement at fixed compute; the series quotes a 32-expert configuration as delivering ~5% lower loss at the same compute budget or ~62% fewer FLOPs to reach the same loss as a Chinchilla-optimal dense baseline (~3× faster training) [Fundamentals, Fig. 1].
Behind the marketing names, every production MoE today uses some variant of learned routing with auxiliary load-balancing loss; “shared experts”, “capacity factors”, “adaptive aux loss”, “expert bias” are engineering tricks layered on top of methods from 2017–2022 [Router Wars, Table 1].
DeepSeek-V3’s routing is vanilla learned routing with sequence-level aux loss plus utilization tricks; Qwen3’s “breakthrough” is the same learned routing with the aux loss computed on the global batch (relaxes balance, increases specialization) [Router Wars, §“Routing Landscape”].
The three fundamental routing approaches map cleanly onto a utilization-vs-specialization tradeoff: hash routing gives perfect per-layer load balance but learns overlapping representations; learned routing gives clean expert specialization in middle layers but collapses in early/late layers; Sinkhorn routing recovers hash-level balance with learned-routing-level performance but limits how much experts can differentiate themselves [Router Wars, Fig. 2–4].
Sinkhorn has a known but rarely-stated implementation gotcha: its iterative normalization detaches gradients, so the router never learns to route — only to load-balance. Fix: use Sinkhorn weights to select experts but compute the mixing weights from the original (non-detached) logits [Router Wars, §“Sinkhorn Routing”].
Perfect load balancing can still leave the router useless: in part 3 the authors show a hands-on example where load balancing looks fine but the router fails to learn meaningful token→expert assignments, and walk through the debugging step-by-step [Debugging Dead MoE Models, §“MoE routing is tricky”].
The single-GPU MoE wall is memory-bound (weights + KV-cache); the multi-GPU MoE wall is communication-bound (all-to-all for expert parallelism is bandwidth-bottlenecked and does not amortize with batch size, unlike attention compute) [MoE Math Demystified, §“Two walls”].
Cerebras’s wafer-scale architecture is presented as compute-bound rather than memory- or bandwidth-bound: each layer is mapped to a physical wafer region with weights and KV-cache resident in on-chip SRAM, removing the all-to-all bottleneck and enabling MoE training without expert parallelism [MoE Math Demystified, §“Cerebras hardware”].
A 4-expert MoE trains fine; scaling the same code to 64 experts produces an OOM, and the fix is non-obvious — the Debug post is built around walking through this specific failure [Debugging Dead MoE Models, §“Scale”].
The complete training recipe is pedagogically open-sourced on top of modded-nanogpt: train_gpt_moe.py contains the full MoE implementation (each expert is a copy-pasted GPT-2 FFN with two linear layers around a nonlinearity), and train_gpt.py is the dense baseline for readers who want to add MoE primitives step-by-step [Debugging Dead MoE Models, §“Implementation”].

Method

The series is structured as a five-part blog walkthrough, not a research paper. Each part covers one rung of the MoE stack:

MoE Fundamentals: Why Sparse Models Are the Future of AI (/blog/moe-guide-why-moe) — first-principles motivation: dense models hit a compute-vs-parameter scaling wall (Hoffmann et al., 2022); MoE breaks it by conditionally activating a subset of parameters per token. Defines the basic MoE layer: replace one feedforward block with N experts and a router. Quantifies the scaling claim with a curve of loss vs expert count at fixed compute.
Router Wars: Which MoE Routing Strategy Actually Works (/blog/moe-guide-router) — head-to-head of hash routing (Roller et al., 2021), learned routing with aux loss (Shazeer et al., 2017), and Sinkhorn routing (Clark et al., 2022). Reports a 16-expert experiment where learned routing gives ~3× the loss improvement of hash routing, and traces why via expert-representation PCA visualizations. Documents the Sinkhorn-detaches-gradients trap.
Debugging Dead MoE Models: A Step-by-Step Guide (/blog/moe-guide-debug) — hands-on implementation atop Keller Jordan et al.’s modded-nanogpt speedrun codebase. Each expert is a copy-pasted GPT-2 FFN (train_gpt_moe.py L184-196), and the post walks through (a) the naive batched expert-mixing implementation (correct but wasteful), (b) what routing failures look like in practice (e.g. load balancing is fine but the model is dense-equivalent), and (c) the OOM that hits when you go from 4 to 64 experts.
MoE at Scale: Making Sparse Models Fast on Real Hardware — co-authored with Quentin Anthony (Zyphra); diagnoses the 64-expert OOM as the symptom of the all-to-all communication wall and walks through the systems-level fixes (referenced in part 5).
MoE Math Demystified: What Does 8x7B Actually Mean? (/blog/moe-guide-calculator) — closed-form FLOP and memory arithmetic for MoE inference. Walks through model-weight memory and KV-cache memory under standard modern transformer assumptions (RoPE, SwiGLU, untied embeddings, multi-head attention, learned routing). Then contrasts the two walls — memory-bound single-GPU, communication-bound multi-GPU — against Cerebras’s wafer-scale pipeline alternative.

The pedagogical setup is unusual: a working, runnable codebase (modded-nanogpt + a train_gpt_moe.py overlay) and an explicit “you should break things and try weird settings” framing.

Results

This is a tutorial series, not a benchmarked artifact, so headline numbers are scaffolded across the parts rather than gathered in one table:

32 experts at fixed compute: ~5% better loss vs Chinchilla-optimal dense, or ~62% fewer FLOPs to match a target loss (~3× faster training) [Fundamentals, Fig. 1].
16 experts, hash routing: ~1.5% loss improvement over Chinchilla-optimal dense, barely scaling with more experts [Router Wars, Fig. 2].
16 experts, learned routing: ~4% loss improvement, ~3× the gain of hash routing at the same scale [Router Wars, Fig. 2].
Learned routing collapses in early and late layers, funneling tokens to 1–2 experts (the standard motivation for DeepSeek-V3 / Qwen2 shared experts) [Router Wars, Fig. 3b].
Sinkhorn routing achieves hash-level per-layer load balance with learned-routing-level performance — but limits expert specialization (Fig. 4c) and has not been adopted at production scale due to implementation complexity, particularly the gradient-detachment issue [Router Wars, Fig. 2–4].

Why it’s interesting

The wiki’s [[moe-routing-design]] cluster is dense with 2026 papers that propose specific fixes to the standard MoE routing pipeline — [[2604.00801-routing-free-mixture-of-experts]] eliminates the router, [[2603.18297-path-constrained-mixture-of-experts]] shares it across layers, [[2501.11873-demons-detail-implementing-load-balancing]] rescopes the aux loss to global-batch, [[blog-jianlinsu-quantile-balancing-moe-load-balancing-2026-02]] removes the aux-loss coefficient — but the cluster has been missing a plain-language map of what the standard pipeline actually is and which of its design choices are load-bearing in production. This series is that map: it names hash / learned / Sinkhorn as the three fundamental approaches, points at the specific failure modes (router collapse in edge layers, Sinkhorn’s gradient detachment, the 4→64 expert OOM), and reframes recent “breakthroughs” (DeepSeek-V3, Qwen3) as engineering layered on 2017-era methods. It complements [[2505.11432-megascale-moe-large-scale-communication]] (production MoE systems recipe from ByteDance Seed) by working at the algorithm/ debugging layer rather than the parallelism-strategy layer, and it complements the Cursor / DeepSeek-V3 inference-side posts ([[blog-cursor-better-moe-model-inference-warp-2026-04]], [[blog-pytorch-enabling-up-41-faster-pre-2026-03]]) by giving the training-side counterpart.