Liam Fedus reflects on Switch Transformers — 1.6T-param / 2048-expert MoE was ahead on perplexity, behind on reasoning

TL;DR

Liam Fedus (co-author of Switch Transformer, now at Periodic Labs) posts a retrospective quote-tweet on @jietang’s scaling-law thread. He recalls that in 2020 Switch Transformer routed each token to 1 of 2048 experts — <3B activated parameters but 1.6T total, a scale comparable to today’s frontier MoEs. The 1.6T model beat T5 on C4 perplexity at much lower compute and set a TriviaQA SOTA, but was “dumb as bricks” on SuperGLUE-style reasoning. The stated lesson: the compute-optimal tokens-per-parameter ratio is highly task-dependent, and — invoking Shazeer — FLOPs are intelligence while parameters are knowledge.

Key claims

Switch Transformer (2020) routed each token to 1 of 2048 experts, with <3B activated parameters and 1.6T total parameters [tweet body].
The 1.6T Switch Transformer beat T5-family models on C4 perplexity using far less compute and set a new SOTA on TriviaQA [tweet body].
The same 1.6T model was very weak on reasoning benchmarks like SuperGLUE despite its perplexity and knowledge-QA wins [tweet body].
Fedus’s stated takeaway: the compute-optimal tokens-per-parameter ratio is task-dependent — knowledge tasks reward parameters, reasoning tasks reward FLOPs — captured in the Shazeer aphorism “FLOPs were intelligence; parameters were knowledge” [tweet body].

Method

Retrospective tweet, not a paper — no new experiments. Fedus is characterizing a 2020 result (Switch Transformer, top-1 routing over 2048 experts) as a historical precedent for the ultra-sparse MoE frontier of 2025–2026 (Kimi K3 at 16-of-896, Qwen3-Next at 512 experts, Sand.ai MAGI-2 at 3,072 head-local expert units per layer). Attached image is a plot from the Switch Transformer paper; the tweet does not restate its axes.

Results

The historical numbers Fedus quotes: <3B activated / 1.6T total parameters; better C4 perplexity than T5 at lower FLOPs; SOTA on TriviaQA; weak on SuperGLUE. No new metrics.

Why it’s interesting

Two hooks for the current wiki. First, it plants a flag on the knowledge vs reasoning split as a scaling-laws problem: several 2026 papers filed here — How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size on how to allocate tokens across steps and batch, Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design on the attention-vs-experts FLOPs split, and Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies on vocabulary size — all argue that a single Chinchilla-style tokens-per-parameter number understates the design space; Fedus is making the same point retrospectively about Switch Transformer. Second, the “1 of 2048” configuration is a factual anchor for MoE Routing Design: it predates and out-sparsifies almost every filed 2026 MoE except Sand.ai MAGI-2’s Head Parallel routing (MAGI-2 Preview: Scaling Video Generation Models Efficiently, 12 heads × 256 experts) and Kimi K3’s 16-of-896 (Kimi K3 — Open Frontier Intelligence (2.8T MoE with KDA + AttnRes)) — suggesting the current frontier’s return to ultra-sparse MoE is partly a re-litigation of a 2020 bet that lost on reasoning at the time.