The Best Optimizer Depends on Batch Size

TL;DR

Dang, Wen & Malladi challenge a load-bearing assumption of modern hyperparameter-scaling practice: that a single “best” optimizer, correctly tuned, remains best across batch sizes. They show two things across LM pretraining sweeps: (1) no principled scaling rule for Muon transfers hyperparameters consistently across training settings, and (2) the optimizer that wins at one batch size is not the optimizer that wins at another, even after extensive per-batch-size tuning. The implication is that single-batch-size optimizer benchmarks (the current standard, including nanoGPT-speedrun-style arenas) systematically mis-rank optimizers for production training, and that batch-size-invariant hyperparameter transfer for Muon in particular is still open.

Key claims

No principled scaling rule for Muon works consistently across training settings — hyperparameters that transfer cleanly in one regime fail in another [Abstract].
The identity of the best optimizer for language-model pretraining changes with batch size, and this is after extensive per-batch-size hyperparameter tuning (not an artifact of under-tuning the loser) [Abstract].
Scaling rules that promise to preserve relative optimizer ordering as batch size and gradient noise vary — the implicit contract behind µP-style LR transfer, Hyperball weight-sphere retraction, and $\sqrt{B/B_0}$ batch scaling — do not hold for Muon [Abstract].

Method

Compare adaptive optimizers (Muon and baselines) across multiple batch sizes on LM pretraining, with per-(optimizer, batch size) hyperparameter tuning rather than inheriting hyperparameters across settings. Evaluate whether a single scaling rule for Muon can preserve its position in the optimizer ranking across batch sizes, and whether the winner at one batch size remains the winner at another. (Full experimental details — model sizes, token budgets, exact baseline set, scaling rules tested — are in the paper body; this page is a stub pending a closer read of the full PDF.)

Results

Headline claims from the abstract are the two negative results above. Specific numerical breakdowns of which optimizer wins at which batch size, and which scaling rules were tested and failed for Muon, require reading the PDF — not yet pulled into this page.

Why it’s interesting

Directly contradicts the implicit premise of several entries on Hyperparameter scaling laws: the Fitting paradigm (How to Set the Learning Rate for Large-Scale Pre-training?), the Hyperball / AdamH weight-sphere retraction (Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization, Delphi: Scaling Laws That Extrapolate 300× Past the Fit), and the “Muon is half-aligned with µP” fix (Controlled LLM Training on Spectral Sphere, How Muon Lost Its Geometry) all treat Muon’s scaling behavior as fixable with the right geometric constraint on the update or weight spectrum. This paper’s claim that no principled scaling rule works consistently for Muon reframes the problem: the batch-size axis may be the one that geometric constraints on weights/updates can’t absorb. Also directly relevant to the “batch size as a free hyperparameter” open question on the concept page — The Design Space of Tri-Modal Masked Diffusion Models‘s SDE-reparameterization result is derived for AdamW, and How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size‘s three-term L(N, T, B) fit is also AdamW-centric; this paper is evidence that Muon’s $B$ -dependence is qualitatively different.