Pretraining Data Scale Reverses the Muon-AdamW Fine-Tuning Gap

ListenWhen the Optimizer Winner Flips19:50

Show notes

Muon’s advantage during pretraining does not necessarily survive fine-tuning: Li and colleagues report a reversal toward AdamW as pretraining data grows, with larger models crossing over at smaller token-per-parameter ratios. Maya and Ethan unpack the proposed margin-to-spread explanation, distinguish the language-model evidence from the one-step associative-memory theorem, and discuss what an optimizer comparison at Luma should measure. The supplied abstract and wiki summary establish the trend, but do not provide exact crossover budgets, effect sizes, or detailed training recipes. ## Sources - Pretraining Data Scale Reverses the Muon-AdamW Fine-Tuning Gap

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

Li, Gai, Risteski, Springer & Raghunathan (CMU) report a crossover in the Muon-vs-AdamW ranking as pretraining data budget grows: Muon wins on pretraining loss and on fine-tuning performance at small-to-moderate token-per-parameter (TPP) ratios, but beyond a sufficiently large pretraining data budget AdamW overtakes Muon after fine-tuning, and the crossover TPP decreases with model size. They propose a mechanistic explanation — the margin-to-spread ratio of the output logits — which tracks the crossover in both LMs and a provable linear-associative-memory toy model. The result directly complicates the “Muon is a strict compute multiplier over AdamW” narrative that the pretraining-speedrun literature has been pushing, and is the first filed paper that argues Muon’s advantage inverts at scale rather than merely narrows.

Key claims

On language-model pretraining followed by fine-tuning, Muon beats AdamW at small pretraining data budgets but AdamW beats Muon at large pretraining data budgets — a surprising sign-flip, not just a narrowing gap [Abstract].
The crossover TPP at which AdamW overtakes Muon decreases monotonically with model size — larger models flip to AdamW-favored earlier in TPP terms [Abstract].
The fine-tuning gap is explained by viewing fine-tuning as a perturbation of the output logits and measuring the margin-to-spread ratio: the distance from the correct logit to the next-highest wrong logit, divided by the spread of the wrong logits [Abstract].
In pretrained LMs, Muon has the larger margin-to-spread ratio at small data budgets and AdamW has the larger ratio at large data budgets — the same crossover as fine-tuning performance, in the same direction [Abstract].
In a linear associative-memory setting, after one update the margin-to-spread ratio of gradient descent gains on that of Muon as the data budget grows — a provable mechanism, not just an empirical correlate [Abstract].
Simulations in the same associative-memory model show the margin-to-spread ratio trend persists after extended training, not just at one step [Abstract].

Method

The paper runs matched Muon-vs-AdamW pretraining sweeps across model sizes and TPP ratios, then fine-tunes each resulting checkpoint and measures downstream performance. For each pair they compute a margin-to-spread ratio on the fine-tuning task logits — margin = gap from the correct logit to the next-highest wrong logit, spread = standard deviation (or analogous spread measure) of the wrong logits. They show the ratio tracks fine-tuning performance across the crossover regime. In parallel, they analyze a toy linear-associative-memory model where gradient descent and Muon-style spectral updates can be compared in closed form after a single update, and prove a condition under which GD’s margin-to-spread ratio exceeds Muon’s as the data budget grows. Simulations extend the single-update claim to the multi-step trajectory.

Results

Headline: Muon loses to AdamW on fine-tuning performance once pretraining data is large enough, and this cutoff arrives earlier in TPP as model size grows [Abstract].
The margin-to-spread ratio mirrors the fine-tuning crossover in LMs (not a loss-only signal) [Abstract].
A one-step linear-associative-memory analysis proves GD’s margin-to-spread ratio gains on Muon’s as data budget grows [Abstract].
Simulation evidence extends the trend past the one-step regime [Abstract].

Specific exponents, headline sizes, and the exact crossover TPP by model scale are reported in the full paper; the abstract establishes the direction and the proposed mechanism.

Why it’s interesting

This is the first filed paper that argues the Muon-vs-AdamW ordering inverts with scale — the existing optimizer-crossover literature on the wiki (The Best Optimizer Depends on Batch Size on batch-size reordering, To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters on Muon’s inductive-bias cost in parallel-saddle dynamics, Lucas Beyer — curb the enthusiasm on zeroth-order LM training; loss 4.2 is near the unigram floor on the related “speedrun gains near unigram floor” skepticism) had only reached “Muon’s advantage is batch-size-dependent” or “Muon loses GD’s rank-monotone simplicity bias.” Li et al. close that line by adding a data-scale axis and a concrete mechanism (margin-to-spread) that spans both LMs and a provable toy model. It also directly supports the filer’s team-internal observations — small-LM tuned AdamW beating tuned Muon at ~100 TPP, and 500M T2I diffusion Muon never beating tuned AdamW at compute optimality — and complements the “no one has done a real at-scale AdamW vs Muon ablation” critique that has been circulating since Kimi K2 shipped with Muon. Relevant action item for the µP-vs-Muon fork tracked on Training stability at scale and Hyperparameter scaling laws: a speedrun-optimal optimizer is not necessarily a fine-tune-optimal optimizer at the scales Luma cares about.