Spectral Scaling Laws of Muon

TL;DR

First systematic study of how Muon’s Newton–Schulz (NS) orthonormalization budget needs to change as models scale. Tracks singular-value quantiles of the momentum-buffer matrices across every layer in transformer pretraining runs from 77M to 2.8B parameters and finds that, after a short burn-in, quantiles stabilize at layer- and size-dependent values that follow clean power laws in model size $M$ . Early-to-mid layers scale very mildly ( $\sim M^{-0.25}$ ), so the standard 5-step NS configuration will keep orthonormalizing them at frontier scale; some late layers scale much more aggressively (up to $M^{-0.96}$ ) and will fall into the NS failure regime unless given more NS iterations or better-tuned coefficients. Gives practitioners a layer-aware recipe for the minimum NS configuration that still orthonormalizes the directions that matter.

Key claims

After a short burn-in period, the singular-value quantiles of Muon’s momentum buffer stabilize at a value determined by layer type and model size — they don’t drift further during training [Abstract].
Stabilization values follow clean power laws in model size $M$ , with layer-dependent exponents [Abstract].
Layers up to mid-late depth scale very mildly (around $M^{-0.25}$ ); the standard 5-step NS configuration used at academic scale will continue to orthonormalize them at frontier scale [Abstract].
Some late layers scale much more aggressively, with exponents up to $M^{-0.96}$ , and will fall into the NS failure regime at frontier scale unless given more NS iterations or better-tuned NS coefficients [Abstract].
A layer-aware NS schedule (more iterations on the aggressive-scaling late layers, the default 5-step config elsewhere) is the prescribed recipe for choosing the minimum NS configuration that still orthonormalizes the directions that matter — NS iterations are computationally expensive at scale, so this avoids unnecessary compute without sacrificing update quality [Abstract].

Method

The study tracks singular-value quantiles of Muon’s momentum buffer matrix at every layer over the course of pretraining, across a model-size sweep from 77M to 2.8B parameters. Because Muon applies the Newton–Schulz iteration to the momentum buffer (not to the raw gradient or to the weight), the buffer’s spectrum is what NS actually has to orthonormalize each step; small singular values are the ones NS fails to bring to 1, so quantile statistics directly tell you how much of each layer’s update Muon is leaving un-orthonormalized at any given NS budget. Per-layer power-law fits of the stabilization quantile against model size $M$ give layer-specific exponents, which feed back into a layer-aware NS-step prescription.

Results

Power-law exponents in $M$ span roughly $-0.25$ (mild) for layers up through mid-late depth to $-0.96$ (aggressive) for some late layers [Abstract].
At the academic-scale 5-step NS default, mild-scaling layers stay inside the NS-orthonormalization regime as $M$ grows; the aggressive-scaling late layers cross out of it without more NS steps or retuned coefficients [Abstract].
Quantiles stabilize quickly (short burn-in) and remain at the layer/size-determined value through training [Abstract].

Why it’s interesting

This is the cleanest filed answer to a question the wiki’s optimizer thread has been circling but not pinning down: which parts of Muon break first at scale, and what’s the minimum compute fix? It complements Gram Newton-Schulz: A Fast, Hardware-Aware Newton-Schulz Algorithm for Muon (which speeds up each NS step via hardware-aware rewrite) and Muon is Not That Special: Random or Inverted Spectra Work Just as Well (which questions whether NS-style spectral structure is what makes Muon work) by giving a layer-resolved, model-size-resolved empirical map of where NS approximation actually fails — and turning it into a per-layer iteration prescription rather than a global one. It also tightens the Controlled LLM Training on Spectral Sphere critique that “Muon is half-aligned with µP”: the spectral-sphere paper argues weight spectra drift; this paper measures the related quantity (momentum-buffer spectrum) and shows the drift is layer-dependent in a predictable way. Practical implication for runs at Luma scale: budget extra NS iterations on the late layers and keep the default elsewhere.