To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters

ListenMuon's Hidden Tax: When Faster Optimization Forgets to Generalize10:36

Show notes

Muon is the fast new optimizer everyone's adopting — but Dragutinović, Zhang, and Ranganath argue the speedup isn't free. Using a simplified, exact-SVD version of Muon they call Spectral GD, they show it drops the saddle-to-saddle simplicity bias that plain gradient descent preserves, learning all singular components in parallel instead of low-rank-first. We walk through the theory on deep linear networks, two head-to-head experiments where SGD beats Muon, and what a "learn everything at once" bias might mean for low-rank shared structure in T2V and MMDiT pretraining. **Sources** - To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters — Sara Dragutinović, Yedi Zhang, Rajesh Ranganath

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

Dragutinović et al. push back on the “Muon just works” consensus by identifying a specific inductive-bias cost of Muon-style spectral optimizers: they lose the saddle-to-saddle simplicity bias that gradient descent naturally preserves. Under an exact-SVD, no-momentum abstraction of Muon (Spectral GD) applied to deep linear networks, GD sequentially increases the rank of the solution (small singular values learned last, one saddle at a time), whereas Spectral GD learns all singular components in parallel at equal rate — explaining Muon’s speedup but also its worse structural generalization. The paper demonstrates two settings where SGD beats Muon: (a) a multi-source “routing” task where SGD discovers shared latent representations while Spectral GD memorizes each input-output pair, and (b) an MNIST-with-spurious-pixel benchmark where SGD’s dominant-first learning gives a longer window of clean-digit accuracy before spurious features kick in. The framing matters for Luma’s optimizer discourse: it puts a cost on the “Muon is more isotropic/uniform” claim that Vasudeva/Wang-line papers celebrate.

Key claims

Under exact-SVD Spectral GD on 2-layer deep linear networks with joint-diagonalizability assumptions, all singular values of the product matrix W₂W₁ evolve simultaneously at the same rate, following a quadratic curve with convergence time proportional to 1/σᵢ — small σᵢ finish first, unlike GD which has sigmoidal per-mode saturation ordered largest-σᵢ-first [§3.1, Theorem 3.2].
GD gradient flow on the same setup has the classical saddle-to-saddle structure: the k-th singular value of W₂W₁ saturates on a sigmoid with time constant 1/σₖ, so learning proceeds by sequentially increasing solution rank and passing near intermediate saddle points [§3.1, Theorem 3.1].
The “faster training” Muon gets is exactly the price of losing this rank-monotone simplicity bias — Spectral GD avoids the saddle plateaus but no longer has the rank-based implicit regularization of GD [§3.2].
On Saxe et al.’s “routing” task (four input-source encoders → shared linear hidden layer → four output-source decoders, trained only on the diagonal + off-diagonal-by-one pairs), SGD generalizes to unseen input-output source pairs by learning the shared 4-D underlying mapping; stochastic Spectral GD achieves the same training loss but fails to generalize, converging instead to a high-effective-rank, heavy-tailed hidden-layer spectrum consistent with pair-wise memorization [§4, Fig. 2c–e].
On MNIST + spurious-pixel-per-class (all training samples contain the spurious pixel; validation splits into with-spurious and without-spurious), SGD’s peak clean-digit accuracy is higher than Adam’s or Muon’s, and its curves for the two validation sets stay coupled longer — consistent with the theory that SGD learns the “dominant” (actual-digit) solution first and only then starts fitting the spurious feature [§5, Fig. 3b–c].
Varying the intensity of the spurious pixel maps out a “dominance switch”: at spurious intensity ≈2, SGD’s clean-validation accuracy crosses below Muon’s, because relying-on-digits stops being the dominant algorithm for SGD to learn first, but Muon’s equal-speed learning of both algorithms is nearly unaffected [§5, Fig. 3d].
After the theoretical rank-1, rank-2, … modes are learned, discretization noise makes the Spectral GD update matrix full-rank, and Spectral GD then fits the noise in a highly oscillatory trajectory that momentum (i.e. the actual Muon update) does not damp — hinting that Muon may be less robust at high learning rates in richer settings [§3.1, discussion around Figs. 5–6].
The framing contradicts the Vasudeva et al. (2026) / Wang et al. (2026) reading of Muon’s “richer, more isotropic” weight spectra as strictly beneficial: this paper constructs a task where “richer” means “memorized every pair” and hurts generalization [§6, Related Work].

Method

The theoretical core replaces Muon’s Newton-Schulz orthogonalization + momentum with Spectral GD: use the exact SVD of the GD update matrix and set all non-zero singular values to 1 (so the update is U V^T where U Σ V^T is the compact SVD of the raw gradient step). This drops both the NS approximation and the momentum, keeping the essence — orthogonalization — while making the flow-limit tractable. On 2-layer deep linear networks with $W_2 W_1$ = product matrix, jointly-diagonalizable input/cross statistics, and infinitesimal init, the authors derive closed-form dynamics for both GD flow (recovering the standard Saxe/Gidel saddle-to-saddle result) and Spectral GD flow (Theorem 3.2 above).

The empirical section then translates the theoretical prediction — “Spectral GD parallel-learns everything, so it should memorize when a task has a low-rank shared solution and a full-rank pair-wise solution” — into two head-to-head experiments (routing task, spurious-MNIST). Crucially, the empirical Muon runs use the full algorithm (Newton-Schulz + momentum + minibatch), not Spectral GD; the theoretical simplification is used only for the flow analysis.

Results

Two quantitative points survive summarization:

Routing task: both SGD and stochastic Spectral GD reach perfect training convergence [Fig. 2e]. Only SGD generalizes to held-out input-output source pairs; Spectral GD hidden-layer effective rank is significantly higher than 4 (the underlying task rank), with a heavy-tailed spectrum characteristic of memorization [§4].
MNIST + spurious pixel: peak clean-validation accuracy is highest for SGD across the sweep of spurious-pixel intensities up to ≈2 [Fig. 3b–d]. At intensity ≈2, the ranking flips; beyond it, Muon and Adam are nearly flat across the switch while SGD’s clean accuracy collapses.

There are no LLM-scale results. The paper positions itself as a counterexample-generator to the “Muon is uniformly better” reading of the current empirical Muon literature, not as a from-scratch competitive claim.

Why it’s interesting

The wiki already tracks a rich Muon debate — the Hyperparameter scaling laws page frames it as three positions (Fitting, Transfer, Enforce), and Training stability at scale catalogs a growing list of Muon-family fixes (Controlled LLM Training on Spectral Sphere SSO, Aurora: A Leverage-Aware Optimizer for Rectangular Matrices Aurora, How Muon Lost Its Geometry the µP correction). This paper adds a fourth axis the others don’t touch: even if Muon were implemented perfectly (µP-correct scaling, distributed, spectral-sphere-constrained), its inductive bias is still qualitatively different from SGD’s — and there exist realistic-flavored tasks where SGD’s rank-monotone bias is what you want. It complements Muon is Not That Special: Random or Inverted Spectra Work Just as Well (which attacks the “geometry is what matters” story from the symmetry side by showing random spectra work equally well) by attacking it from the simplicity-bias side. For Luma’s diffusion/flow training stack the concrete question this raises is: does the “learn all singular components in parallel” bias hurt when the target task has a low-rank shared structure across modalities/conditioning axes (a plausible shape for T2V/MMDiT pretraining), and would SGD-family baselines actually generalize better on some evaluation slices even at slower wall-clock?