The Newton-Muon Optimizer

TL;DR

Newton-Muon derives a Muon-style matrix-orthogonalization optimizer from a one-step Newton analysis of a “triplet quadratic surrogate” loss model, and ends up with a single algorithmic change to standard Muon: right-precondition each layer gradient by the inverse activation second moment $(ZZ^\top)^{-1}$ before the usual Newton-Schulz orthogonalization. Under the analysis, standard Muon falls out as the special case where the activation second moment is isotropic — an assumption the paper measures to be sharply violated in practice. On the Modded-NanoGPT short-track Record #4 baseline (124M GPT-2, FineWeb, single H100), Newton-Muon reaches the target validation loss in 6% fewer iterations and ~4% less wall-clock time than standard Muon, while reusing the rest of the Muon pipeline (Newton-Schulz polynomial, momentum, weight decay). The contribution is as much interpretive as algorithmic: Muon is recast as an implicit Newton-type method that drops the input-data preconditioner.

Key claims

Standard Muon can be derived as the one-step minimizer of a “triplet quadratic surrogate” loss model $\mathcal{L}(W + \Delta) \approx \mathcal{L}(W) + \langle G, \Delta\rangle + \frac{1}{2}\mathrm{tr}(\Delta H \Delta^\top Z Z^\top)$ — i.e. an implicit Newton-type method — under the additional assumption that the input activation second moment is isotropic ( $ZZ^\top \propto I$ ) [§2.3, Corollary 1].
Dropping the isotropic-activation assumption and instead modeling only the displacement $W - W^*$ as isotropic yields the closed-form update $W \leftarrow W - \eta \cdot \mathrm{msgn}(G (ZZ^\top)^{-1})$ , which differs from Muon only by a right preconditioner $(ZZ^\top)^{-1}$ applied to $G$ before orthogonalization [§2.3, Theorem 1].
The Newton-Muon direction is provably a descent direction: $\langle \mathrm{msgn}(G (ZZ^\top)^{-1}), G \rangle \ge 0$ unconditionally on the activation geometry [§2.3, Proposition 2].
Under a single-spike model where the activation matrix has one anisotropic direction with condition number $\kappa$ and an isotropic bulk, gradient descent and Muon both take $\Theta(\log(1/\epsilon)/\log\kappa) \cdot \kappa$ -flavored iterations to converge, while Newton-Muon’s contraction rate is independent of $\kappa$ — a factor- $O(\kappa)$ asymptotic speedup [§3, Corollary 2].
In practice $ZZ^\top$ is highly anisotropic during LLM training (large condition number, bounded stable rank), so the right-preconditioner step “explicitly cancels this right-side anisotropy” instead of letting the Muon orthogonalization absorb it implicitly [§3, after eq. (9); Table 4].
The compute overhead is small: the activation second moment is maintained as an EWMA and its damped inverse is recomputed via Cholesky + triangular solves only every $K$ steps (refresh interval, e.g. 16); between refreshes the cached inverse is reused. Cost is “no more than about [one] matrix multiplication” per refresh on modern GPUs [§1, §2.3, Appendix B].
On Modded-NanoGPT short-track Record #4 (124M GPT-2, 3.25B FineWeb tokens, single H100), Newton-Muon reaches lower final validation loss than Muon (3.2611 vs 3.2793 vs AdamW 3.3801), and the announcement claim is 6% fewer iterations and ~4% wall-clock reduction to reach the target loss [§5.1, Table 1, Fig. 1]. Wall-clock-total in Table 1 is actually slightly higher for Newton-Muon (7443.3s vs 7314.1s); the headline 4% improvement is time to target validation loss 3.28, not total run time.
Newton-Muon retains left-rotation equivariance with Muon but loses right-rotation equivariance unless the input space is co-rotated: $\mathrm{msgn}(G(ZZ^\top)^{-1})$ is not basis-free on the input side, by design [§2.3, Fig. 2].

Method

Start from a local quadratic loss model where the per-sample contribution is approximated by a shared output-space curvature matrix $H$ acting on the layer-output perturbation $\Delta z_i$ . Summing over samples gives the triplet quadratic surrogate:

$\mathcal{L}(W + \Delta) \approx \mathcal{L}(W) + \langle G, \Delta\rangle + \tfrac{1}{2}\mathrm{tr}(\Delta H \Delta^\top Z Z^\top),$

where $G$ is the gradient matrix, $Z$ stacks the layer inputs, and the quadratic term is the matrix form of the Kronecker-factored curvature approximation used in K-FAC. Minimizing this over $\Delta$ subject to the auxiliary assumption $W - W^* \propto I$ (least-informative isotropic prior on the displacement, not on the activations) gives the closed-form Newton step

$\Delta^* \propto \mathrm{msgn}(G (ZZ^\top)^{-1}),$

where $\mathrm{msgn}(X) = UV^\top$ for SVD $X = USV^\top$ . In implementation, $(ZZ^\top)^{-1}$ is maintained as a damped EWMA inverse refreshed every $K$ steps via Cholesky, then the Muon Newton-Schulz polynomial (e.g. Polar Express or Gram Newton-Schulz from Gram Newton-Schulz: A Fast, Hardware-Aware Newton-Schulz Algorithm for Muon) is applied to the right-preconditioned gradient. The change to a Muon codebase is one preconditioner step. Newton-Muon recovers Muon exactly when $ZZ^\top \propto I$ , but the paper notes (Table 4 and supporting analysis) that this assumption is sharply violated in practice — $ZZ^\top$ has large condition number and bounded stable rank during LLM training.

A complementary one-step analysis under a spiked-activation model assigns a scale-invariant “quadratic score” $\langle \Delta, G\rangle^2 / \mathrm{tr}(\Delta H \Delta^\top ZZ^\top)$ to candidate update directions; Newton-Muon’s score sits between Muon’s and the exact Newton direction’s, closer to exact Newton, in the regime where both curvature and activation anisotropy are strong [§4.2, Fig. 3].

Results

Baseline experiment is Modded-NanoGPT short-track Record #4: 124M GPT-2, 3.25B FineWeb tokens, single H100. Newton-Muon vs Muon vs AdamW [§5.1, Table 1]:

Method	Final val loss	Total wall-clock (s)
AdamW	3.3801	7228.4
Muon	3.2793	7314.1
Newton-Muon	3.2611	7443.3

Newton-Muon achieves the lowest final loss. Time-to-target-loss (val 3.28) is reported as 6% fewer iterations and ~4% less wall-clock than Muon [§1, Fig. 1]. Note that Table 1’s total training time is slightly higher for Newton-Muon because the per-step cost of the preconditioner refresh is non-zero; the headline speedup is in iterations-to-target, not raw step rate.

Settings used: Newton-Muon LR $\eta = 0.05$ , EWMA coefficient $\beta = 0.9$ , ridge scaling, refresh interval $K = 16$ . Newton-Schulz polynomial unchanged from Modded-NanoGPT Muon. Section 5 also reports adapted runs against Record #28 (short) and Record #17 (medium) — these later records are “heavily optimized around the original Muon update,” so the authors flag that direct comparisons there evaluate Newton-Muon under heavily Muon-tuned recipes rather than at parity.

The single-spike case study (§3) gives the theoretical headline: Newton-Muon’s contraction rate is independent of the spike condition number $\kappa$ , while Muon and gradient descent inherit a $\kappa$ -flavored slowdown. The spiked-activation numerical study (§4.2, Fig. 3) shows Newton-Muon’s quadratic score is substantially higher than Muon’s whenever activation anisotropy and curvature anisotropy are both strong, and approaches the exact Newton score.

Why it’s interesting

Newton-Muon lands in the optimizer-side fork of Training stability at scale alongside SSO (Controlled LLM Training on Spectral Sphere) and NSISA (Preconditioned Inexact Stochastic ADMM for Deep Models (PISA / SISA / NSISA)) — all three are “Muon is missing something” arguments, but they identify different missing pieces. SSO says Muon controls update spectra but not weight spectra, and proposes constraining both to a spectral sphere. NSISA says the issue is the proximal-step structure, and proposes ADMM with Newton-Schulz-orthogonalized momentum. Newton-Muon says Muon implicitly assumes isotropic input activations, and the fix is one extra right-preconditioner $(ZZ^\top)^{-1}$ — a K-FAC-style data-geometry term that Muon’s matrix-sign was almost absorbing but not quite. The three diagnoses are not directly contradictory; the experimental gap between them on a common LLM pretraining benchmark is the load-bearing comparison this line of work still needs.

Compared to the explanatory Muon coverage already on the wiki — This Simple Optimizer Is Revolutionizing How We Train AI [Muon] and the Gram Newton-Schulz hardware work Gram Newton-Schulz: A Fast, Hardware-Aware Newton-Schulz Algorithm for Muon — Newton-Muon is the first concrete first-principles re-derivation of Muon as a Newton-type method that filed on the wiki, and the first to make the K-FAC connection algorithmically actionable rather than just structural. The single-spike $O(\kappa)$ separation between Muon and Newton-Muon is also the cleanest theoretical statement yet on the wiki about when Muon’s matrix-sign step under-preconditions.