This Simple Optimizer Is Revolutionizing How We Train AI [Muon]

TL;DR

Jia-Bin Huang visual explainer on Muon — the matrix-aware optimizer (Jordan et al., 2024) that orthogonalizes Adam-style momentum via Newton–Schulz iteration before each update step, producing roughly unit-spectral-norm updates per 2D weight matrix. Muon underpins several recent open-weights training runs (Kimi K2, Moonshot) and is a direct baseline in multiple filed wiki papers. Posted as refresher material in a batch of four videos.

Key claims

No transcript was available at fetch time; the video could not be read in detail. Key-claim extraction is deferred. [video]

Method

Not extracted — transcript unavailable. Muon’s core mechanic, briefly: maintain an Adam-style momentum buffer $M_t$ for each 2D weight matrix; before applying the update, replace $M_t$ with $\mathrm{msign}(M_t) = U V^\top$ where $U \Sigma V^\top = M_t$ — the matrix sign function, approximated cheaply by a few steps of Newton–Schulz iteration on $M_t$ scaled by its spectral norm. The resulting update has roughly unit singular values, which controls the spectral norm of the per-step parameter change and (per its proponents) yields better-conditioned training than vanilla AdamW.

Results

Not applicable — educational video.

Why it’s interesting

Muon is the de facto baseline that several filed papers benchmark against. Controlled LLM Training on Spectral Sphere argues Muon is “half-aligned” with µP — it controls update spectra via Newton–Schulz but places no constraint on weight spectra, so weights drift away from the µP-prescribed scale; SSO closes this loop and reports outperforming Muon on Dense 1.7B, MoE 8B-A1B, and a 200-layer DeepNet. Universal One-third Time Scaling in Learning Peaked Distributions explicitly frames Muon as an “intercept-only” intervention — it moves the LM-loss intercept but not the 1/3 scaling-law slope, since the slope is set by softmax-CE structure rather than by the optimizer. A refresher on what Muon actually does is useful scaffolding for both lines of argument, and for understanding why “Muon vs. AdamW” is a recurring axis in current pretraining papers.