Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization

ListenHyperball: Killing Weight Decay to Keep Muon Fast10:07

Show notes

A look at *Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization* — a wrapper that replaces decoupled weight decay with an explicit Frobenius-norm constraint on both weights and updates. We unpack why Muon's speedups shrink at scale, how pinning the weight norm turns the learning rate into an explicit angular step size, and where the numbers land: 20–30% token-equivalent speedup at 1.2B, near-total LR transfer across widths and depths, and a modded-nanogpt Track 3 record for KL-SOAP-H. We close with what this might mean for DiT and video training, and the open questions the paper leaves open. **Sources** - Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization (Kaiyue Wen, Xingyu Dang, Kaifeng Lyu, Tengyu Ma, Percy Liang)

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

Hyperball is a per-matrix optimizer wrapper that replaces decoupled weight decay with an explicit Frobenius-norm constraint: after each base-optimizer step, the weight matrix is re-projected onto a sphere of fixed radius $\rho = \|W_0\|_F$ , and the update itself is Frobenius-normalized so the effective per-layer angular step size is a designed quantity. On Qwen3-style pretraining at 1B parameters, Muon-Hyperball (MuonH) attains 1.4–1.6× token-equivalent speedup over MuonW and the gap grows with training duration, whereas MuonW gives only ≈1.1× over AdamW at that scale. The paper derives the mechanism from the rotational-equilibrium literature (Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks, Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD): under scale-invariant weight blocks, weight decay’s job is to set the equilibrium radius $\rho^*$ that in turn sets the angular learning rate $\eta / \rho^*$ — Hyperball skips the equilibration and pins $\rho$ directly. Consequences: LR transfers across widths and depths with maximal drift ~5% (vs 24–40% for AdamW/MuonW), and paired with KL-SOAP the same wrapper (KL-SOAP-H) sets the fastest step-count on modded-nanogpt Track 3.

Key claims

Hyperball is a two-part constraint applied to attention and MLP matrices: pin $\|W\|_F = \rho$ (fixed at initialization) and take a unit-Frobenius update direction scaled by learning rate $\eta$ , then re-project to the sphere every step [§2 eq. 1, Algorithm 1]. Embeddings, norm gains, and other parameters whose magnitude carries semantic information continue to use unwrapped Adam [§2].
On 1B Qwen3-style models over Chinchilla ratios 1–10, MuonW gives ≈1.1× token-equivalent speedup over the AdamW scaling law, whereas MuonH sustains 1.4–1.6× and the gap grows with training duration — MuonH starts slightly worse but overtakes weight decay as the learning rate decays [§3.2, Fig. 2].
On the public Marin Speedrun benchmark (FineWeb-Edu, Chinchilla), AdamH and MuonH match weight-decay baselines that are 1.5–2× larger [§3.2, Fig. 3 left]; scaling MuonH to 8B parameters on Marin Ferries yields a further loss improvement over the AdamW baseline at the same manually chosen hyperparameters [§3.2, Fig. 3 middle/right].
On modded-nanogpt Track 3, AdamH reaches average validation loss 2.92 in 3325 steps, MuonH reaches 2.92 in 2750 steps, NorMuonH reaches 2.92 in 2500 steps, and KL-SOAP-H reaches 2.92 in 2200 steps — beating the tuned single-run WD baselines (AdamW, MuonW, NorMuonW) at every optimizer pair [§3.2, Fig. 4].
Learning rate transfer under Hyperball is essentially width- and depth-invariant: the maximal drift of the optimal LR is ≈5% for AdamH and MuonH across the width sweep and the depth sweep, versus 24% for AdamW / 40% for MuonW at the same sweep points [§3.3, Figs. 5–6].
In an overtrained data-scaling sweep at 400M parameters over 40B–1.6T tokens, MuonH attains lower best C4 validation loss than MuonW across the full range; fitting best-loss curves gives exponent $\alpha \approx 0.09$ for MuonH vs $\alpha \approx 0.07$ for MuonW [§3.4, Fig. 7].
Theorem 4.6 (the mechanism claim): under an infinite-history idealized stationary model, decoupled weight decay causes the weight norm to converge to an equilibrium radius $\rho^*$ set by learning rate, weight decay, and the optimizer’s update-norm and autocorrelation statistics — and this equilibrium radius in turn determines the angular step size $\eta / \rho^*$ ; the base optimizer enters only through the update-norm and autocorrelation-sum scalars [§4.2, Table 1].
The autocorrelation-sum $s_1$ for SVD-Muon and Moonlight-Muon is negative and bounded in $[-1, 0]$ via a Hermite-expansion of the Gaussian noise operator applied to the matrix-sign kernel — so at stationarity the current update is negatively correlated with the current weight matrix, with the magnitude of that correlation controlled by momentum $\beta_1$ [§4.2 eqs. (30), (36), Fig. 8].
Five empirical phenomena of WD-based training are recovered by the theory: (1) weight norm tracks LR warmup and decay [Fig. 9 top]; (2) gradient norm grows through training as $\|W\|$ shrinks (matches the concurrent explanation in Defazio 2025) [Fig. 9 bottom]; (3) holding $\eta / \sqrt{\eta \lambda}$ fixed keeps train loss essentially unchanged while weight norm scales with $\sqrt{\eta / \lambda}$ [Fig. 10]; (4) WD runs start with higher loss but ultimately converge lower under a shared schedule, consistent with river-valley theory [Fig. 11]; (5) transfer under µP is more sensitive to WD scaling than to init-scale [§4.3].
Expressivity is preserved: in normalized networks a trainable RMSNorm gain can absorb the scale of the constrained matrix, so fixing $\|W\|_F$ does not remove any represented function [§4.1 eqs. (4)–(5)].
Frobenius-sphere projection is chosen over spectral-sphere projection primarily on cost: $O(mn)$ per matrix vs $O(mn \min(m,n))$ for exact spectral projection; the paper reports the stable-rank ratio empirically stays well above 1, so Frobenius control tracks spectral control up to a slowly-varying factor [§2 discussion, eq. (3)].

Method

Hyperball wraps any base optimizer step $\tilde U$ (Adam’s preconditioned update, Muon’s Newton–Schulz msign, Moonlight, KL-SOAP, NorMuon, …) with two operations: normalize $\tilde U$ in Frobenius norm to unit direction, then rescale by $\eta$ and project the resulting displacement back to the sphere of radius $\rho = \|W_0\|_F$ . Concretely, $W_{t+1} = \rho \cdot \Pi(W_t - \eta \, \hat U_t)$ where $\Pi(\cdot) = (\cdot)/\|\cdot\|_F$ and $\hat U_t = \tilde U_t / \|\tilde U_t\|_F$ . The scaling $\rho = 0.5 / \sqrt{d}$ per matrix at init (with $d$ the embedding dimension) matches what @wen_kaiyue’s tweet observes empirically in Muse Glimmer’s weights: per-matrix weight RMS pinned at ~6e-3 ≈ 0.5/√6656.

The theory in §4.2 makes this substitution principled. Under an infinite-history stationary model with isotropic Gaussian gradients, the decoupled-WD recursion $W_{t+1} = (1 - \eta\lambda) W_t + \eta U_t$ has a well-defined equilibrium: $\|W\|_F$ converges to $\rho^*$ determined by $\eta$ , $\lambda$ , and the base-optimizer scalars $\|U\|_F^2$ (update-norm second moment) and $s_1 = \sum_{k \ge 1} \langle U_t, U_{t+k}\rangle / \|U\|^2$ (normalized autocorrelation sum). The angular step size at this equilibrium is $\Delta\theta \approx \eta/\rho^*$ modulo the autocorrelation-induced radial correction. Under this decomposition, decoupled WD is doing two coupled jobs — setting $\rho^*$ and setting $\Delta\theta$ — and any change to $\eta$ or $\lambda$ shifts both. Hyperball severs the coupling: fix $\rho$ directly, and $\eta$ becomes an explicit angular learning rate. The KL-SOAP-H record on modded-nanogpt is the strongest evidence the wrapper is optimizer-agnostic — it is not a Muon-specific hack.

Results

1B / Chinchilla scaling (headline speedup): MuonW ≈1.1× over AdamW; MuonH sustains 1.4–1.6× with the gap growing with training duration [§3.2, Fig. 2].
Marin Speedrun: AdamH and MuonH match WD baselines that are 1.5–2× larger [Fig. 3 left]. Marin Ferries 8B: MuonH beats AdamW baseline at matching manual HPs [Fig. 3 middle/right].
modded-nanogpt Track 3 leaderboard (fixed model & data, measured in step count to reach val loss 2.92): AdamW 4578 (single run) / MuonW 3350 (tuned) / NorMuonW 2900; AdamH 3325 / MuonH 2750 / NorMuonH 2500 / KL-SOAP-H 2200 [Fig. 4].
LR transfer across widths and depths: maximal drift ≈5% for AdamH/MuonH vs 24% AdamW / 40% MuonW at large depth [§3.3, Figs. 5–6].
Overtraining: MuonH lower best C4 val loss than MuonW from 40B–1.6T tokens at 400M params; fitted best-loss exponent $\alpha \approx 0.09$ (MuonH) vs $\alpha \approx 0.07$ (MuonW) [§3.4, Fig. 7].
Empirical validation of the theory: five predicted phenomena recovered (weight norm tracks LR schedule; gradient norm inversely scales with weight norm; train loss depends on $\eta / \sqrt{\eta\lambda}$ not on $\eta$ and $\lambda$ separately; WD trajectory beats no-WD via a river-valley mechanism; µP transfer is sensitive to WD scaling not to init scale) [§4.3, Figs. 9–11].

Why it’s interesting

This is the formal writeup of the empirical result the wiki has been tracking piecemeal. The MuonH (hyperball) sets new modded-NanoGPT speedrun record at 3325 steps — Keller Jordan announcing @wen_kaiyue's result speedrun record was the announcement; Delphi: Scaling Laws That Extrapolate 300× Past the Fit showed AdamH was the load-bearing recipe change that let Delphi’s scaling law extrapolate 300× past the fit (from 3e20 to 1e23 FLOPs); Rethinking Language Model Scaling under Transferable Hypersphere Optimization (HyperP) proved weight decay is a first-order no-op under Frobenius-sphere constraint. Hyperball collects the algorithm, the mechanism, and the multi-scale empirical validation in one place — and the modded-nanogpt Track 3 sweep (AdamH, MuonH, NorMuonH, KL-SOAP-H all beating their WD counterparts, with KL-SOAP-H setting the leaderboard) is the paper’s strongest optimizer-agnosticism claim.

Two connections make this load-bearing for the team’s optimizer stance: (1) it sits alongside Controlled LLM Training on Spectral Sphere (SSO) as the two production-tested “constrain weights + updates to a manifold” recipes — Hyperball uses Frobenius sphere ( $O(mn)$ projection) and closes the loop by rejecting weight decay entirely; SSO uses spectral sphere ( $O(mn\min(m,n))$ projection) and argues Muon is only “half-aligned” with µP because it controls updates but not weights. Both take direct aim at the µP-vs-QK-Norm framing on Training stability at scale; (2) the Slack pointer’s original observation — Muse Glimmer’s per-matrix weight RMS pinned at ~6e-3 = 0.5/√6656 — is prima-facie evidence that at least one frontier open-weights release (MSL’s, per Introducing Muse Glimmer: An Open Agentic Model That Runs on Your Device) is running something Hyperball-shaped in production, though the launch post does not disclose the optimizer. If that inference holds, Hyperball has shipped in a frontier release before the paper hit arxiv, which is a striking adoption speed for a Muon-family variant.

For our own DiT ablation budget, Kyle’s original diffusion note on MuonH (hyperball) sets new modded-NanoGPT speedrun record at 3325 steps — Keller Jordan announcing @wen_kaiyue's result still stands: DiT training appears largely insensitive to weight decay, so an optimizer that removes WD without cost is a strictly cheaper drop-in — and the Frobenius projection is $O(mn)$ per matrix, cheaper than the SSO route. The open questions the paper does not answer: does the mechanism carry over to flow-matching objectives (where the stationary-Gaussian-gradient assumption may break), and does the “one fewer hyperparameter to sweep” story hold for the AdaLN modulation weights that are known to behave differently from attention/MLP projections under Muon (CMuon: Accelerating and Stabilizing Diffusion Transformer Training via Chunked Momentum Orthogonalization).