Skip to content
Bud's Wiki

MuonH (hyperball) sets new modded-NanoGPT speedrun record at 3325 steps — Keller Jordan announcing @wen_kaiyue's result

TL;DR

Section titled “TL;DR”

Keller Jordan announces a new modded-NanoGPT speedrun record set by @wen_kaiyue: replace AdamW/Muon weight decay with “hyperball optimization” (i.e. MuonH, the Frobenius-sphere variant of Muon), and the run hits target validation loss in 3325 steps — beating both prior AdamW and Muon baselines on the standard speedrun benchmark. This is the first time the speedrun leaderboard, which has historically tracked the Muon line, has been topped by a sphere-constrained variant rather than a tweak to schedule/architecture.

Key claims

Section titled “Key claims”
  • New modded-NanoGPT record: 3325 steps, improving over both AdamW and Muon baselines on the standard speedrun benchmark [tweet body].
  • The change is mechanically minimal — replace weight decay with hyperball (Frobenius-sphere) optimization on top of Muon [tweet body].
  • Author: @wen_kaiyue; announced by @kellerjordan0 (the speedrun benchmark’s primary curator) [tweet header].

Method

Section titled “Method”

“Hyperball optimization” here is the MuonH recipe: each hidden weight matrix is re-projected onto a fixed-Frobenius-norm sphere after every Muon update, and the update itself is Frobenius-normalized so per-layer relative step size is constant. In the HyperP framework (Rethinking Language Model Scaling under Transferable Hypersphere Optimization (HyperP)), weight decay’s radial-shrinkage component is proven to be a first-order no-op under this projection — so dropping the weight-decay term is principled, not a kludge. Embeddings/LM head use AdamH (the same renormalization applied to AdamW). No further details are given in the tweet beyond the screenshot of the leaderboard delta.

Results

Section titled “Results”
  • Headline: 3325 steps to target validation loss on the modded-NanoGPT speedrun, beating both Muon and AdamW baselines [tweet body].
  • No tables or training curves released in-thread; the result is a benchmark-leaderboard delta, not a paper.
  • 59.1K impressions on the tweet as of fetch time, indicating the optimizer community noticed.

Why it’s interesting

Section titled “Why it’s interesting”

Two reasons this matters for the team. First, it is the empirical small-scale confirmation of the theoretical story already filed under Rethinking Language Model Scaling under Transferable Hypersphere Optimization (HyperP) — HyperP proves Frobenius-sphere weight constraint makes weight decay a first-order no-op, and now MuonH actually wins the speedrun against tuned Muon+WD. The Enforce camp (sphere-constrain both weights and updates) just took a concrete win on a public benchmark over the previously dominant Muon recipe, complementing the optimizer-side spectral-norm variant Controlled LLM Training on Spectral Sphere. Second — Kyle’s point in the Slack post — diffusion training appears to be largely insensitive to weight decay in the first place, so an optimizer that removes the WD knob without quality cost (and with a small speed win on LM pretraining) is a strictly cheaper drop-in on the DiT side. If the no-op claim holds on flow-matching objectives, the practical benefit is “one fewer hyperparameter to sweep” rather than a loss-curve gain — which still matters for any from-scratch DiT ablation budget. The benchmark itself does not test diffusion, so this is a hypothesis to verify, not a result to inherit.

See also

Section titled “See also”