Rethinking Language Model Scaling under Transferable Hypersphere Optimization (HyperP)

TL;DR

HyperP is a hyperparameter-transfer framework built on top of hypersphere optimization (MuonH / Frobenius-norm-constrained weights) that transfers a single small-scale-tuned base learning rate across width, depth, training tokens, and MoE granularity. The authors prove weight decay is a first-order no-op on the Frobenius sphere (so it disappears as a hyperparameter), recover Depth-µP as still necessary (correcting the original MuonH paper), and rediscover the “magic exponent” 0.32 for LR-vs-tokens scaling first observed for AdamW — suggesting it is optimizer-independent. At 6e21 FLOPs HyperP gives 1.58× compute-efficiency-leverage over a strong P++/weight-decayed Muon dense baseline, and the MoE variant adds another 3.38× over dense; gains grow monotonically with scale. Also introduces SqrtGate, a $\sqrt{k}$ replacement for the $1/k$ softmax MoE gate, derived from the hypersphere constraint to keep router output RMS invariant to granularity.

Key claims

Under Frobenius-sphere weight constraint $\|W\|_F = c$ , weight decay is a first-order no-op: the radial shrinkage component is killed by the post-update re-projection, so weight decay drops out of the hyperparameter search entirely [Theorem 1, Corollary 1.1, §3.1].
Empirical optimality of MuonH over Muon: at dense 208M / 10.4B tokens, MuonH (weight decay 0) reaches val loss 2.475 vs Muon’s 2.479 at its best $(\eta, \lambda)$ ; Muon is sensitive to weight decay (2.479 best, 2.500 with $\lambda=0.05$ ) [Table 2, Fig. 2].
Width transfer holds under MuonH without explicit LR scaling, provided updates are approximately isotropic (which Muon’s orthogonalization tends to enforce) — recovering µP’s width-transfer property structurally rather than by hand [Theorem 2, §3.2].
Depth-µP is required for MuonH: at fixed aspect ratio, optimal LR drifts from 0.020 ( $L=8$ ) to 0.012 ( $L=32$ ) without Depth-µP, but stays at 0.012–0.014 with it. This refutes the original MuonH paper’s claim of inherent depth-transferability [Theorem 3, §4.2, Figs. 3, Tables 5–7].
Data scaling: optimal LR follows a clean power law $\eta^* \propto D^{-0.32}$ over 10.4B–166.4B tokens, leave-one-out MAE of 1.50%. The exponent 0.32 matches Bjorck et al.’s AdamW result exactly, suggesting it is a universal property of gradient-based optimization rather than an optimizer-specific quantity [§3.4, Fig. 1, Eq. 8].
Hypersphere optimization bounds attention/router/LM-head logit magnitudes structurally: $|\text{logit}_i| \le \sqrt{m}\|x\|_2$ and logit RMS $\le \|x\|_2$ for $W$ on the unit sphere, removing the need for z-loss regularization [Proposition 4, §3.6].
Stability transfer: all six monitored instability indicators ( $\zeta$ -values, output RMS, activation outlier %, both attention and MoE sub-layers) are bounded and non-increasing as MoE models scale from 913M → 13.3B total parameters under HyperP [§5.1].
SqrtGate (replace $1/k$ in MoE gate with $1/\sqrt{k}$ ) preserves routed-branch output RMS across granularity $k$ in the equal-RMS, weak-correlation regime that hypersphere optimization induces, vs classical softmax gating which scales as $1/\sqrt{k}$ (shrinks with $k$ ) [Propositions 5–6, §3.7].
SqrtGate consistently improves val loss across $k \in \{2, ..., 32\}$ , with largest gains at small $k$ ( $k{=}4$ : 0.028 nats; $k{=}8$ : 0.020 nats); baseline saturates at $k{=}16$ but SqrtGate keeps improving to $k{=}32$ (best loss 2.310) [Fig. 7, §4.4].
Under MuonH, the largest auxiliary load-balancing weight $\beta=10^{-2}$ achieves both the best val loss (2.332) and the lowest mean MaxVio (0.086) — contradicting the prior finding (Wang et al. 2024) that large aux-balance weights hurt quality and motivating “auxiliary-loss-free” routing [Table 3, §4.4]. The bounded-logits property is the conjectured mechanism.
Compute Efficiency Leverage: at the largest FLOPs, MuonH+HyperP achieves 1.58× CEL over a P++/weight-decay-tuned Muon dense baseline; MuonH+HyperP MoE ( $S{=}8, k{=}8$ ) achieves 3.38× CEL over the dense baseline. CEL advantage grows monotonically with scale [Fig. 9, §4.5].

Method

The setup is hypersphere optimization with MuonH (Wang et al. 2025): each hidden weight matrix is re-projected to a fixed Frobenius norm after every Muon update, with the update itself also Frobenius-normalized to ensure constant relative update magnitude per layer. The LM head uses AdamH (the same renormalization recipe applied to AdamW); embeddings and vector parameters use unmodified AdamW. The four transfer-law derivations are: (i) width — Frobenius/spectral-norm equivalence under orthogonalized updates gives µP-like width transfer without explicit scaling; (ii) depth — the cumulative angular drift across residual blocks forces $\eta \propto 1/L$ (Depth-µP), regardless of pre/post-norm; (iii) data — empirically $\eta^* \propto D^{-0.32}$ on a fine LR grid across 10.4B–166.4B tokens; (iv) MoE granularity — replace the gate’s $1/k$ with $1/\sqrt{k}$ (SqrtGate) so routed-branch RMS becomes invariant to $k$ .

Architecturally the experiments use a “Transformer-Next” stack inspired by Qwen3-Next: GQA (4 KV heads), head-dim 128, aspect ratio $d_{\text{model}}/L \approx 128$ , QK-Norm, headwise gated attention, SwiGLU with intermediate size $4d/3$ , Pre-Norm. MoE adds SqrtGate, one always-on shared expert, and Top-k-then-softmax routing. Training is on SlimPajama at 4K context, batch 2M tokens, linear-decay LR schedule with no warmup. Codebase released at microsoft/ArchScale.

Results

Dense scaling (Fig. 9, §4.5): across $L \in \{8, 16, 32\}$ at 50 TPP, MuonH+HyperP scaling curve sits below P++-Muon throughout, with the gap widening with scale. CEL at the largest measured FLOPs = 1.58× over the strong baseline.
MoE scaling: MuonH+HyperP MoE ( $S{=}8$ sparsity, $k{=}8$ granularity, 208M active → up to 22.9B total) gives a further 3.38× CEL over the dense HyperP curve at the same FLOPs.
LR sweep alignment: at $L \in \{8, 16, 32\}$ , loss-vs-LR curves under HyperP all peak at the same base LR; the small-scale optimum transfers exactly to all larger scales [Fig. 8].
Sparsity sweep: $S \in \{2, 4, 8\}$ at $k{=}8$ → optimal LR stays within 0.012–0.016. Moving $S{=}2 \to S{=}8$ saves 0.224 nats of loss, equivalent to ~6× more dense FLOPs to achieve [Fig. 6].
Granularity sweep: $k \in \{2, ..., 32\}$ at $S{=}8$ → optimal LR stays within 0.012–0.014; SqrtGate’s largest gain is at small $k$ (closing the gap that the baseline $1/k$ gate opens) [Fig. 7].
Batch-size sweep: optimal LR scales as $\eta \propto B^{0.78}$ across $B \in [256\text{K}, 2\text{M}]$ tokens, sitting between linear (1.0) and square-root (0.5) rules. Best loss is essentially invariant in this range, indicating all tested $B$ are below critical batch size [§4.3].
Stability indicators: $\zeta$ -values, output RMS, and activation outlier % for both attention and MoE sub-layers all stay flat or decrease as the model scales from 913M → 13.3B [§5.1].

Why it’s interesting

This is the cleanest entry yet in the Muon / hypersphere / spectral-sphere line and lands directly on the open fork in Hyperparameter scaling laws: Fitting (How to Set the Learning Rate for Large-Scale Pre-training?) vs Transfer (µP / µA, Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning) vs Enforce (SSO, Controlled LLM Training on Spectral Sphere). HyperP is squarely in the Enforce camp but goes further than SSO by also collapsing weight decay to a no-op and supplying explicit transfer laws across the four scaling axes — addressing the open question on Hyperparameter scaling laws about whether enforce-style optimizers restore µP-transfer’s edge at industrial scale. The 0.32 data-scaling exponent matching AdamW’s exactly is a striking universality claim — contrasts with Universal One-third Time Scaling in Learning Peaked Distributions‘s 1/3 softmax-CE exponent (a different universality, set by the loss head). The SqrtGate result also gives an alternate framing on MoE balancing that sits next to Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method‘s Quantile Balancing — both remove tuning knobs but via different mechanisms (router constraint vs router LP). The Depth-µP claim directly contradicts the original MuonH paper, which is the kind of correction-in-print that’s worth pulling forward for anyone building on MuonH downstream. Codebase release (microsoft/ArchScale) is a plus for Open foundation-model releases-style reproducibility.