How To Scale

TL;DR

A living, anonymous tutorial / “dump” on how to scale neural networks — concretely, how to set per-parameter learning rate, init std, multiplier, batch size, weight decay, and adjacent knobs as model width, depth, batch size, and training horizon grow. The spine is a from-first-principles derivation of Maximal Update Parameterization (µP) as the unique parameterization that achieves three desiderata (training stability, maximal feature learning, non-triviality) when width → ∞, derived via just LLN + CLT on the dot products that NN forward/backward boil down to. Then it walks past µP into the things µP doesn’t cover — depth, batch size, dataset/training-horizon, weight decay, AdamW β/ε — surveying SCION, Muon (with QK-clip / MuonClip), AdamW-as-EMA, CompleteP, Spectral Condition, and recent module-router stability work for MoE. Last updated 2025-12-31.

Key claims

µP is fundamentally about maximal feature learning at every layer at every optimization step in the infinite-width limit; HP transfer across width is a consequence, not the goal [§“It’s important to note that muP literally stands for Maximal Update”].
Standard Parameterization (SP) is well-defined only at initialization — after one optimizer step, the dot product of a column of $W$ with the new input becomes correlated and grows as $\Theta(n)$ unless explicitly countered, which is why SP “left-shifts” optimal LR curves as width grows [§“SP is only well defined at the initialization point”].
The whole µP derivation reduces to choosing LLN vs CLT for each dot product in fwd/bwd based on whether the two vectors are correlated; correlated dot products scale as $\Theta(n)$ (LLN), uncorrelated as $\Theta(\sqrt n)$ (CLT), and µP picks per-parameter $(a, b, c) = (\text{multiplier}, \text{init std exp}, \text{lr exp})$ to cancel these growths [§“All you need is LLN, CLT, …”].
Under SGD the upstream gradient is already $\Theta(1/n)$ for hidden matrices so no extra LR scaling is needed; under Adam the per-coordinate rescaling kills this and the µP prescription becomes the canonical “Adam LR ∝ $1/n$ for hidden layers” rule [§“Adaptive optimizers like Adam … LR should be scaled by $1/n$ ”].
Vector-like parameters (embedding, unembedding, biases — one infinite dim) need a different abc triple than matrix-like hidden parameters (two infinite dims), which is why the µP table has three rows (hidden / embedding / unembedding) [§“Vector-like vs Matrix-like”].
µP exhibits an abc-parameterization symmetry: if multiplier is scaled by $\theta$ , init std should be scaled by $1/\theta$ , and (under SGD) LR by $1/\theta^2$ — under Adam, by $1/\theta$ . The fwd/bwd of the network is then literally unchanged. This is why three different-looking µP tables in TP-V are equivalent implementations [§“abc-parameterization symmetry”, with PyTorch sanity-check snippet].
µP does not guarantee HP transfer across training tokens or batch size — the framework is silent on the horizon axis, and the author cites the OLMo team’s experience as confirmation that lr drifts predictably with token count even under correct µP [§“it does not guarantee HP transfer across training tokens or batch size”].
Recently proposed optimizers — Muon with the proper scaling factors, and SCION (Training Deep Learning Models with Norm-Constrained LMOs) — also deliver LR transfer across width, but via spectral-norm control rather than the per-parameter abc recipe; the author treats normalization + adaptive optimizers + parameterization as three loci of the same underlying goal: stabilize and balance training dynamics [§“Recently proposed advanced optimizers like Muon and SCION show that lr can transfer across model widths”].
In real-world large-scale training, µP’s edge over SP shrinks as scale grows; the author’s own experience is that µP wins benchmarks at 200–300B tokens but the gap closes at frontier scale, conjecturing SP’s embedding/output layers eventually escape the lazy-training regime [§“my own experience”].
For MoE, µP is “compatible” but sensitive to init std and output scale of routed vs shared experts; the linked Moonlight paper appendix (Jingyuan Liu, 1st author of Muon is Scalable for LLM Training) emphasizes matching RMS scale of each module’s output to prevent expert collapse [§“in my experience, MoE is bit sensitive to init std and output scale of routed experts”].
The closest the field has to a completed scaling table — covering width × depth × batch size × training horizon — is “Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration” (cited as the current SOTA recipe and superseding the post’s own earlier table) [§“this scaling table is outdated now (Dec 31st 2025) … Completed Hyperparameter Transfer …”].
Pre-norm vs post-norm is implicitly a parameterization choice too: post-norm doesn’t preserve identity-mapping (gradient norm), needs lr warmup; pre-norm trades that for representation collapse in deep layers. Cites depth-scaled sandwich norm (Pangu Ultra), DeepNorm, Mix-LN as architectural attempts at the same “stabilize + maximally learn” goal that µP attacks parametrically [§“Many researchers studied normalization module or parameterization”].

Method

The post is a single long-form blog (running update; last edit Dec 31 2025). Structure: motivation (scaling laws don’t tell you how to set HPs) → SP-as-initialization-only-recipe → derivation of µP from LLN + CLT on dot products in fwd and bwd, with the SGD-vs-Adam distinction made explicit → vector-vs-matrix-like classification → the three abc desiderata (forward stability, feature-learning, no-blowup gradient) → abc-parameterization symmetry with a runnable PyTorch sanity check that under SGD the symmetry is $(\theta, \theta^{-1}, \theta^{-2})$ and under Adam $(\theta, \theta^{-1}, \theta^{-1})$ → coordinate-check evidence on LLaMA-3 architecture → axes µP does not cover (depth, batch size, training tokens, weight decay, AdamW β/ε) → a tour of more recent work that fills those gaps (CompleteP, Spectral Condition for Feature Learning, Muon/SCION, AdamW-as-EMA following Wang et al., MoonLight / MuonClip for MoE-RMS-scale). The post is heavily cross-linked to its primary sources: TP-IV/V (Yang et al. 2022/2023), TP-IVb (Yang et al. 2023; adaptive optimizers in infinite-width), A Spectral Condition for Feature Learning (Yang, Simon, Bernstein 2023), the original “What to do to scale up?” by Simo Ryu (the post’s stated inspiration), Keller Jordan’s Muon writeup, and SCION (2502.07529).

Results

This is a tutorial / survey, not a paper with metrics. The two empirical anchor points the author asserts from personal experience:

µP’s pretraining-benchmark advantage over SP is largest at ~200–300B tokens and diminishes at frontier scale. No tables.
MoE under µP is “bit sensitive” to routed vs shared expert RMS scale; the corrective recipe is the Moonlight-paper appendix prescription (match per-module output RMS).

The bibliographic value is the curated reading list across the muP / Muon / scaling-table line: Tensor Program IV / IVb / V, Spectral Condition, “Why do Learning Rates Transfer?” (2402.17457), CompleteP, “How to set AdamW’s weight decay as you scale model and dataset size” (Wang et al.), “Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration”, Muon (Jordan), SCION (2502.07529), Pangu Ultra’s depth-scaled sandwich norm (Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs), and the “Muon is Scalable for LLM Training” / Moonlight line.

Why it’s interesting

This post is the highest-density entry point on the wiki for the µP-vs-Muon-vs-Fitting fork that Hyperparameter scaling laws is currently tracking: the same author’s other post Beyond MuP 3: Special Cases, Special Treatment (Embedding, LM Head, RMS Norm) is also indexed, but this one carries the derivation (LLN + CLT on dot products) that makes the rest of the cluster intelligible. The author’s own admission that µP’s edge erodes at frontier scale is the steelman version of the position How to Set the Learning Rate for Large-Scale Pre-training? argues (Fitting beats µTransfer at 4B/12B MoE on 500B tokens) and is directly opposed by Controlled LLM Training on Spectral Sphere (Muon is “half-aligned” with µP; close the loop and µP transfer holds). It also pre-figures the architectural-stability story that Training stability at scale is built around — the post explicitly frames normalization choices, adaptive optimizers, and parameterization as three loci of the same underlying problem, which is the framing both How to Set the Learning Rate for Large-Scale Pre-training? and A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training later pick up and push in different directions. The MoE-RMS-scale aside also previews the Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method / QK-Clip: Taking Muon Further on the Scaleup Journey “eliminate the tuning knob entirely” line.