Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks

TL;DR

Qiu et al. show that compute-optimally trained neural networks exhibit a striking scaling symmetry: loss curves from models of different sizes collapse onto a single universal curve after normalizing compute and reducible loss to unity at the end of training. With learning-rate decay, the collapse becomes so tight that cross-scale deviations fall below the noise floor of individual runs across random seeds — a phenomenon the authors term supercollapse. The paper provides both an empirical demonstration (transformers on CIFAR-5M and Lichess next-move, plus MLPs on power-law Fourier features) and a theoretical explanation grounded in power-law scaling laws and a simple noisy-quadratic SGD model. Supercollapse serves as a sensitive diagnostic: mis-parameterization (dropping µP for constant LR across widths) or a wrong compute-optimal data exponent both break the collapse well before final loss differences are visible.

Key claims

• Under compute-optimal training with µP, affinely rescaling loss curves as $\tilde{L} = (L - L_\infty)/(L_{\text{final}}(N,\zeta) - L_\infty)$ against normalized compute $\tau = t/T^*(N)$ collapses curves across model sizes onto one universal shape [§2.3, Fig. 1].
• Under LR decay, the collapse deviation $\Delta$ drops below the per-model seed-to-seed noise floor $\sigma$ for a substantial fraction of training — supercollapse — reaching $\Delta \le \sigma/2.5$ [§2.5, Fig. 1].
• Normalizing by the stochastic final loss (not the expected final loss) is essential for supercollapse: it acts as a control-variate that exploits the strong time-correlation of noise along a single trajectory, cancelling shared noise and sharply reducing variance in the collapsed curve [§3.3].
• For a family of expected loss curves $L(t,N) = A/N^\alpha + B/t^\beta + L_\infty$, compute-optimal scaling $t \propto N^{\alpha/\beta}$ balances the two power-law terms in the derivative and the terms themselves, yielding an exact collapse; this equivalence is unique up to a multiplicative constant [§3.1, Eq. 7].
• A simple noisy-quadratic SGD model predicts full loss curves across LR schedules via a two-term formula involving the schedule integral and the preconditioned gradient covariance trace $\text{tr}\,\Sigma$; a single fitted constant $c$ reproduces empirical curves across model sizes, schedules, and training horizons [§3.2, Eq. 16, Fig. 6].
• The preconditioned gradient covariance trace under µP appears schedule-dependent but weakly model-size-dependent, verified empirically on regression and next-token prediction — this is the ingredient that transfers constant-LR collapse to arbitrary decay schedules [§3.2.3, Eq. 17, Fig. 7].
• Supercollapse variance-reduction magnitude scales with the incremental optimization noise between time $t$ and the end of training, not the cumulative noise; LR decay directly suppresses this incremental term, quantitatively matching the noisy-quadratic prediction [§3.3, Eq. 24, Fig. 8].
• Replacing µP with a constant LR across widths breaks collapse even at small scales where final losses look indistinguishable, so collapse is a more sensitive probe of proper scaling than final loss alone [§2.6, Fig. 4 top].
• When the data exponent $\zeta$ deviates from the compute-optimal value, larger models make faster early relative progress and decelerate later, producing a systematic downward shift of the normalized curve — offering a novel way to tune $\zeta$ by maximizing collapse quality rather than fitting Pareto-frontier points [§2.6, Fig. 4 bottom].

Method

The setup is a scaling ladder of models trained compute-optimally: for each task (transformer next-token prediction on CIFAR-5M and Lichess; 7-layer MLPs on power-law Fourier features), the authors estimate $T^*(N) = kN^{\alpha/\beta}$ from the constant-LR Pareto frontier and reuse it as the horizon under other LR schedules. Losses are affine-rescaled by subtracting the fitted irreducible loss $L_\infty$ and dividing by the per-seed final reducible loss $L_{\text{final}}(N,\zeta) - L_\infty$, then plotted against training fraction. Collapse quality is quantified by $\Delta(\tau)^2 = \text{Var}_{N,\zeta}[\tilde{L}]/\text{E}_{N,\zeta}[\tilde{L}]^2$ and compared against the per-model noise floor $\sigma(\tau)^2 = \text{E}_N[\text{Var}_\zeta[\tilde{L}]/\text{E}_\zeta[\tilde{L}]^2]$.

Theoretically, the paper first proves collapse for sum-of-power-law loss laws under compute-optimal token allocation, then extends via a noisy-quadratic SGD/gradient-flow model that predicts the expected loss as forcing_function + Σ EMA_i(η·tr Σ) — where the forcing function is schedule-independent and the excess loss is a schedule-dependent EMA of gradient noise. Under an approximation where $\text{tr}\,\Sigma$ tracks the loss itself with weak N-dependence (verified empirically), the paper derives Eq. 16 which predicts target losses from a reference constant-LR trajectory using a single fit parameter. The same model then gives the variance-reduction formula Eq. 24 that explains supercollapse quantitatively.

Results

• On CIFAR-5M transformers (~10M–80M params), Lichess transformers, and Fourier MLPs, collapse deviation quickly saturates at $\Delta \approx \sigma$ under constant LR and drops well below $\sigma$ once LR decay is turned on [Fig. 1, Fig. 3].
• The Eq. 16 predictor with a single fitted $c$ reproduces empirical loss curves across three schedules and multiple model widths on CIFAR-5M; larger MLP experiments give even sharper agreement [Fig. 6, App. I].
• Constant-LR loss curves are well fit by $L(t,N) = A/N^\alpha + B/t^\beta + L_\infty$ on CIFAR-5M with the fitted exponents that also yield the best power-law Pareto frontier; deviating from the optimal data exponent breaks both the fit and collapse simultaneously [Fig. 5, Fig. 4 bottom].
• Under µP, collapse holds; under a constant LR across widths (mis-scaling), collapse fails even at small scales — the paper positions this as a precise scaling diagnostic strictly stronger than comparing final losses [Fig. 4 top].
• Depth scaling (rather than width) produces similar collapse [App. B], supporting the conjecture that collapse persists under joint width/depth/batch/weight-decay scaling ladders — the setup Scaling with Collapse: Efficient and Predictable Training of LLM Families later tests at LLM scale.

Why it’s interesting

This is the paper that puts whole training-loss-curve prediction on the wiki. Everything filed so far under Hyperparameter scaling laws has been about picking scalars — $\eta^*(N,D)$, $\tau^*$, TPP — from a scaling law. Qiu et al. show that under proper compute-optimal scaling, the loss trajectory itself is universal, and that supercollapse can identify mis-parameterization earlier and more sensitively than final-loss comparisons — a diagnostic sharper than any of the point-scalar tools currently on the page. The noisy-quadratic derivation shares DNA with the SDE reparameterization line in The Design Space of Tri-Modal Masked Diffusion Models and directly complements Universal One-third Time Scaling in Learning Peaked Distributions‘s “the slope is structural” thesis: both argue key training-dynamics quantities are set by a small number of coarse controls (softmax/CE for the slope; power-law compute-optimality + LR schedule for the whole curve shape) rather than by details of architecture or optimizer.

See also

• Scaling with Collapse: Efficient and Predictable Training of LLM Families — extends this paper to full LLM training with joint width/depth/batch/weight-decay scaling and adds AdamW timescale $\tau$ as the third essential control.
• Hyperparameter scaling laws — Fitting/Transfer/Enforce three-way fork; this paper’s collapse diagnostic is a cross-cutting fourth axis that measures whether any of the three succeeded.
• Training stability at scale — µP mis-parameterization breaks collapse; the collapse-residual diagnostic is directly usable as a stability signal.
• How to Set the Learning Rate for Large-Scale Pre-training? — the Fitting-paradigm paper this line complements; Qiu et al.’s collapse is a validation criterion for whether a fit $\eta^*(N,D)$ actually preserves dynamics-consistency across scales.
• The Design Space of Tri-Modal Masked Diffusion Models — same noisy-quadratic / SDE toolkit applied to tri-modal MDM design space; both papers use SDE analysis to derive scale-invariance predictions.
• Universal One-third Time Scaling in Learning Peaked Distributions — softmax-CE structural universality of the slope; complements Qiu et al.’s universality of the shape.