Universal One-third Time Scaling in Learning Peaked Distributions

TL;DR

The paper argues that the slow power-law convergence of LLM pre-training loss is not a property of data or model size per se, but a structural consequence of pairing softmax with cross-entropy when the target distribution is peaked (as next-token distributions are). Through a toy student–teacher model with a closed-form analysis and empirical checks against real LLM training curves, the authors derive a universal time-scaling exponent of 1/3 for the loss — invariant to most architectural and data interventions. This gives a mechanistic origin story for the “tyranny of the slope”: interventions like optimizer choice (Muon), data mixes, and architecture tweaks shift the intercept of the scaling law but not its slope, because the slope is set by the softmax-CE bottleneck itself.

Key claims

Loss exhibits a universal power-law-in-time decay with exponent 1/3 — i.e. $L(t) \propto t^{-1/3}$ — when learning peaked target distributions under softmax + cross-entropy [Abstract; §1].
The 1/3 exponent is derived analytically from a toy student–teacher model whose student weights become aligned to the teacher’s, with a closed-form prediction matching numerics [Fig. 1, Fig. 3].
Softmax and cross-entropy together are the load-bearing primitive: replacing CE with MSE / KL or removing the softmax non-linearity breaks the 1/3 prediction (this is the missing piece in prior dynamic-view scaling analyses that used MSE losses) [Fig. 3; §1 discussion of Bordelon/Pehlevan 2021 line of work].
The mechanism is a vanishing-gradient effect: as the student approaches a peaked teacher distribution, softmax outputs saturate and the cross-entropy gradient decays as a power law in time, producing the slow convergence [§1].
The toy model’s “aligned student ansatz” — student weight proportional to teacher weight — is empirically observed in actual LLM training, validating the mechanism beyond the toy [Fig. 2].
Since training time is proportional to dataset size in the single-pass / fixed-batch regime, time-based and data-based scaling laws are equivalent here, so the 1/3 exponent transfers directly to the data-scaling slope [§1, citing Michaud et al. 2023; Bordelon et al. 2024–2025].

Method

The paper builds a toy student–teacher setup where the teacher emits a peaked categorical distribution (modeling the empirical sharpness of next-token distributions) and the student is trained with softmax + cross-entropy under SGD. Under the “aligned student” ansatz (student weight proportional to teacher weight, observed in actual LLM training), the dynamics reduce to a tractable ODE whose loss decays as $t^{-1/3}$ . The authors then check the prediction empirically against LLM training curves and against ablations that swap CE for MSE / KL-divergence or perturb the softmax — these break the 1/3 prediction in the expected way. The construction connects to but is distinct from prior dynamic-view scaling-law analyses (Bordelon & Pehlevan 2021; Bordelon et al. 2024/2025) which all worked with MSE loss and therefore missed the softmax-CE specific bottleneck.

Results

Theoretical: closed-form prediction $L \propto t^{-1/3}$ for the toy model under aligned-student dynamics, matching numerics [Fig. 3].
Empirical: actual LLM training curves are consistent with the 1/3 exponent and exhibit the aligned-student weight geometry [Fig. 2].
Ablation: swapping CE for MSE or KL changes the exponent — the 1/3 universality is specific to the softmax-CE pairing, not a generic consequence of NN training [Fig. 3, KL-divergence comparison].
Predictive implication: interventions that don’t touch the softmax-CE pairing (optimizer, data, most architecture tweaks) can shift the prefactor of $L \propto t^{-1/3}$ but cannot change the exponent — operationally, they move the intercept, not the slope, of the log-log scaling-law fit [framing throughout].

Why it’s interesting

This is a structural origin story for a phenomenon the team has felt empirically for a while: a long list of pre-training interventions — Muon, careful data mixes, modern normalization recipes — all shift the loss-vs-tokens curve down by some intercept but leave its slope alone. If the slope really is set by the softmax-CE combination, then “shifting the slope” requires touching the output-distribution primitive itself: alternative losses (MSE, KL, contrastive), non-softmax output layers, or training targets that are less peaked. It also reframes recent stability-via-architecture results — see Training stability at scale, where outlier-driven rescaling and QK-Norm appear to be doing implicit work — as interventions on the intercept family, not the slope. The most direct implication for Luma is that diffusion-style objectives (MSE / flow-matching velocity regression) live in a different universality class than LLM softmax-CE pretraining, which may be part of why DiT loss curves behave qualitatively differently.