Pre-training under infinite compute

TL;DR

A Stanford CRFM study (Kim, Kotha, Liang, Hashimoto) asking how to pretrain a language model when data is fixed and compute is unconstrained — the regime web text growth (8%/yr) lags compute growth (400%/yr). The standard recipe (more epochs + more parameters) overfits and stops improving; once weight decay is re-tuned up to ~30× standard practice, loss becomes monotonically decreasing in parameter count with a clean power law. Independently ensembling models gives a lower asymptote than scaling a single model, and the two recipes compose: the joint scaling recipe (parameter scaling × ensemble scaling) is 5.17× more data-efficient than the unregularized baseline at 200M tokens, with data-scaling-law extrapolation predicting the improvement persists at higher token budgets. Distillation lets a single 300M student retain ~83% of an 8-ensemble’s benefit, removing the inference-compute cost.

Key claims

Standard data-constrained recipes (increase epoch count, increase parameter count) eventually overfit: at fixed 200M seed tokens, loss starts increasing past a certain epoch count or past a certain parameter count, bounding what hyperparameter-tuning alone can recover — contradicting Muennighoff et al.’s monotone-in-epoch-count scaling-law functional form [§2, Fig. 2].
The optimal weight decay for over-parameterized data-constrained pretraining is ~30× standard practice: $\lambda \in \{0.8, 1.6, 3.2, 3.2\}$ for $\{150\text{M}, 300\text{M}, 600\text{M}, 1.4\text{B}\}$ -param models, vs the GPT-3 default of 0.1; with this, loss follows a clean power law $L(N) = E + A/N^\alpha$ in parameter count, $\alpha \approx 0.296$ [§3, Fig. 3].
Asymptote of the regularized power law as $N \to \infty$ is $E_\text{reg} = 3.43$ on the 200M-token corpus; the asymptote is the right success metric under infinite compute, not loss at a fixed compute budget [§3, Fig. 3].
Independently training $K$ models on the same data and averaging logits (ensembling) yields a lower asymptote ( $E_\text{ens} = 3.34$ as $K \to \infty$ ) than parameter scaling — meaning that at sufficiently large total parameter count, two 300M models beats one 600M model; even a 2-member 300M ensemble outperforms the regularized recipe’s asymptote [§4, Fig. 4].
Ensembling and parameter scaling compose: taking the joint limit $K \to \infty$ , $N \to \infty$ gives $E_\text{joint} = 3.17$ , a substantial improvement over both the unregularized recipe ( $E_\text{std} = 3.75$ ) and the regularized recipe ( $E_\text{reg} = 3.43$ ) [§4.3, Fig. 6].
Ensemble hyperparameters are not the same as single-model hyperparameters — the best single-model regularized run is not the best ensemble member; ensembles benefit from members each being slightly more overfit (more epochs, less weight decay), supporting the “multi-view feature learning” interpretation [§4.2, Fig. 5].
Data efficiency: at 200M tokens, the joint scaling recipe is 5.17× more data-efficient than the standard recipe; the regularized recipe alone is 2.29× more efficient. Data-scaling-law extrapolation across {200M, 400M, 800M, 1.6B} seed token counts predicts these gains persist at higher token budgets [§5].
Ensemble distillation transfers ~83% of the ensemble benefit to a single 300M student, outperforming the regularized recipe’s $N \to \infty$ asymptote with finite parameter count at inference [§6.1, Fig. 9].
Self-distillation improves a 300M model: distilling a single 300M teacher into a fresh 300M student of identical architecture, mixing real + synthetic tokens, beats the teacher and the best standard 300M run — without ever training a model of higher parameter count [§6.2, Fig. 9].
Downstream benchmarks track validation loss: the best ensemble outperforms the best unregularized model by +9% averaged over PIQA / SciQ / ARC Easy; on continued pretraining of MegaMath-Web-Pro (math mid-training), ensembling reaches default-CPT performance with 17.5× less data [§7].

Method

The setup is a 200M-token DCLM seed corpus (extended in §5 to 400M / 800M / 1.6B), Llama-style decoder-only autoregressive transformer, AdamW with cosine LR schedule, context length 4096. The three recipes are: Standard (jointly tune learning rate and epoch count at each parameter count via coordinate-descent local optimization; report the best); Regularized (additionally tune weight decay — the load-bearing knob — over a wide range, finding the local optimum where increasing or decreasing any hyperparameter degrades validation loss); Ensembling (independently train $K$ models with different data orders and initializations, then average per-token logits at generation/eval time, counting total params as $K \cdot N$ for comparison). Each recipe’s “best possible loss under infinite compute” is estimated as the asymptote of the fitted power law: $E_\text{reg} = \lim_{N \to \infty} L(N)$ , $E_\text{ens} = \lim_{K \to \infty} L(K \mid N)$ , $E_\text{joint} = \lim_{N \to \infty} \lim_{K \to \infty} L(K, N)$ . Data scaling is studied by repeating the full procedure at four seed-token budgets and fitting a second-tier scaling law of asymptote-vs-tokens. Distillation uses sequence-level knowledge distillation (Kim & Rush 2016): the teacher unconditionally samples $\beta T$ synthetic tokens that are mixed with the original $T$ real tokens, the student is trained from scratch on the mixture.

Results

200M-token corpus headline asymptotes [§3, §4]: standard $E_\text{std} = 3.75$ ; regularized $E_\text{reg} = 3.43$ (Δ = −0.32 from regularization alone); ensembling at fixed 300M base $E_\text{ens} = 3.34$ (Δ = −0.09 vs regularized asymptote, despite the regularized recipe extrapolating to infinite parameter count); joint scaling $E_\text{joint} = 3.17$ (Δ = −0.58 from baseline, Δ = −0.26 from regularized). Concrete configurations: best regularized 300M uses LR 3e-3, 16 epochs, weight decay 1.6; best 1.4B uses LR 1e-3, 8 epochs, weight decay 3.2 [§3, Table]. Data efficiency at 200M tokens: regularized 2.29×, joint scaling 5.17× over standard; the best 1.4B regularized model alone is 2.06×, the best 5-member 1.4B ensemble is 5.10× [§5]. Downstream evals: best ensemble +9% over best unregularized on PIQA / SciQ / ARC Easy average; math-mid-training CPT: ensembling at 4B tokens beats default CPT at 73B tokens (17.5× data efficiency) [§7]. Distillation: 8-ensemble 300M → 300M student retains 83% of the ensemble benefit, even outperforming the regularized recipe’s $N \to \infty$ asymptote [§6.1]. Self-distillation: a 300M student distilled from a 300M teacher beats the teacher [§6.2] — the surprising “model collapse is avoidable when real + synthetic data are mixed” finding.

Why it’s interesting

This paper is the empirical recipe behind what Deep Learning is Not So Mysterious or Different argues theoretically: a flexible (over-parameterized) hypothesis space combined with a soft inductive bias (here, much stronger weight decay than standard practice) achieves better generalization than restriction-based recipes — exactly the soft-bias claim from Wilson. The “weight decay should be ~30× higher” finding lands directly on top of the wiki’s Hyperparameter scaling laws cluster: Why Gradients Rapidly Increase Near the End of Training locates an instability in the weight-decay term at end-of-training; QK-Clip: Taking Muon Further on the Scaleup Journey and How To Scale argue for principled weight-decay schedules; this paper says the level is wrong even before you worry about the schedule. The composition of ensembling × parameter scaling × distillation is also the strongest filed empirical defense of the Distillation Scaling Laws thesis — distillation from large teachers is what pushes the loss-vs-parameter tradeoff curve to the left, especially when the teacher is itself an ensemble. The self-distillation result ([§6.2]) extends the synthetic-data-doesn’t-collapse-if-you-mix-it-with-real-data story already on file in Synthetic Data Powering Pretraining (UC Berkeley EE 290/194-11 Lecture 11a) and FineWeb-C: A Community-Driven Dataset for Educational Quality Annotations in 122 Languages. Useful for the “we have lots of compute and not enough data” planning question that comes up around every pretraining decision.