Scaling Laws for Neural Language Models

TL;DR

The foundational Kaplan et al. paper on empirical scaling laws for autoregressive Transformer language models. Test loss decreases as clean power laws in three factors — non-embedding parameter count $N$ , dataset size $D$ , and compute $C$ — with trends spanning over seven orders of magnitude and depending only weakly on model shape (depth, width, attention heads). Larger models are dramatically more sample-efficient, so under a fixed compute budget the optimal recipe is to train a very large model on a modest amount of data and stop significantly before convergence — the “Kaplan” prescription that Chinchilla later revised. The critical batch size (from An Empirical Model of Large-Batch Training) is shown to depend on the loss alone, growing as a power law as loss decreases.

Key claims

Test cross-entropy loss is a power law in each of $N$ (non-embedding params), $D$ (tokens), and $C_\mathrm{min}$ (optimally-allocated compute) when the others are not bottlenecking: $L(N) = (N_c/N)^{\alpha_N}$ , $L(D) = (D_c/D)^{\alpha_D}$ , $L(C_\mathrm{min}) = (C_c^\mathrm{min}/C_\mathrm{min})^{\alpha_C^\mathrm{min}}$ , with trends spanning over six orders of magnitude and no sign of deviation on the upper end [§1.1, §3, Fig. 1, Eqs. 1.1–1.3].
Performance depends very weakly on architectural hyperparameters (depth, width, attention heads, feed-forward dimension) once total non-embedding parameter count is fixed — the shape-invariance result that justifies treating $N$ as the primary architecture axis [§3.1, Fig. 5].
A single equation $L(N, D) = [(N_c/N)^{\alpha_N/\alpha_D} + D_c/D]^{\alpha_D}$ governs the joint dependence and predicts overfitting; every 8× increase in model size requires only ~5× more data to avoid a penalty [§4, Eq. 1.5, §4.2].
Learning curves follow predictable power laws whose parameters are roughly model-size-independent: $L(N, S_\mathrm{min}) = (N_c/N)^{\alpha_N} + (S_c/S_\mathrm{min})^{\alpha_S}$ ; early parts of a training curve can be extrapolated to predict late-training loss [§5, Eq. 1.6].
Under a fixed compute budget, the optimal allocation strongly favors model size: $N_\mathrm{opt} \propto C^{0.73}$ , $B_\mathrm{opt} \propto C^{0.24}$ , $S_\mathrm{opt} \propto C^{0.03}$ , and $D_\mathrm{opt} \propto C^{0.27}$ [§1.2, §6, Eq. 1.7, Fig. 3]. Data grows very slowly with compute in this framing — the “Kaplan” prescription that predates and disagrees with Chinchilla’s revised $N^* \propto C^{0.5}$ , $D^* \propto C^{0.5}$ .
The critical batch size is a function of loss alone, not of model size: $B_\mathrm{crit}(L) \approx B_* / L^{1/\alpha_B}$ with $B_* \approx 2 \cdot 10^8$ tokens and $\alpha_B \approx 0.21$ , approximately 1–2 million tokens at convergence for the largest models trained [§1.1, §5.1, Eq. 5.3, Fig. 10]. This validates and extends An Empirical Model of Large-Batch Training‘s prediction for the specific case of Transformer LMs.
Transfer to out-of-distribution text incurs a roughly constant offset in loss but otherwise tracks in-distribution performance — improving on a different distribution requires only improving the training-set fit [§3.2.2, Fig. 8].
LSTMs match Transformers on tokens appearing early in the context but fall behind on later tokens, and cannot close the gap by scaling parameters alone [§3.2.1, Fig. 7].
Sample efficiency improves with model size — larger models reach a given loss on fewer examples, which is the mechanism behind the “train very large, stop early” prescription [§6, Fig. 19].

Method

Trains decoder-only Transformer LMs on WebText2 (a 22B-token web-scraped extension of GPT-2’s training corpus), varying (i) model size $N$ from 768 to 1.5B non-embedding parameters, (ii) dataset size from 22M to 23B tokens, (iii) shape parameters ( $n_\mathrm{layer}$ , $d_\mathrm{model}$ , $d_\mathrm{ff}$ , $d_\mathrm{attn}$ , $n_\mathrm{heads}$ ), (iv) context length (mostly 1024), and (v) batch size. Uses Adam (Adafactor for models >1B) with a 3000-step linear warmup then cosine decay, cross-entropy averaged over 1024-token contexts as the primary metric.

Parameterizes model size by non-embedding parameters $N \approx 2\,d_\mathrm{model}\,n_\mathrm{layer}(2 d_\mathrm{attn} + d_\mathrm{ff})$ and compute by $C \approx 6NBS$ FLOPs (forward+backward on $B$ -sized batches over $S$ steps). Introduces the critical batch size $B_\mathrm{crit}(L)$ from the empirical relation $(S/S_\mathrm{min} - 1)(E/E_\mathrm{min} - 1) = 1$ (adapted from An Empirical Model of Large-Batch Training) and defines the batch-size-adjusted step count $S_\mathrm{min}$ and adjusted compute $C_\mathrm{min}$ that would obtain if all runs used batch $B \ll B_\mathrm{crit}$ .

Fits the single joint equation $L(N, D)$ using three principles: (i) vocabulary-rescaling covariance, (ii) correct asymptotic behavior at $D \to \infty$ (recovers $L(N)$ ) and $N \to \infty$ (recovers $L(D)$ ), and (iii) analyticity in $1/N$ . Extrapolates the compute-optimal frontier by fitting $S_\mathrm{min}(N)$ against $C_\mathrm{min}$ across the swept configurations and solving for $N_\mathrm{opt}(C)$ .

Results

Three power laws with fitted exponents: $\alpha_N \approx 0.076$ , $\alpha_D \approx 0.095$ , $\alpha_C^\mathrm{min} \approx 0.050$ (nats), with critical scales $N_c = 8.8 \times 10^{13}$ , $D_c = 5.4 \times 10^{13}$ , $C_c^\mathrm{min} = 3.1 \times 10^8$ PF-days [§3, Eqs. 1.1–1.3, Fig. 1].
Joint fit: $L(N, D)$ four-parameter fit with $\alpha_N \approx 0.076$ , $\alpha_D \approx 0.103$ , $N_c$ and $D_c$ constants; excellent match across all sweeps except the smallest 22M-token dataset (where overfitting starts within 40 updates) [§4.2, Fig. 9].
Compute-optimal recipe (Kaplan-2020): $N_\mathrm{opt} \propto C^{0.73}$ , $D_\mathrm{opt} \propto C^{0.27}$ — model size grows much faster than data with compute [§1.2, §6, Eq. 1.7]. (Note: Hoffmann et al. 2022’s Chinchilla revision put both exponents near 0.5, making this the “wrong” recipe of the two — but the paper as filed is the origin of the Kaplan scaling framework that all subsequent work either follows or contrasts against.)
Critical batch size: $B_\mathrm{crit}(L) \approx (2.1 \times 10^8) / L^{1/0.21}$ tokens, ranging from ~1M tokens at convergence for the largest models trained to smaller values earlier in training [§5.1, Fig. 10, Eq. 5.3]. Model-size independence at fixed loss is the sharper claim — the [MKAT18] prediction confirmed on Transformers.
Early-stopping prediction from finite data: $S_\mathrm{stop} \gtrsim S_c \cdot [L(D)-L(\infty, \infty)]^{-1/\alpha_S}$ , tested against actual overfitting behavior [§5.3, Fig. 16, Eq. 5.7].
LSTM ablation: no matter the parameter count, LSTMs cannot match Transformer performance on late-context tokens; increasingly large powers on the context-position curve for larger Transformers imply improved fast-pattern-recognition [§3.2.1, Fig. 7].

Why it’s interesting

This is the paper that operationalized the “Just make it bigger” empirical thesis that shaped 2020–2022 LLM development, and its critical-batch-size result is exactly what Sid’s Slack post is asking about: CBS depends on loss alone, grows steeply as loss falls, and is model-size-independent at fixed loss. That last point is the load-bearing claim for taking McCandlish’s cross-domain gradient-noise-scale statistic and turning it into a portable recipe — if it holds for Luma’s diffusion/flow objectives, then a small-scale CBS measurement at one loss level extrapolates to the target loss with the fitted power. The “Kaplan-2020 recipe favors model size” vs “Chinchilla favors data” revision is the other famous contribution of this paper’s methodology and the frame the whole Hyperparameter scaling laws concept debates. Complements An Empirical Model of Large-Batch Training (co-filed) by moving from a cross-domain intuition to an LM-specific quantitative fit, and sets the methodology that Scaling Laws For Diffusion Transformers (DiT isoFLOP), The Design Space of Tri-Modal Masked Diffusion Models (tri-modal MDM Kaplan-form fit), and How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size (three-term $L(N, T, B)$ ) all inherit and revise.