Scaling Laws, Carefully

TL;DR

A long-overdue Lil’Log post by Lilian Weng walking through scaling laws as a tool for compute allocation between model size $N$ and dataset size $D$ before committing to a large run. Covers the early empirical foundations (Amari 1992, Hestness 2017, Rosenfeld 2020), the Kaplan vs. Chinchilla disagreement and its reconciliation, why power laws appear in the first place, and the data-limited regime where data repetition and overfitting penalties enter the fit. Closes with practical warnings on how seemingly trivial fitting choices — loss precision, summing vs. averaging the Huber loss, fit-region selection — can swing extrapolated predictions by orders of magnitude. Useful as a single citable survey when arguing about token budgets at Luma’s scale.

Key claims

Generalization error in deep learning has been empirically observed to follow a power-law $\epsilon \sim c \cdot D^{-\alpha} + E$ across at least four problem domains (NMT, image classification, LM, ASR), with architecture shifting the offset $E$ but not the exponent $\alpha$ — i.e. the slope is a property of the problem domain, not the model [§Early days].
The Kaplan et al. (2020) joint fit $L(N, D)$ implies overfitting depends predominantly on the ratio $N^{\alpha/\beta}/D$ , and prescribes $N_\text{opt} \propto C^{0.73}$ — grow model size faster than data, and stop large models before convergence [§Kaplan].
The Chinchilla recipe (Hoffmann et al. 2022) instead prescribes $N_\text{opt} \propto C^{0.5}$ — double tokens for every doubling of model size — and is validated by three independent fitting methods (fix- $N$ vary- $D$ ; isoFLOP profiles; parametric fit) that all agree, plus the Chinchilla-vs-Gopher head-to-head where a 4× smaller model trained on 4× more tokens outperforms Gopher [§Chinchilla].
Pearce & Song (2024) reconcile Kaplan and Chinchilla by accounting for embedding-parameter fraction: in the small- $N$ regime embeddings are non-negligible and the local exponent $g$ on non-embedding $N$ rises toward Kaplan’s 0.73, while at large $N$ it converges to Chinchilla’s 0.5 [§Reconciling].
Power-law shapes in LLM loss have two main families of explanation: the data-manifold partitioning hypothesis (Sharma & Kaplan 2020 — a model partitioning a $d$ -dimensional manifold into $O(N)$ regions has typical resolution $\sim N^{-1/d}$ ) and the quantized-skills hypothesis (Michaud 2023, Brill 2024 — skills are discrete, frequency-distributed as a power law, learned common-first) [§Why power law?].
In the data-limited regime, Hernandez et al. (2022) show that mixing in repeated data produces a double-descent in test loss as the repeated fraction grows, and that fixed-fraction repetition hurts OOD evaluation and downstream fine-tuning [§Scaling Laws in Data-Limited Region].
Muennighoff et al. (2023) propose an effective-data correction $D' = U_D + U_D r_D(1 - \exp(-R_D/r_D))$ where each repetition costs a $(1-1/r_D)$ fraction of the token’s remaining value, with a symmetric form for excess model size; their empirical fit gives $r_N < r_D$ — spend the budget on more epochs rather than more parameters [§Scaling Laws in Data-Limited Region].
Lovelace et al. (2026) revisit data-constrained scaling and instead model overfitting explicitly via a penalty term built on the capacity ratio $N/U_D$ — fitted on ~300 models (15M–1B params, 50M–6B unique tokens) — finding that larger models are more sensitive to data repetition and that strong weight decay reduces the overfitting penalty [§Scaling Laws in Data-Limited Region].
The Chinchilla method-3 parametric fit was found by Besiroglu et al. (2024) to be biased by procedural choices: averaging (vs summing) Huber loss values led to premature L-BFGS-B termination and implausibly narrow CIs, and the reported $\alpha, \beta$ were rounded to 2 digits [§Trickiness].
Loss-precision rounding, milli-loss noise, and choice of fit-region (small/medium/all models) each visibly shift the fitted exponents in a toy simulation built around Besiroglu et al.’s re-estimated ground truth $N_\text{opt} \propto C^{0.5126}, D_\text{opt} \propto C^{0.4874}$ [§Toy simulation].

Method

The post is a survey/tutorial, not a new empirical result. The synthesis Weng adds beyond restating prior work is twofold. First, an organizing taxonomy that splits scaling-law work into a data-infinite region (Kaplan, Chinchilla, the Pearce-Song reconciliation) and a data-limited region (Hernandez on repetition double-descent; Muennighoff on effective-data discounting; Lovelace on a capacity-ratio overfitting penalty), making explicit that the two regions ask different questions and use different parametric forms. Second, a “trickiness” section that walks through how the Chinchilla method-3 fit was off (Huber-loss summation, rounded exponents) and provides a toy widget demonstrating that loss precision, milli-loss noise, and fit-region choice each shift the fitted exponents enough to change extrapolated predictions.

Results

No new numbers; the post is a survey. The most useful quantitative anchors it consolidates: Kaplan’s $N_\text{opt} \propto C^{0.73}$ vs Chinchilla’s $N_\text{opt} \propto C^{0.5}$ ; Chinchilla’s training-token recipe of ~20 tokens per parameter; Besiroglu et al.’s re-estimate $N_\text{opt} \propto C^{0.5126}, D_\text{opt} \propto C^{0.4874}$ after fixing the Huber-loss summation bug; Muennighoff et al.’s finding that excess-parameter half-life $r_N$ is shorter than excess-data half-life $r_D$ (spend on epochs, not parameters); Lovelace et al.’s ~300-model sweep (15M–1B params, 50M–6B unique tokens) showing larger models are more sensitive to data repetition.

Why it’s interesting

This is the cleanest single-source primer to drop into onboarding when someone needs to argue about token budgets — every result Luma’s pretraining team cites in slack threads (Kaplan, Chinchilla, Pearce-Song, Besiroglu, Muennighoff, Lovelace) is consolidated in one place with the disagreements named explicitly. It directly complements Hyperparameter scaling laws by giving the canonical $(N, D, C)$ -axis story that the concept page’s optimizer-axis (Fitting/Transfer/Enforce) and architecture-axis (Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design‘s $r^*$ ) entries are extensions of. It also sets up the data-wall framing for the data-limited entries on Diffusion Language Models — Weng cites the same Muennighoff et al. (2023) result that Diffusion Beats Autoregressive in Data-Constrained Settings extends to show MDMs tolerate ~100 epochs of repetition vs ~4 for AR LMs, so the next obvious question for the wiki is whether Lovelace et al.’s capacity-ratio penalty also reshapes the MDM-vs-AR data-tolerance gap.