Scaling Domain Data Repetition in LLM Pretraining

TL;DR

Li et al. study high-quality domain-data repetition in LLM pretraining under a practical constraint: as models scale, the tokens-per-parameter (TPP) budget must scale too, but high-quality domain data (code, math, curated sources) is harder to scale than general web data. Repeating the scarce high-quality slice counteracts this dilution but risks overfitting. Across a controlled sweep, they find (a) at fixed TPP the optimal repetition count mildly increases with model size, (b) the optimal repetition count is strongly negatively correlated with a domain’s final validation loss — “easier” domains tolerate more repetition — and (c) the amount of unique domain data is only weakly predictive of the optimal repetition count. The practical implication is that repetition counts tuned on smaller proxy models at the same TPP transfer to larger models.

Key claims

• At fixed tokens-per-parameter (TPP), the optimal per-domain repetition count increases mildly with model size, so a proxy-model repetition sweep at the same TPP transfers to larger runs [Abstract, §Results].
• The optimal repetition count is strongly negatively correlated with the domain’s terminal validation loss — lower-loss (“easier”) domains benefit from more repetitions than higher-loss domains at matched budget [Abstract].
• The amount of unique domain data is only weakly related to the optimal repetition count, meaning “how much data you have” is a poor proxy for “how many times to replay it” [Abstract].
• The paradigm assumes training-token budget scales proportionally with model size (fixed TPP), which is the practical regime for scaling frontier runs and differs from earlier fixed-token-budget repetition studies (Hernandez et al. 2022) [Abstract].

Method

The setup fixes TPP (tokens per parameter) across scales and, for each domain, sweeps the per-domain repetition count in the training mixture. A no-repetition baseline is trained at each TPP as reference. Repetition counts are varied independently per domain rather than uniformly across the whole corpus, so “how many times to replay math” and “how many times to replay code” are separate design variables. The correlation between optimal repetition count and final validation loss is measured across multiple domains at each scale; the transfer claim is tested by taking repetition counts optimal at a small model and evaluating them at a larger model at the same TPP.

Results

The paper’s headline results are qualitative fits over a scaling sweep rather than a single scalar: the optimal-repetition-count-vs-model-size curve rises gently under fixed TPP; the optimal-repetition-count-vs-domain-loss curve falls sharply; and the optimal-repetition-count-vs-unique-data-volume relationship is essentially flat. The intended deliverable is a proxy-model recipe: fit repetition counts on a small model at the target TPP, then reuse them at scale.

Why it’s interesting

Sits alongside Internal Data Repetition Destroys Language Models as the “positive” half of the current data-repetition scaling debate: Li et al. ask “how much repetition can each domain benefit from?” at fixed TPP, while Kazdan et al. ask “what repetition structure damages the model most?” at fixed repeated-token fraction. Rylan Schaeffer’s LinkedIn framing (the pointer that filed both papers) explicitly positions them as complementary: knowing the amount of repetition a domain tolerates isn’t enough — you also need to know how that repetition is structured (pool size × repeat count), and both axes shift with model scale.

Extends the data-axis of the wiki’s Chinchilla-style scaling-law literature (Scaling Laws, Carefully, How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size, Delphi: Scaling Laws That Extrapolate 300× Past the Fit) with a domain-heterogeneous repetition axis that most prior work either averages out (uniform-epoch Chinchilla) or holds fixed (single-corpus fits). The proxy-model transfer claim is the actionable piece — analogous to Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning‘s “µA lets you transfer a LoRA-rank sweep to FFT” for hyperparameters.

See also

• Internal Data Repetition Destroys Language Models — the sibling paper posted alongside this one; measures the damage side (worst-case repetition structure) rather than the benefit side (optimal per-domain repetition count).
• Hyperparameter scaling laws — the concept page this contributes a domain-repetition axis to.
• Scaling Laws, Carefully — canonical reference for the Kaplan/Chinchilla data-axis background that this paper’s TPP-fixed framing extends.
• How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size — proposes a three-term scaling law that separates training steps and batch size; conceptually adjacent to breaking out repetition as its own axis.