Internal Data Repetition Destroys Language Models

TL;DR

Chudnovsky, Kazdan, Levi & Schaeffer et al. (Stanford, Tel Aviv, IMC) modernize Hernandez et al. 2022’s repeated-data study for the Chinchilla era. They hold the repeated-token fraction fixed at 10% and vary only the repetition structure — how a fixed 10% of the token budget is split between pool size (unique repeated documents) and per-document repeat count. Across Qwen3-style models from 34M to 344M trained on FineWeb-Edu-Dedup, eval loss is non-monotonic in repeat count: a moderately sized pool replayed a moderate number of times damages performance more than either endpoint. The peak location follows a power law in model size — larger models are damaged most by fewer repeats of a larger pool. Translated to Compute-Equivalent Loss via a fitted no-repetition Chinchilla scaling law, the worst repetition structure at 344M costs ~33% of training compute (matches a no-rep run that used only 67% of the FLOPs). A closed-form misspecified linear-regression model reproduces the same non-monotonic peak, suggesting the phenomenon is a generic statistical feature of duplicated samples rather than a transformer-specific artifact.

Key claims

At fixed repeated-token fraction (10%), eval loss is non-monotonic in the per-document repeat count $R$ : an intermediate $R$ maximizes damage; both extremes (tiny pool × many repeats and large pool × few repeats) produce smaller loss increases [§4.1, Fig. 2].
The location of the loss peak scales as a power law in model size. Fitted peak locations shift from $R^\ast \approx 32$ at 34M (repeated pool $\approx 40$ M tokens) to $R^\ast \approx 4.8$ at 344M (repeated pool $\approx 700$ M tokens); larger models are hurt most by fewer repeats of a larger pool [§4.2, Eqs. 4–5, Fig. 3].
Small loss gaps map to large compute gaps because the no-repetition Chinchilla curve is shallow. A 2–4% loss increase translates into Compute-Equivalent Gain values of $CEG \approx 0.67$ –0.85; at 344M the worst repetition structure costs ~33% of training compute [§4.3, Fig. 5].
The peak location in $R$ is approximately invariant to overtraining multiplier $M$ ; only the level of $CEG$ shifts with $M$ , so the worst-case repetition structure persists across training durations [§4.3, Fig. 5].
A closed-form misspecified linear regression with verbatim duplicates reproduces the same non-monotonic peak in test loss vs. repeated-pool size, and the peak shifts to larger pools as capacity grows — the statistical analogue of the empirical language-model scaling law [§4.4–§4.5, Fig. 6].
The excess-loss curve in the simulated linear model has a matching sample-efficiency version of $CEG$ that falls sharply at intermediate pool size and partially recovers at extreme sizes, mirroring the language-model curve of Fig. 5 [§4.5, Fig. 7].

Method

Qwen3-style decoder-only transformers are trained at six parameter counts (34M–344M) on FineWeb-Edu-Dedup. For each $(N, M)$ pair (model size, overtraining multiplier), the total compute $C \approx 6NMD_0$ is fixed; 10% of the token budget goes to repeated documents and 90% to unique documents, disjoint from both the evaluation split and each other. The repeated pool size $U$ and per-document repeat count $R$ trade off as $U = 0.1 \cdot MD_0 / R$ , so varying $R$ concentrates the same 10% budget onto a smaller pool. Repeat counts are swept on an approximately logarithmic grid from no-repeats to $R \approx 10^4$ . A no-repetition baseline is trained at each of the six $N$ to fit a three-parameter Chinchilla law $L(C) = E + A/C^\alpha$ ; loss increases from repetition runs are inverted through this law to recover Compute-Equivalent Gain and Compute-Equivalent Loss. Peak locations $\hat R^\ast(N)$ are extracted from log-Gaussian fits over eval loss vs. $\log R$ . The closed-form statistical model is misspecified OLS on observed features $x_1$ with unobserved-feature signal $x_2$ ; the shared $x_2$ realization across repeated copies produces a block-diagonal noise covariance whose repeated block is a rank-one $R \cdot \mathbb{1}\mathbb{1}^T$ — the extra factor of $R$ is what makes duplication qualitatively different from adding independent samples.

Results

The headline number: on Qwen3-style 344M at $M=1$ on FineWeb-Edu-Dedup, the worst-case repetition structure (a repeated pool of ~700M tokens replayed ~4.8× each) reaches the same loss as a no-repetition run trained with only ~67% of the FLOPs — a Compute-Equivalent Loss of 33%. Peak eval-loss increases over the no-repetition baseline are 0.9%–3.7% (median 2.4%) but map to $CEG$ values 0.67–0.85 because the no-repetition curve is shallow. Peak prominence over the endpoints (no-repeats and largest $R$ ) is 0.5% to 3.5% (median 1.7%). The fitted scaling law $R^\ast \propto N^{-0.66}$ (or equivalently, worst-case pool size $U^\ast \propto N^{1.66}$ ) implies the memorization capacity of models grows faster than compute — at large enough scale the peak crosses $R^\ast = 1$ , at which point the qualitative story becomes “any repetition is worst.” The closed-form linear model reproduces the peak in simulation to within numerical precision and recovers the shift of the peak to larger pool sizes as capacity increases.

Why it’s interesting

Complements Scaling Domain Data Repetition in LLM Pretraining (the sibling paper posted in the same LinkedIn thread) which asks the benefit question — how many repeats each domain tolerates at fixed TPP — while this paper asks the damage question at fixed repeated-token fraction. Together the two sharpen a picture the wiki has so far only touched through Lilian Weng’s survey (Scaling Laws, Carefully) and Muennighoff et al.’s “up to 4 epochs is free” result: it’s not enough to know how much of your corpus is duplicated — the structure of duplication (pool size × repeat count) has an intermediate worst case whose location scales predictably with $N$ .

Methodologically, the Compute-Equivalent Gain framing pushes the wiki’s Hyperparameter scaling laws page’s small-loss-gap-vs-large-compute-gap observation into a repetition-damage setting: the no-repetition Chinchilla curve is shallow enough that a 2–4% loss bump is a 15–33% compute cost. The closed-form linear-regression analogue also suggests that scaling-law diagnostics like “duplication produces a non-monotonic peak” may generalize beyond LLMs to any misspecified statistical model with repeated samples — a prediction directly testable on the diffusion-pretraining side where duplicate detection is currently perceptual-hash based rather than compute-cost aware.