The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data

TL;DR

The paper argues against the standard pretrain-then-finetune split for domain specialization. Instead of reserving the small proprietary dataset for finetuning, mix it into pretraining at a small fraction (often 2–5% of tokens, repeated 10–50×) — specialized pretraining (SPT). Across ChemPile, MusicPile, and ProofPile, SPT→FT beats NPT→FT on domain test loss, retains general capability better, and on ProofPile lets a 1B SPT model surpass a 3B NPT model. The paper also derives overfitting scaling laws that decompose test loss into a power-law training loss (negative exponent) and a power-law train–test gap (positive exponent), letting practitioners pick the optimal mixture fraction from cheap pilot runs.

Key claims

SPT (repeating a 2–5% domain mixture during pretraining for 10–50 epochs) followed by finetuning beats NPT→FT on domain test loss across all three studied corpora: MusicPile 2.0%, ProofPile 1.5%, ChemPile 0.8% relative gain [§2.2, Fig. 4].
The post-finetuning improvement holds simultaneously on the domain task and on general (Dolma) loss: SPT models enter finetuning from a lower domain loss, require less adaptation, and therefore forget less of the general distribution [§2.3, Fig. 5(a)].
SPT reduces pretraining tokens needed to reach a fixed post-finetuning domain loss by up to 1.75× [Abstract, §2.2 “compute multiplier”, Fig. 4 center].
On the furthest domain from web text (ProofPile), a 1B SPT model closes 133% of the gap between 1B and 3B NPT — i.e., surpasses the 3B model entirely. On MusicPile it closes 81%; ChemPile 23% [§2.2, Fig. 4 right].
The mechanism is regularization, not optimization: with general data filling each pretraining batch, the model tolerates far more domain-data repeats before the train–test gap opens than during finetuning, where the batch is pure domain data [§2.4, Fig. 6].
SPT’s relative gain grows when the domain is under-represented in the pretraining corpus, established via a controlled English↔Japanese translation sweep that varies Japanese monolingual fraction from 10% → 0.1% and observes Δ₍SPT₎ rising from ~2% to ~5% [§3.1.1, Fig. 7].
Among standard distributional-similarity metrics (unigram/bigram/trigram JSD, MAUVE, C2ST), only post-finetuning NPT domain loss correctly ranks all three benchmark domains by realized Δ₍SPT₎ [§3.1.2, Appendix E].
Domain dataset size determines whether early integration helps: at 300M tokens SPT helps across all pretraining scales; at 3M–30M tokens excessive repetition flips the sign at large pretraining budgets, and specialized continued pretraining (introducing the domain at the last 20B of pretraining) is preferable [§3.2, Fig. 8 left].
SPT’s gain widens with model size — the 3B SPT–NPT gap on MusicPile-300M is larger than the 1B gap; the recipe scales [§3.3, Fig. 8 right].
The compute-optimal mixture fraction depends on pretraining horizon: large fractions (50%) win below ~30B tokens; moderate (5–10%) win at 50B; small (2%) win at 200B; in every regime some SPT beats NPT [§3.4, Fig. 9].
SPT→FT consistently beats NPT→FT(replay), even at 10% replay during finetuning — replay during finetuning is not a substitute for early domain exposure [§4, Fig. 10].
The proposed overfitting scaling law decomposes test loss as train_loss(T) + (train_test_gap)(T), fitting each as a separate power law: training loss with a negative exponent (T₋ᵅ), train–test gap with a positive exponent (T₊ᵝ). This avoids the nested nonlinearities of prior repeated-data scaling laws and extrapolates the overfitting regime from small runs [§5.1, §5.2].
Practical rule from the scaling law: at a given pretraining compute budget, predict optimal f (domain mixture fraction) without a full training sweep — useful for tuning a single SPT recipe from a small number of pilot runs [§5.3].
Headline reframing — “the finetuner’s fallacy”: finetuning a large off-the-shelf model is not the cheapest path to domain adaptation when amortized over inference, because a smaller SPT model matches a larger NPT+FT model’s domain performance and is cheaper at serving time [§1, Fig. 3].

Method

The controlled OLMo-1B sandbox pretrains for 200B tokens with mixture fraction f of MusicPile / ChemPile / ProofPile (each ≈300M tokens) and 1−f of Dolma. Finetuning is on the same domain dataset with a WSD schedule and early stopping on test loss. Variants at 300M / 600M / 1B / 3B parameters are produced by adjusting OLMo depth + MLP/hidden dims. Headline metric is relative gain — (L_NPT − L_SPT) / L_NPT of best post-FT domain test loss across all finetuning configurations.

Distributional-similarity is studied two ways: (i) a controlled En↔Ja sweep over 160M LLaMA models trained on 20B FineWeb2 tokens of mixed En/Ja monolingual data plus 1B parallel JParaCrawl tokens, varying Japanese fraction from 0.1% to 10%; (ii) a correlation analysis on the three benchmark domains comparing five candidate metrics (n-gram JSD, MAUVE, C2ST, NPT post-FT loss) against realized Δ₍SPT₎.

The overfitting scaling law fits L_test(T; f) = L_train(T; f) + Δ(T; f). The training-loss component is a standard A·T^{−α} decay; the train–test gap is B·T^{+β} growing with pretraining tokens because the domain corpus is repeated more times. Both α(f), β(f), A(f), B(f) are fit from a small grid of (T, f) pilot runs, then extrapolated.

Results

OLMo-1B + 200B tokens, three domains: relative gain Δ = 2.0% (MusicPile), 1.5% (ProofPile), 0.8% (ChemPile) [§2.2, Fig. 4 left].
Compute multiplier vs NPT to reach the NPT 200B-token best post-FT loss: ≥1.5× on MusicPile and ProofPile, smaller but >1× on ChemPile [§2.2, Fig. 4 center].
1B SPT vs 3B NPT: gap closure 133% / 81% / 23% on ProofPile / MusicPile / ChemPile [§2.2, Fig. 4 right]. ProofPile result means 1B SPT strictly beats 3B NPT.
Downstream task accuracy: at 200B pretraining tokens, SPT improves MATH (MCQA) by up to 6 pp and MusicTheoryBench by up to 4 pp over NPT→FT [§2.3, Fig. 5(b)].
Replay ablation: at 0–10% replay rate of Dolma during finetuning, NPT→FT(replay) closes part of the gap to SPT→FT (no replay), but never matches it — SPT→FT remains best across the replay grid [§4, Fig. 10].
Dataset-size regime flip: at 3M and 30M domain-token corpora, full SPT overfits at large pretraining budgets; specialized continued pretraining (last-20B-of-pretraining domain injection) recovers gains at 30M tokens and partially recovers at 3M [§3.2, Fig. 8].

Why it’s interesting

Sam’s posting note is the headline: “the concept of pre/mid/post training is a red herring. It’s all just training. Let the model see your most important tokens earlier than later.” This paper is the strongest controlled evidence on the wiki for that frame — it dissolves the pretrain/finetune phase boundary and shows that position in the training trajectory of a token (not which “phase” calls it) is the load-bearing variable.

This complements Replaying pre-training data improves fine-tuning (Kotha & Liang) exactly. That paper found that replaying generic data into finetuning improves the target task — i.e., bring general data later into the trajectory. This paper finds that bringing domain data earlier into pretraining also wins. Both treat the training trajectory as one continuous data-schedule and both argue the standard NPT→FT phase split is suboptimal. They land on opposite-sign interventions (generic-into-finetuning vs domain-into-pretraining) that are mechanistically the same insight — distribution overlap at every step of the optimizer trajectory matters, and the typical sequence breaks it. Replaying pre-training data improves fine-tuning §4 also tests its replay recipe against early domain integration as the reference, and finds replay alone cannot substitute for early domain exposure — which is the exact result Section 4 of this paper independently confirms from the other direction.

It also dovetails with Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training (CLIMB) and Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training (DeMix), which optimize which data to mix at pretraining time but hold the schedule fixed. This paper holds the data fixed but optimizes when. A joint sweep — CLIMB-style mixture optimization over (data, schedule) jointly — is the obvious follow-up, and Shaping capabilities with token-level data filtering adds the orthogonal negative-curation axis (mask tokens to suppress a capability). Together these four papers are the strongest pointer on the wiki to a future pretraining-data-mixture concept page.

The overfitting scaling law is a non-trivial methods contribution by itself: prior repeated-data scaling laws (Kaplan, Muennighoff, Goyal) nest a decaying exponent inside the power law, which is hard to fit from limited runs. Decomposing L_test = L_train + Δ and fitting two separate power laws — a positive-exponent power law for the gap — is a cleaner functional form that should generalize beyond SPT. This is adjacent to How to Set the Learning Rate for Large-Scale Pre-training?‘s Fitting paradigm (quadratic-in-log-LR × power-law-in-D) and to The Design Space of Tri-Modal Masked Diffusion Models‘s Kaplan-form fits over 262 isoFLOP runs.