How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text

TL;DR

Russell et al. pretrain 800 LMs varying the ratio of wild AI-generated web text (Pangram-labeled, ~27.5–31.1% of post-FineWeb-filtered June–Aug 2026 Common Crawl) to human text, and fit scaling laws to held-out loss on both. For data-starved models, a small amount of AI text initially lowers loss on human text, but the benefit saturates and reverses into harm as more is added; for human-text-rich runs, AI tokens raise loss immediately while fresh human tokens keep lowering it. They propose a new scaling law with separate benefit and harm terms whose sum can change sign, reducing to Chinchilla in the absence of AI text — predicts the effect on held-out human-text loss for models up to 3.6× larger with 41% lower error than the best existing law. Release the WildAI 83B-token corpus (with AI, topic, format labels), all 800 models, and code.

Key claims

27.5% of tokens in June 2026 Common Crawl (after FineWeb quality filtering) are Pangram-labeled AI-generated, rising to 31.1% by August — unfiltered scale of the “wild” AI-text problem on current pretraining corpora [Abstract].
Wild AI text is a distinct regime from the synthetic-data / model-collapse setups prior work studied: it comes from many models, is written for human readers, arrives unlabeled in pretraining corpora, and is not self-distillation [Abstract].
At low human-text budgets, adding AI tokens initially lowers held-out human-text loss, but the benefit saturates and reverses into harm as the AI fraction grows — a sign-flipping marginal value for the same token source within a single run [Abstract].
At high human-text budgets, AI tokens raise held-out human-text loss immediately while the same number of additional human tokens continues to lower it — AI tokens are strictly dominated by fresh human tokens in the data-rich regime [Abstract].
Existing scaling laws (specifically Hoffmann et al. 2022 / Chinchilla) do not predict this behavior — their closed-form fits fail to capture the sign flip [Abstract].
Proposed scaling law has separate benefit and harm terms so the net value of an AI token can change sign across the (N, D, AI-ratio) surface, and reduces to Chinchilla when the AI fraction is zero [Abstract].
When fit on smaller models, the new law predicts the effect of AI text on held-out human-text loss for models up to 3.6× larger with 41% lower error than the best existing law, measured across all AI ratios tested [Abstract].
AI tokens remain valuable when the target evaluation is itself AI text — the asymmetry is specifically between held-out human and held-out AI distributions, not a universal “AI data is bad” claim [Abstract].
Practical recommendations (stated by the authors): filter AI text when the target is human text; repeat human text before expanding the training dataset with AI-generated web text; report validation loss on human and AI text separately [Abstract].
Release: WildAI, an 83B-token corpus with AI / topic / format labels; all 800 pretrained models; code [Abstract].

Method

The paper is an 800-model controlled pretraining sweep designed to isolate the effect of AI-generated web text at scale. First, Pangram is run over FineWeb-filtered Common Crawl dumps from June and August 2026 to measure the AI fraction (27.5% → 31.1%) and to construct WildAI — an 83B-token corpus labeled by AI-generated status plus topic and format metadata. The models are then pretrained varying (model size, training tokens, ratio of added AI tokens to human tokens), and held-out loss is measured separately on human text and on AI text for every configuration. The authors fit scaling laws to the resulting (N, D, AI-ratio) → loss surface, show that the Chinchilla additive form cannot reproduce the sign flip in marginal AI-token value, and introduce a two-term “benefit + harm” scaling law whose net can go negative in regimes where Chinchilla cannot. The law is validated by fitting on smaller models and extrapolating to models up to 3.6× larger. The full §-level fit form and the per-domain Pangram false-positive-rate characterization are not retrievable from the abstract — see paper.

Results

Pangram labels 27.5% of FineWeb-filtered June 2026 web tokens as AI-generated, rising to 31.1% by August 2026 — the ~2-month trend already in the data at filing time is strongly upward [Abstract].
In the data-starved regime, added AI tokens → initial loss decrease on held-out human text, then saturation, then increase beyond a crossover AI-fraction (specific fraction not quoted in abstract) — within-regime sign flip [Abstract].
In the data-rich regime, AI tokens → immediate loss increase on human text; same count of fresh human tokens → continued loss decrease [Abstract].
Chinchilla-form fits give substantially larger extrapolation error than the proposed benefit+harm form; the paper reports 41% lower error for models 3.6× larger than the fitting-ladder top under the new law, across all AI ratios [Abstract].
AI tokens remain beneficial when evaluated on held-out AI text — the asymmetry is between the two eval distributions, not a global signal [Abstract].
(Per-ratio and per-scale numeric tables / law coefficients are not captured in the abstract — see paper for the figures that back the 41% / 3.6× claims.)

Why it’s interesting

This is the first filed scaling law that treats the current wild AI-text contamination of web data as a first-class axis with its own benefit and harm terms, rather than lumping it into either the “synthetic data is a magic multiplier” or “model collapse is doom” buckets. Three concrete connections:

It directly contradicts the position of A Bitter Lesson for Data Filtering in a specific regime: Mohri/Duchi/Hashimoto’s “no filter beats filter at large compute” bitter-lesson claim is a data-starved + distractor-data regime result, and this paper measures exactly the same regime and reports that AI tokens do help there — before reversing into harm as the AI fraction grows. The sign-flipping-within-regime finding re-frames the filter-vs-no-filter debate as “filter only when you are no longer data-starved” rather than Sutton-shaped filters-always-lose. The Synthetic Training Data page should flag this as a sharpening of its “curation gate dominates returns” thesis, not a contradiction of either side.
It complements the data-repetition scaling line (Internal Data Repetition Destroys Language Models, Scaling Domain Data Repetition in LLM Pretraining, Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data) by giving the fourth axis they do not cover: human-vs-AI token origin, independent of document-level repetition count. The practical recommendation “repeat human text before expanding with AI web text” is a direct inheritance of Hernandez-era repetition rules into the AI-contamination regime.
The two-term benefit+harm scaling law that reduces to Chinchilla when AI-fraction = 0 is the first filed example of a scaling-law generalization on the data-quality axis — most entries on Hyperparameter scaling laws add axes for architecture (expert-vs-attention ratio, loop count, vocabulary) or optimizer (µA, SSO, AdamH); this one adds an axis to the data term itself.