A Bitter Lesson for Data Filtering

TL;DR

Stanford scaling study (Mohri / Duchi / Hashimoto) that targets the high-compute, data-scarce pretraining regime and inverts the standard “high-quality data filtering is essential” wisdom: with enough compute, the best filter is no filter. New scaling experiments show that sufficiently trained large-parameter models tolerate — and in fact benefit from — low-quality and distractor data that small-scale studies would have filtered out. The framing is explicitly Sutton’s bitter lesson applied to data curation: hand-designed quality filters win at small scale but are overtaken by scale-graceful no-filter baselines.

Key claims

Standard data-filtering practice is calibrated on small-scale experiments and may not survive when extrapolated to the high-compute, data-scarce regime that frontier pretraining now operates in [Abstract].
In the high-compute regime, the best data filter on the experiments studied is no data filter — sufficiently trained large-parameter models do not just tolerate “poor” data, they benefit from it [Abstract].
The phenomenon is framed as an instance of Sutton’s bitter lesson: human-designed filters that win at small scale are eventually replaced by simple, compute-leveraging no-filter approaches [Abstract, paper framing].
Distractor / low-quality data has positive marginal value at large model scale on the studied pretraining setup, not merely zero marginal value [Abstract].

Method

The paper is a scaling study on data filtering for large-model pretraining, designed specifically for the high-compute, data-scarce corner of the design space — the regime current frontier pretraining is moving into as scaled compute outpaces high-quality token supply. The experimental protocol contrasts filtered vs no-filter pretraining at multiple model/compute scales and reads off the cross-over point at which the no-filter baseline overtakes the filtered one. (The abstract is the only material returned by the fetcher; the full §-level method and exact filter recipes / scales / corpora are not summarized here — see paper.)

Results

Filtering to “high-quality only” underperforms a no-filter baseline once compute is large enough on the studied setup [Abstract].
Large-parameter sufficiently-trained models benefit from nominally poor data — the gradient w.r.t. low-quality data fraction is positive at scale, not just non-negative [Abstract].
(Headline numbers per setting and the exact scale at which the no-filter baseline takes over are not captured by the abstract pulled at filing — see paper for the per-scale figures.)

Why it’s interesting

This is the cleanest “negative” result so far on the wiki’s Synthetic Training Data thesis that the curation gate is the dominant lever. The line of papers filed on that page — Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training, Shaping capabilities with token-level data filtering, Smol Training Playbook (GPU MODE talk on SmolLM3), Data Darwinism Part I: Unlocking the Value of Scientific Data for Pre-training — all argue, in different shapes, that more curation (mixture search, token-level masking, processing-depth ladders, synthetic-corpus filtering) compounds with scale. This paper argues the opposite: the regime they were calibrated in is the small-scale regime, and in the high-compute / data-scarce regime, the curation gate is exactly the human-designed-heuristic that the bitter lesson tells you to throw out. It is also a direct match for Terrance’s framing in the Slack thread — “more motivation for training a big MoE and throwing everything into it” — which is the operational implication: if filters lose to compute, the data-side play at frontier scale is to ingest broadly and let parameter count + tokens absorb the noise. Conflict-check: this directly contradicts the CLIMB / token-filtering / Data-Darwinism claims that filtering scales with model size, and that tension needs to be reflected on the Synthetic Training Data page (see edit below).