Skip to content
Bud's Wiki

Data Darwinism Part I: Unlocking the Value of Scientific Data for Pre-training

TL;DR

Section titled “TL;DR”

Part I of the GAIR-NLP “Data Darwinism” framework: a ten-level taxonomy (L0 raw acquisition → L9 full ecosystem synthesis) that organizes every pretraining data operation by processing depth. L0–L2 are conventional acquisition and rule-based filtering; L3–L5 add classifier-based quality filtering, generative refinement, and cognitive completion (making implicit reasoning explicit); L6–L9 are the theoretical frontier of contextual/environmental/ecosystem synthesis. The framework is operationalized in daVinci-LLM: Towards the Science of Pretraining (daVinci-LLM), where every datum in the 7.5T-token corpus is tagged with its Darwin Level so processing decisions are auditable and reusable.

Key claims

Section titled “Key claims”
  • Processing depth is a complementary scaling axis to data volume — L3 (model-based filtering) gives modest gains on basic tasks (+3.4 MBPP), L4 (generative refinement) gives substantial gains on complex reasoning (+7.0 MATH), L5 (cognitive completion) gives strong in-domain alignment but limited transfer [companion-repo §3.3 / §Ablations].
  • Pretraining data operations split cleanly along a “what changes about the text” axis: format (L1), inclusion (L2–L3), surface content (L4), latent reasoning structure (L5), and the generation of new context, environments, or ecosystems (L6–L9) [§Framework, Table mapping L0–L9].
  • Annotating every datum with its Darwin Level makes a heterogeneous corpus (public datasets + in-house processing) auditable: downstream researchers can reuse / re-mix at the level they trust [§Data release].
  • Cognitive completion (L5) — expanding compressed logical steps into explicit chains — is positioned as a distinct operation from generative refinement (L4) and from synthetic-data generation; it operates on existing text by making implicit content explicit, not by inventing new content [§L5 definition].

Method

Section titled “Method”

Data Darwinism is a taxonomy and accompanying processing pipeline, not a single model or algorithm. The ten levels are organized so that each successive level operates on the output of the previous level and modifies a strictly broader scope: L0 (acquisition) → L1 (format normalization) → L2 (rule-based filtering: dedup, malformed text, non-target languages) → L3 (model-based filtering: classifier-judged educational value and domain relevance) → L4 (generative refinement: remove structural noise, repair content while preserving semantics) → L5 (cognitive completion: surface implicit reasoning) → L6–L9 (higher-order synthesis at the contextual / environmental / ecosystem level; described as theoretical frontier in this Part I release).

Every token in the released 7.5T-token corpus carries the Darwin Level of its deepest processing step, which functions as both provenance and a re-use handle: a researcher who trusts L3 but distrusts L5 can drop the L5 fraction and retrain.

Results

Section titled “Results”

The framework itself does not report numbers — its empirical validation lives in daVinci-LLM: Towards the Science of Pretraining (daVinci-LLM 3B matches OLMo-3 7B at 51.72 overall, with MATH 62.80 vs 39.60). The repo’s ablation summary, attributed to this taxonomy, quantifies per-level contributions: L3 filtering +3.4 MBPP, L4 refinement +7.0 MATH, L5 synthesis strong in-domain.

Why it’s interesting

Section titled “Why it’s interesting”

This is the most explicit attempt yet on the wiki to taxonomize pretraining-data operations rather than treat them as an unstructured grab-bag. It dovetails with two existing threads. First, Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training (CLIMB) automates mixture-weight search over real web data; Data Darwinism is orthogonal — it taxonomizes processing depth of each source. The two compose: CLIMB’s clustering would happily ingest Darwin-Level-tagged data. Second, Synthetic Training Data already covers L5/L6-style generation as the dominant 2025–2026 trend; Data Darwinism reframes this as one rung on a longer ladder, with L4 (refinement of existing text) and L5 (making implicit reasoning explicit) as distinct, often more transferable, intermediate operations. The Smol Training Playbook (GPU MODE talk on SmolLM3) (SmolLM3) playbook collapses everything under “data curation”; this paper supplies vocabulary for the per-source-per-level ablations that playbook implicitly conducts.

See also

Section titled “See also”