Skip to content
Bud's Wiki

daVinci-LLM: Towards the Science of Pretraining

TL;DR

Section titled “TL;DR”

A fully-open 3B pretraining recipe from GAIR-NLP that operationalizes the Data Darwinism Part I: Unlocking the Value of Scientific Data for Pre-training framework (L0–L9 data taxonomy) on a 7.5T+ token corpus, using a dynamically-monitored adaptive two-stage curriculum (6T general → 2T structured-QA, with a 70%-QA Stage-2-2 phase). Headline result: daVinci-LLM-3B reaches 51.72 average, matching OLMo-3 7B (51.65) despite half the parameters, and substantially beating it on MATH (62.80 vs 39.60). Releases include the final checkpoint, all intermediate checkpoints, the full 7.5T-token corpus with per-datum Darwin Level annotations, and the documented results of 200+ controlled ablations including failed experiments.

Key claims

Section titled “Key claims”
  • A 3B model trained on systematically-curated data matches OLMo-3 7B (51.72 vs 51.65 overall on a 5-task aggregate) [Repo “Key Results” table].
  • The MATH gap to OLMo-3 7B (+23.20 points: 62.80 vs 39.60) is attributed to L4-refined and L5-completed reasoning content in the corpus, not to architectural changes [Repo §3.3, Tech report §4].
  • Capability maturation is decoupled across domains: general-knowledge benchmarks saturate around ~1T tokens of Stage-1 training, while code and science reasoning continue improving past 4T tokens — motivating progressive reallocation toward reasoning-intensive domains within Stage 1 [Repo §Stage 1, Fig “train-traj”].
  • A format shift (introducing structured QA in Stage 2) unlocks growth beyond what mixture rebalancing alone achieves; Stage 2-2 (intensify to 70% QA after balanced Stage 2-1) yields a +12.14 aggregate gain over Stage 1’s end state [Repo §Stage 2].
  • Extreme specialization triggers collapse: jumping directly to a 70%-QA mix without the balanced Stage 2-1 foundation degrades retention of general capabilities — a “balance first, then intensify” rule [Repo §Q3 ablation].
  • Evaluation protocol matters and can produce ranking reversals: PPL-based and generative evaluations disagree on high-QA-trained models; the paper recommends reporting multiple protocols for a complete capability profile [Repo §Q4 ablation].
  • Per-level data-processing contributions, isolated: L3 model-based filtering +3.4 MBPP, L4 generative refinement +7.0 MATH, L5 cognitive completion strong in-domain alignment but limited transfer [Repo §Q1 ablation, Tech report §4].

Method

Section titled “Method”

Three pillars, jointly released:

  1. Data: 7.5T+ tokens, every datum tagged with its Darwin Level (L0–L9 per Data Darwinism Part I: Unlocking the Value of Scientific Data for Pre-training). Combines public datasets (passed through deeper processing where appropriate) with in-house corpora. Annotation makes the corpus reusable at any processing depth.

  2. Training: Adaptive two-stage curriculum. Stage 1 (6T tokens) builds broad foundations; benchmark probes during training reveal that general knowledge saturates around 1T while code/science keep growing past 4T, so the mix is progressively reallocated toward reasoning-heavy domains within the stage. Stage 2 (2T tokens) introduces structured QA: Stage 2-1 balances domains to stabilize, Stage 2-2 intensifies to 70% QA for targeted amplification.

  3. Science: 200+ controlled ablations conducted as verifiable research questions — processing-depth contribution per level (Q1), capability-maturation curves and mix adaptation (Q2), specialization vs forgetting (Q3), evaluation-protocol reliability (Q4). Negative results and failed experiments are released alongside the positive ones.

The deliverables — model, all intermediate checkpoints, full corpus with Darwin Level metadata, technical report with the full ablation set — are pitched as an OLMo-style reproducibility package rather than just a model release.

Results

Section titled “Results”
CapabilitydaVinci-3BOLMo-3 7BLLaMA-3.2-3BQwen-2.5-3B
Overall51.7251.6537.5851.44
General Knowledge52.9655.1351.0855.16
Code Generation55.9954.4232.4056.13
Scientific Reasoning48.3045.9822.4544.65
MATH62.8039.609.0037.20

(Source: Repo “Key Results” table.) The MATH delta is the largest single result — daVinci-3B beats OLMo-3 7B by 23.20 points on MATH and Qwen-2.5-3B by 25.60. The Stage-2 +12.14 aggregate gain comes from format-shift + QA intensification, not from architecture.

Why it’s interesting

Section titled “Why it’s interesting”

This is the most legible “full-stack open pretraining recipe” filed since Smol Training Playbook (GPU MODE talk on SmolLM3) (SmolLM3), and it sits in interesting tension with that playbook’s conclusion. SmolLM3’s argument is that data-curation is the dominant return — daVinci-LLM agrees but goes further: it claims the axes of data curation (processing depth per L0–L9, capability-aware mix adaptation, format shift, QA intensification) are themselves separately ablatable and that explicit ablation is the difference between matching a 7B-parameter peer at 3B and not. Three specific connections worth noting: (i) the dynamic-mix-adaptation story complements Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training (CLIMB) — CLIMB automates the mix search; daVinci runs it as an in-training feedback loop driven by per-domain benchmark probes; (ii) the “balance first then intensify” Stage-2 finding parallels Replaying pre-training data improves fine-tuning (data replay during fine-tuning) — both push back against the naive “more of the target distribution, faster” intuition; (iii) the PPL-vs-generative ranking-reversal observation echoes On Data Engineering for Scaling LLM Terminal Capabilities‘s concern that protocol choice can flip dataset rankings, and is the most explicit pretraining-side report of that effect on the wiki.

See also

Section titled “See also”