OpenThoughts: Data Recipes for Reasoning Models

TL;DR

OpenThoughts is a fully open dataset/model release plus a 1,000+-experiment ablation study of the SFT data-curation pipeline for reasoning models in math, code, and science. The authors sweep each pipeline stage (question sourcing, source mixing, question filtering, deduplication × multi-answer sampling, answer filtering, teacher model) independently at 31.6K samples on Qwen2.5-7B-Instruct, then scale the winning recipe to 1.2M examples to produce OpenThinker3-7B. The resulting 7B SFT-only model beats DeepSeek-R1-Distill-7B by 12.4 points average across 12 reasoning benchmarks and edges out the next-best open-data baseline (Nemotron-Nano-8B) by 2.1 points. Several findings cut against common intuition — concentrating on the top 1–2 sources beats diversity-mixing, answer-filtering of any kind gives no significant gain, and QwQ-32B is a stronger teacher than DeepSeek-R1 despite scoring lower on the eval benchmarks.

Key claims

Sampling multiple answers per question from the teacher (1× → 4× → 16×) is an effective scaling axis on top of question diversity — sweeping deduplication × answer-sampling jointly, the best math configuration is exact-dedup + 16× answers/question, and the chosen code configuration is no-dedup + 16× answers [§4.4, Table 6].
Concentrating on the top 1–2 question sources beats mixing the top 4/8/16 across all three domains; for code, top-2 sources beats top-16 by ~5 average points (41.3 vs 36.4) [§4.2, Table 4].
LLM-based question filtering (difficulty-based for code; response-length-based for math/science) outperforms classical embedding- or fastText-based filters and improves over random by 4–6 average points; stronger filtering LLMs (GPT-4.1-mini) beat weaker ones (GPT-4.1-nano) [§4.3, Table 5].
Answer filtering of any flavor (GPT verifier, shortest-answer, longest-answer, non-English removal, fastText classifier) does not significantly outperform no-filtering on the same compute, so the final pipeline skips it [§4.5, Table 7].
QwQ-32B is a strictly stronger teacher than DeepSeek-R1 for SFT, yielding +1.9 / +2.6 / >5 average points for code / math / science respectively, despite DeepSeek-R1 itself scoring higher on CodeElo (+9), GPQA Diamond (+8), and JEEBench (+23) — teacher benchmark score and teacher utility for SFT are decoupled [§4.6, Table 8].
Scaling the stacked best-per-stage pipeline from 316 → 31.6K samples shows monotonic improvement and a strong positive correlation between data scale and downstream score, motivating the 1.2M scale-up [§5, Fig. 3].
OpenThinker3-7B reaches 69.0 AIME24, 53.3 AIME25, 53.7 GPQA-Diamond, 64.5 LCB 05/23–05/24 and 51.7 LCB 06/24–01/25 — +15.3 / +20.5 / +17.2 vs DeepSeek-R1-Distill-Qwen-7B on three of its main benchmarks, and a 12.4-point 12-task average improvement [§1, Table 1].
The progression OpenThoughts-114K → 2-1M → 3-1.2M shows compounding gains from data-recipe-only changes: AIME24 14.3 → 29.3 → 60.7 → 69.0 on Qwen2.5-7B-Instruct, same base model and architecture throughout [§3, Table 2].

Method

The pipeline has six stages — question sourcing, question mixing, question filtering, deduplication + multi-answer sampling, answer filtering, and teacher selection — each ablated independently while holding the others at a fixed default. For each ablation, 31,600 samples (a log-midpoint between 10K and 100K) are generated, used to SFT Qwen2.5-7B-Instruct, and scored by the average across eight in-domain benchmarks (AIME24, AMC23, MATH500, CodeElo, CodeForces, LCB 05/23–05/24, GPQA-Diamond, JEEBench); generalization is monitored on a held-out set (AIME25, HMMT 02/25, HLE-MCQ, LCB 06/24–01/25). At each stage the best strategy is locked in and carried forward. The default teacher during ablations is DeepSeek-R1; the final pipeline swaps in QwQ-32B [§4]. The final OpenThoughts3-1.2M dataset is then produced by scaling the stacked winners to 1.2M samples and SFT-ing Qwen2.5-7B-Instruct to yield OpenThinker3-7B [§5].

Results

Headline benchmark numbers at 7B: 55.3 average (12 tasks) for OpenThinker3-7B vs 42.9 for DeepSeek-R1-Distill-Qwen-7B (+12.4), 47.3 for NemoNano-1M (+8.0), 42.1 for AM-1.4M (+13.2), 47.2 for OpenR1-Distill-7B (+8.1) — and 53.2 / 52.9 / 51.6 for the SFT/RL-hybrid Nemotron-Nano-8B, AceReason-7B, and Skywork-7B [Table 1].
Per-source filtering examples: for code, CodeGolf (StackExchange) and OpenCodeReasoning are the top sources (27.5 and 25.3 code-avg); for math, OpenMath-2-Math and NuminaMath-1.5 dominate (58.8 and 58.5 math-avg); for science, StackExchange-Physics and an LLM-extracted organic-chemistry-PDF corpus win (43.2 and 45.3 science-avg) [§4.1, Table 3].
Teacher swap (QwQ-32B vs DeepSeek-R1 vs Phi-4-Reasoning-Plus): for code-domain SFT, 44.2 vs 42.3 vs 29.0 average; for math-domain SFT, 44.2 vs 41.6 vs 30.6. Phi-4-Reasoning-Plus is much weaker as a teacher despite a comparable parameter count [§4.6, Table 8].
Data scaling sweep: stacking winners at each stage produces a steeper scaling curve than the no-curation baseline, with continued gains through 31.6K and motivating the 1.2M scale-up [§5, Fig. 3].

Why it’s interesting

This is the strongest filed evidence to date that data-recipe ablations — not RL, not new architectures — are the load-bearing piece of small-model reasoning gains: same 7B base, same SFT recipe, four generations of dataset, AIME24 climbs 14.3 → 69.0. It directly extends Synthetic Training Data‘s “curation gate as load-bearing piece” thesis from the in-loop multi-agent setting (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing) and the dense-labeling setting (Action100M: A Large-scale Video Action Dataset) into the cleanest controlled-ablation regime yet. It also pushes back on three intuitions held by adjacent papers: (a) the “more diversity is better” mixing assumption (Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training optimizes a 21-cluster mixture; here the winning code config uses 2 sources), (b) the “verify everything” assumption (answer filtering gives no gain), and (c) the “best benchmark score = best teacher” assumption (Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling argued for weaker-but-cheaper teachers at fixed compute; OpenThoughts is the orthogonal observation that even at equal compute, the higher-scoring teacher is not the better teacher). The strong SFT-only baseline also reframes the SFT-vs-RL story in Reasoning RL — OpenThinker3-7B beats SFT+RL pipelines (Nemotron-Nano-8B, AceReason-7B, Skywork-7B) on average, suggesting the data ceiling for SFT is higher than the RL community has been assuming.