Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling

TL;DR

At a fixed sampling-FLOPs budget for generating synthetic reasoning data, sampling many solutions from a weaker, cheaper (WC) model beats sampling fewer from a stronger, more expensive (SE) one across three finetuning paradigms — knowledge distillation, self-improvement, and a new “weak-to-strong improvement” (W2S-I) where a strong student learns from a weaker teacher. The WC data has higher coverage and diversity but also a higher false-positive rate (correct answer, wrong reasoning); empirically the coverage/diversity wins dominate. The result inverts the common practice of always sampling from the strongest available model and scales up to Gemini-1.5-Pro vs 1.5-Flash on MATH.

Key claims

At a fixed FLOPs budget, the WC/SE sample ratio scales linearly with the parameter ratio (inference FLOPs ≈ 2·P·T), so a ~3× smaller WC model can be sampled ~3× more times per problem at parity [§3, Eq. 1].
Gemma2-9B (WC) data has higher coverage and diversity than Gemma2-27B (SE) data at compute parity on MATH and GSM-8K, but a higher false-positive rate measured both by human raters and by Gemini-1.5-Pro as automatic verifier [§5.1, Fig. 3].
Student-LM finetuning (Gemma-7B trained on WC vs SE data) — distillation from WC beats distillation from SE on MATH and GSM-8K at both low and high sampling budgets [§5.2, Fig. 4, Fig. 5].
WC-LM finetuning — self-improvement on WC’s own data beats knowledge distillation from SE data, contradicting Singh et al. 2023, who used number-matched (not compute-matched) sampling [§5.2, §5.3].
SE-LM finetuning — training Gemma2-27B on Gemma2-9B-generated data (W2S-I) beats training Gemma2-27B on its own self-generated data, despite the SE data being more in-distribution; this defines W2S-I as a new finetuning paradigm [§5.2, Fig. 4(c), Fig. 5(c)].
The advantage holds for proprietary frontier models when compute-matching is approximated via $/output-token: at the August 2024 Pro=$ 10.5 / Flash=$0.3 ratio, Flash-generated data beats Pro-generated data when used to distill Gemma-7B, Gemma2-9B, and Gemma2-27B on MATH [§6, Fig. 10].
Number-matched WC sampling (one sample per problem from each model) is worse than SE — the gain is specifically a compute-matched effect that prior comparisons missed [§5.3, Fig. 8].
Ablation: high-coverage / high-diversity > high-coverage / low-diversity > low-coverage / low-diversity, so both axes contribute and the result is not purely a coverage story [§5.3, Fig. 9].
Performance trendlines from the OpenLLM leaderboard suggest the small-LM MATH-performance curve is rising faster than the large-LM curve, projecting that the WC>SE advantage will widen over time [§7, Fig. 11].

Method

For each (WC, SE) pair, they generate two synthetic SFT corpora at matched sampling FLOPs: more samples per problem from WC, fewer from SE. Solutions are filtered by final-answer correctness only. They then finetune one of three models — a separate student (Gemma-7B), the WC model itself (Gemma2-9B), or the SE model itself (Gemma2-27B) — on each corpus and compare pass@1. The three setups instantiate the four cells of the (data-source × tuned-model) matrix in Table 1: WC→Student and SE→Student are both distillation; WC→WC is self-improvement; SE→SE is self-improvement; WC→SE is the newly-introduced weak-to-strong improvement (W2S-I); SE→WC is distillation. Datasets are MATH and GSM-8K; budgets are “low” (1 SE / 3 WC samples per problem) and “high” (10 SE / 30 WC). They also scale up the recipe to Gemini-1.5-Pro vs 1.5-Flash on MATH using output-token price as a proxy for compute, and add ablations on dataset size, number-matched vs compute-matched WC sampling, and (coverage, diversity) factorization.

Results

On MATH at the high sampling budget, Gemma-7B trained on Gemma2-9B (WC) data beats Gemma-7B trained on Gemma2-27B (SE) data in distillation [§5.2, Fig. 4(a)]; Gemma2-9B trained on its own WC data beats Gemma2-9B distilled from the SE data [§5.2, Fig. 4(b)]; and most surprisingly, Gemma2-27B trained on WC data (W2S-I) outperforms Gemma2-27B trained on its own self-generated data [§5.2, Fig. 4(c)]. Transfer to Functional MATH preserves the WC>SE ordering [§5.2, Fig. 6]. With Gemini-1.5-Flash (WC) vs Gemini-1.5-Pro (SE), distilling Flash data into Gemma-7B / 2-9B / 2-27B all beat the Pro-data version on MATH, even when Flash is sampled at a lower total cost than the SE 1-sample-per-problem baseline [§6, Fig. 10]. Number-matched WC sampling underperforms both compute-matched WC and SE [§5.3, Fig. 8] — pinning the headline result to the compute-matching axis specifically.

Why it’s interesting

Directly contradicts the dominant “use the strongest available model to generate SFT data” practice and supplies a clean compute-frontier framing for synthetic-data design. The W2S-I paradigm — a weaker model teaching reasoning to a stronger one — is the surprising piece: it shows the value of coverage at parity exceeds the in-distribution advantage of self-generated data, which is the same intuition the wiki’s Synthetic Training Data page has been accumulating from a different angle (verification-gate as load-bearing piece — here the gate is final-answer correctness, and the FPR cost is more than paid for by the coverage gain). The compute-matching framing also sharpens a recurring open question on the Reasoning RL page: how much of the self-improvement gain comes from the RL/SFT recipe vs from the data the recipe is fed. This paper says: at fixed compute, the data shape (coverage × diversity at the cost of FPR) is the bigger lever than the source model’s quality.