Understanding Reasoning from Pretraining to Post-Training

TL;DR

A controlled end-to-end study of the pretraining → SFT → RL pipeline using chess as a testbed with verifiable rewards. Trains language models from 5M to 1B parameters on human chess games, SFTs on synthetic reasoning traces, and runs RL on chess puzzles, then repeats the recipe on math text to test transfer. The headline findings are quantitative: post-RL performance at any given RL compute is well-predicted from the pretraining loss of the base checkpoint, and the slope of the RL reward curve improves approximately linearly with pretraining tokens — i.e. RL returns scale with pretraining, not just with post-training compute. The mechanism split is also sharp: on easy puzzles RL amplifies moves the SFT policy already preferred, while on hard puzzles RL surfaces correct moves that were nearly absent under SFT.

Key claims

Pretraining loss predicts post-RL performance at a given RL compute level, across model sizes 5M–1B trained on human chess games [Abstract, §Findings].
The slope of the RL reward curve improves approximately linearly with the number of pretraining tokens — longer-pretrained checkpoints not only start higher but learn faster under RL [Abstract, §Findings].
RL does not simply sharpen the SFT policy: on easy chess puzzles RL amplifies correct moves the SFT policy already assigned meaningful mass to, while on hard puzzles RL elicits correct moves that were nearly absent under SFT [Abstract, §Behavioral analysis].
The same predictive pattern — longer pretraining → higher post-RL performance + faster RL improvement — reproduces on a 1B language model trained on math-domain text, i.e. the finding is not chess-specific [Abstract, §Math transfer].
Chess is a viable controlled testbed for studying the pretraining→post-training interface because the pretraining corpus (human games), the reasoning traces (synthetic), and the RL rewards (puzzle correctness) can each be independently scaled and audited — a property standard LLM setups don’t have [Abstract, §Motivation].

Method

The authors instantiate the standard three-stage LLM training pipeline in a domain where every stage’s inputs can be controlled. Pretraining: decoder-only language models from 5M to 1B parameters are trained on human chess game move sequences, sweeping pretraining tokens as an independent variable. Post-training: supervised fine-tuning on synthetic chess reasoning traces (structured move analyses), followed by RL on chess puzzles with a verifiable puzzle-solve reward — the reward is exact and cheap, so RL compute can be swept independently of noise. The core measurement is post-RL performance at matched RL compute across the pretraining sweep, which decouples “the base model got there” from “RL got it there.” A 1B checkpoint is then trained on math-domain text with the same three-stage pipeline to test whether the pretraining ↔ RL coupling transfers off chess.

Results

Post-RL performance at fixed RL compute is a smooth function of pretraining loss across the 5M–1B sweep, giving a predictive pretraining-to-RL relationship rather than only a correlational one [Abstract, §Findings].
The pretraining-token axis has an approximately linear effect on the slope of the RL reward curve — longer-pretrained checkpoints improve faster under RL, not just start higher [Abstract, §Findings].
The behavioral decomposition into easy-puzzle amplification vs. hard-puzzle elicitation is a mechanism-level result: RL is doing different things at different task difficulties, both of which show up as pass@1 gains but with different implications for capability vs. sharpening [Abstract, §Behavioral analysis].
The math-text 1B run reproduces “longer-pretrained → higher post-RL performance + faster RL improvement” — the pattern is not an artifact of chess’s low input entropy or the specific puzzle-reward structure [Abstract, §Math transfer].

Why it’s interesting

This is a direct, controlled complement to two entries already on the wiki. RLP: Reinforcement as a Pretraining Objective argues for pushing RL into pretraining and shows the resulting information-gain reward benefits compound through post-training; this paper is the other side of the same question — it holds the pretraining objective fixed at NTP and quantifies how the pretraining budget itself changes RL’s returns. On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models argues on a synthetic-arithmetic testbed that RL improves the base only when pretraining leaves headroom and RL data sit at the edge of competence, and that mid-training is a more compute-efficient lever than pure RL; the chess study here supplies the smooth-across-scale version of that headroom claim — pretraining loss becomes a continuous predictor of post-RL performance rather than a binary “enough headroom / not enough” gate. Together with Spurious Rewards: Rethinking Training Signals in RLVR (RLVR elicits pretraining behaviors) and The Path Not Taken: RLVR Provably Learns Off the Principals (RLVR stays off-principal in weight space), the Reasoning RL cluster now has three independent measurements — behavioral, weight-space, and now scaling-law — that all point at the same conclusion: RL’s job depends on what pretraining put there. The chess testbed is also a rare instance of the Hyperparameter scaling laws recipe (fit predictors on cheap small-scale runs) being applied to post-training returns rather than to pretraining loss itself.