RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training

TL;DR

Trains a 1B model from OLMo2’s architecture from scratch on 50B tokens of DOLMino and applies GRPO / SFT / SFT→RL directly to intermediate pretraining checkpoints (as early as 4B tokens, well before Chinchilla-optimal). Finds that (1) RL is effective very early — pass@1 gains often match the full SFT→RL pipeline on GSM8K despite never seeing a ground-truth reasoning trace, (2) targeted pretraining data (10B extra math tokens from Dolma3) beats 4× model scale for boosting RL gains on harder MATH problems, (3) the “sharpening” (flat/degraded pass@k) attributed to RLVR is an artifact of a preceding SFT stage — direct RL on base checkpoints expands pass@k, and (4) SFT consistently degrades general capabilities by 5–8 pp while RL leaves them intact. Proposes a parallel-averaging update that mixes RL and SFT gradients from the same snapshot each step, beating all baselines on pass@32 while preserving general capabilities.

Key claims

Direct RL on a pretraining checkpoint with as few as 4B tokens (well below the Chinchilla optimum for a 1B model) significantly improves GSM8K pass@1, and often matches the full SFT→RL pipeline [§3.1, Fig. 2].
Direct RL never sees ground-truth reasoning traces yet matches SFT+RL on pass@1 after ~30B pretraining tokens and matches on pass@8/32 as well [§3.1, Fig. 3].
On harder MATH problems, targeted pretraining data (10B extra math/reasoning tokens from Dolma3) yields substantially larger RL gains than scaling the model from 1B → 4B parameters, which mostly lifts the base checkpoint without improving RL’s marginal effect [§3.3, Fig. 4, Fig. 10, Fig. 11].
Base-model pass@k on the test set is a monotonic predictor of post-RL pass@k, usable as a lightweight pre-run diagnostic for whether RL will yield downstream gains [§3.4, Fig. 5].
Direct RL on base checkpoints expands the model’s distribution — pass@1 and pass@32 both improve — whereas SFT→RL exhibits the sharpening pattern (pass@1 up, pass@32 flat or degraded) that recent work has attributed to RLVR itself [§4.1, Fig. 6].
SFT (with either single or multi-reference solutions) consistently degrades general non-math benchmarks by 5–8 pp, while direct RL leaves general capabilities essentially unchanged [§4.2, Fig. 7].
A parallel-averaging update — one SFT batch and one RL batch through independent Adam optimizers per step, gradients averaged before parameter update — attains the strongest pass@32 across every pretraining checkpoint under a single-demonstration-per-problem regime, exceeding direct RL, SFT, and SFT→RL, while preserving general capabilities on par with direct RL; the trade-off is lower pass@1 than direct RL or SFT alone [§5, Fig. 9].
The SFT-Gold ablation (all ~23 demonstrations per problem) is the only recipe where SFT+RL wins pass@1, but SFT-Gold alone beats it on pass@8/32 — coverage from diverse ground-truth traces is a substitute for on-policy exploration [§3.2, Fig. 2].

Method

The setup is a controlled from-scratch pretraining run: a 1B-parameter model with OLMo2’s architecture is trained on 50B tokens of the DOLMino mix (~1× Chinchilla optimal), saving checkpoints throughout. For each checkpoint the authors run four post-training pipelines on OpenMathInstruct: Direct RL (GRPO with binary final-answer reward), SFT (single random ground-truth demonstration per problem), SFT-Gold (all demonstrations per problem, ~23 avg), and the standard SFT→RL and SFT-Gold→RL pipelines. Evaluation is pass@k for k∈{1,8,32} on GSM8K and MATH at temperature 1. Two additional pretraining regimes control for the “harder problems” setting in §3.3: fixing the 1B model and adding 10B math/reasoning tokens from Dolma3 Dolmino, versus scaling to 4B parameters at the same 50B tokens.

The §5 recipe (parallel averaging) starts from the same parameter snapshot each step, runs one RL batch through an RL optimizer and one SFT batch through a separate SFT optimizer (each with independent Adam moment estimates so adaptive step sizes don’t interfere), averages the resulting gradients, and updates θ. No scheduling, equal weights.

Results

GSM8K (1B model, direct RL beats SFT): At the 30B-token checkpoint, direct RL matches SFT+RL on pass@1 and matches both SFT-Gold and SFT+RL on pass@8/32, without any ground-truth reasoning traces [§3.1, Fig. 2, Fig. 3].
Very early effectiveness: At the 4B-token checkpoint, direct RL lifts GSM8K pass@1 from a low base to a high one (specific numbers reported as major absolute gains in §3.1). This is well before the 20B-token Chinchilla-optimal point for a 1B model.
MATH (harder problems, targeted data beats scale): On MATH the 4B model has a higher base but no larger RL gain than the 1B; adding 10B math tokens to the 1B pretraining mix substantially increases the RL gain and recovers GSM8K-like behavior on MATH [§3.3, Fig. 4].
Sharpening isolated to SFT+RL: SFT+RL improves pass@1 continuously but degrades pass@32 versus the SFT stage alone; direct RL improves both pass@1 and pass@32 simultaneously [§4.1, Fig. 6].
General-capability regression is an SFT artifact: Across a suite of general-purpose benchmarks, direct RL leaves scores essentially unchanged while SFT (both variants) drops scores by 5–8 pp [§4.2, Fig. 7].
Parallel averaging wins pass@32: At every checkpoint evaluated, parallel-averaged RL+SFT attains the strongest pass@32 among single-demonstration recipes, beating direct RL, SFT, and SFT→RL, while matching direct RL on general-capability preservation [§5, Fig. 9, Fig. 17].

Why it’s interesting

Directly opposes the current “RL is a final polish after SFT” default and lines up with three independent measurements already on the wiki: Spurious Rewards: Rethinking Training Signals in RLVR (RLVR at current scales elicits pretraining behaviors), The Path Not Taken: RLVR Provably Learns Off the Principals (RLVR updates live in an off-principal, low-magnitude subspace), and Understanding Reasoning from Pretraining to Post-Training (post-RL performance is predictable from pretraining loss and the RL reward slope improves with pretraining tokens). This paper’s contribution is the controlled from-scratch pretraining sweep — the same picture that Shen et al. show at the chess-testbed scale, reproduced at 1B/4B with real math data, plus two novel results the others don’t have: (a) the sharpening effect commonly blamed on RL is caused by the preceding SFT stage, and (b) parallel-averaged SFT+RL beats every sequential recipe on pass@32. The result also complements the SFT-for-RL-readiness thread (Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning, Behavior Injection: Preparing Language Models for Reinforcement Learning, TailSFT: Filtered Fine-Tuning Improves Post-Training Performance) by pushing further — instead of redesigning SFT to be a better RL prep, drop the sequential structure entirely and mix the gradients. And it directly cites RLP: Reinforcement as a Pretraining Objective as complementary: RLP modifies the pretraining objective, this paper keeps NTP and RL untouched but reorders when they apply.