On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models

TL;DR

A controlled-synthetic study of how pre-training, mid-training, and RL post-training jointly determine reasoning generalization in 100M-parameter LMs. Using GSM-Infinite-style DAG-grounded arithmetic tasks with parseable step traces, the authors disentangle when RL produces genuine capability gains (pass@128) versus merely sharpening the base model. The headline reconciliation: RL improves the base only when pre-training leaves headroom and RL data sit at the model’s edge of competence; mid-training on edge-difficulty data is a substantially more compute-efficient lever than pure RL under a fixed budget; and process-level rewards mitigate reward hacking with minimal performance cost.

Key claims

RL produces true capability gains (pass@128, not just pass@1) only when (i) the target task was not heavily covered during pre-training, leaving sufficient headroom for exploration, and (ii) RL data are calibrated to the model’s edge of competence — neither in-distribution-easy nor out-of-distribution-hard [§3, §1 (insight 1)].
When either condition is violated, RL sharpens pass@1 but does not extend the base model’s reasoning frontier — reconciling the two prior camps (yue2025 / wu2025 “RL is a refiner” vs. liuProRL / sun2025 / yuan2025 “RL induces new capabilities”) as endpoints of a single mechanism [§3, §7].
Contextual (cross-template) generalization requires minimal but sufficient pre-training exposure to the new context; even ~1% coverage of atomic-primitive (op=2) examples in the new template is enough seed for RL to robustly transfer, while 0% coverage means RL does not induce transfer [§4 (insight 2)].
Mid-training — same next-token objective as pre-training but on a narrowed, RL-adjacent distribution at the model’s competence boundary — substantially outperforms pure RL under a fixed compute budget (token-equivalent, flop-normalized), both in-domain and out-of-domain [§5 (insight 3)].
Under the five-point compute-budget sweep (Full mid-training, Full RL, Light-RL α=¼, Medium-RL α=½, Heavy-RL α=¾), the mid-training-heavy mixes dominate the pure-RL endpoint; mid-training is “an underexplored but powerful lever in training design” [§5].
Process-level reward — a composite r_outcome + λ·r_process with r_process measured against the ground-truth DAG’s intermediate nodes — mitigates outcome-reward hacking and improves both accuracy and generalization under compositional settings [§6 (insight 4)].
The 100M-param Qwen2.5-style base is pre-trained on 10B tokens (100× parameters, Chinchilla-style) of op=2-10 problems and reaches near-saturated pass@128 on the in-distribution range — the headroom property is established by construction, not assumed [§2.4].
Three problem-difficulty regimes are defined operationally relative to the base model’s pass@128: In-Distribution (op=2-10, saturated), OOD-edge (op=11-14, non-zero pass@128), OOD-hard (op=15-20, near-zero pass@128) — the framework makes “edge of competence” a measurable quantity rather than an intuition [§3, Appendix A.3.4].

Method

The authors build a fully controlled testbed on top of GSM-Infinite (Zhou et al.). Each reasoning instance is a directed acyclic graph G=(V,E,f,a) over arithmetic operations (+, −, ×, ÷), with operation count |E| as the structural-complexity knob. Abstract parameters specify the graph; instance parameters supply numbers; a contextual template (e.g. animals-zoo, teachers-school) renders the graph into natural language, with a subset of edges verbalized explicitly and the rest left as implicit reasoning steps.

A 100M-parameter Qwen2.5-style decoder-only model is pre-trained on 10B tokens of op=2-10 problems across contexts. Post-training uses GRPO over four distinct RL data slices (op=7-10 ID, op=9-12 mixed, op=11-14 edge, op=17-20 hard) with 200K samples per slice. Mid-training, when applied, uses the same NTP objective on the op=11-14 boundary distribution. Compute is normalized via a token-equivalent cost C_RL ≈ N·k·L_avg so that mid-training and RL phases can be traded under a fixed budget; α = C_RL / C_total parameterizes the sweep.

Evaluation is process-verified: the model’s free-form output is parsed into a predicted dependency graph Ĝ and final answer ŷ; a node is correct only if its operation, dependencies, and numeric value all match the gold DAG; process accuracy is the average over gold nodes; pass@k counts as correct only when both reasoning and answer are right. Process-aware reward design then adds r_process directly to the GRPO reward signal.

Results

The Chinchilla-style 10B-token pre-training run saturates pass@128 on op=2-10 by construction, isolating the OOD-edge and OOD-hard regimes as the meaningful evaluation surface [§2.4].
Across the four RL data slices, GRPO at the edge (op=11-14) is the regime where pass@128 measurably exceeds the base model on op>10 problems; OOD-hard (op=17-20) RL data fails to improve pass@128 because the base model’s coverage is too thin for GRPO to find signal; ID (op=7-10) RL data improves pass@1 but not pass@128 [§3].
Contextual-generalization sweep on long-tailed context B: 0% op=2 atomic exposure during pre-training leaves RL unable to transfer, while 1% atomic exposure is sufficient for RL to lift accuracy on context B substantially — the curve is highly non-linear at the low-exposure end [§4].
The five-point compute-budget sweep finds mid-training-heavy allocations (Light-RL α=¼, Medium-RL α=½) dominate Full-RL (α=1) on both ID and OOD evaluations [§5].
Process-aware reward (r = r_outcome + λ·r_process) reduces the gap between answer accuracy and process accuracy — i.e. mitigates the “right answer, wrong reasoning” reward-hacking mode — and the stricter formulation (outcome reward gated on a fully verified process) reduces it further [§6].

Why it’s interesting

The wiki already holds the two camps this paper reconciles, side by side, with no synthesis: The Path Not Taken: RLVR Provably Learns Off the Principals is a parameter-level argument that RLVR stays in the base model’s off-principal subspace (consistent with “RL refines”), while RLP: Reinforcement as a Pretraining Objective argues for RL as a pretraining objective, and Self-Adapting Language Models / Incorrect Baseline Evaluations Call Into Question Recent LLM-RL Claims sit uneasily between the two. This paper supplies the missing task-level axis: the two camps disagree because they implicitly sample different points on the (pre-training-coverage, RL-data-difficulty) plane, and the headroom + edge-of-competence framing makes that explicit. The mid-training result is the more actionable contribution — it directly contradicts the implicit “RL is the lever, scale it” framing of the Reasoning RL page’s TL;DR by showing mid-training is more compute-efficient than pure RL under fair accounting.

It also lands cleanly next to two specific wiki entries. The Path Not Taken: RLVR Provably Learns Off the Principals‘s “RLVR avoids principal weights” finding is consistent with this paper’s “RL needs the primitive present in the base” claim: if RL cannot move principal-subspace weights, then any capability whose representation is not already in that subspace must be installed during pre-training or mid-training — this paper’s contextual-generalization sweep is the task-level evidence for that parameter-level prediction. JustRL: Scaling a 1.5B LLM with a Simple RL Recipe‘s “simple GRPO recipe scales a 1.5B base” result is sampled at a specific (coverage, difficulty) point that the framework here would predict to be favorable (math RL data near the 1.5B base’s edge of competence); the framework supplies a prediction about when JustRL-style recipes will and will not transfer.