RLP: Reinforcement as a Pretraining Objective

TL;DR

RLP (Reinforcement Learning Pretraining) inserts a sampled chain-of-thought before next-token prediction during pretraining and rewards the thought by the log-likelihood improvement it gives the next observed token, relative to a no-think EMA teacher. The reward is verifier-free, dense (one scalar per token), and runs on ordinary text — including raw web-crawl — so RL-for-reasoning becomes a pretraining augmentation rather than a post-training phase. On qwen3-1.7b-base RLP lifts the eight-benchmark math+science average by 19% over continued pretraining, with gains compounding through identical SFT+RLVR post-training (+7–8% relative). On Nemotron-Nano-12B-v2 it adds 23 absolute points on science reasoning using only 0.125% of the baseline’s token budget.

Key claims

RLP treats CoT as an action sampled from the model itself, then uses the network with the CoT as predictor and the EMA-teacher without the CoT as a no-think counterfactual; the reward is the log-likelihood ratio of the next observed token between the two [§2.1, §2.2 Eq. 5].
The reward equals, in expectation, the reduction in cross-entropy between the reasoned scorer and the no-think baseline — so maximizing it directly maximizes the predictive utility of thinking [§2.3, Prop. 1].
Gradients flow only through the thought tokens; the next-token loss itself is not added — RLP replaces, rather than augments, the NTP term during the RLP phase, optimizing only Eq. 6 via a clipped GRPO-style surrogate with a group-relative inclusive-mean baseline [§2.4 Eq. 6, Eq. 8].
The EMA teacher is updated after each optimizer step ( $\theta_{\bar{}} \leftarrow \alpha \theta_{\bar{}} + (1-\alpha)\theta$ ), yielding a one-step-lagged smoothed counterfactual that prevents the log-likelihood ratio from collapsing to zero or drifting too far from the policy [§2.1, “EMA teacher instantiation and schedule”].
On qwen3-1.7b-base, RLP on a 1B-token PT mix beats compute-equivalent continuous pretraining by ~17% relative and beats the published RPT reinforcement-pretraining recipe by nearly 19% on the eight-benchmark average [§4, Table 1; §4 “RPT Comparison”].
The gains persist and compound through identical strong post-training (SFT on OpenThoughts + RLVR on MATH): RLP+Post beats CPT+Post by 8% and base+Post by 7% on the overall average, with the largest deltas in science reasoning [§4, Table 1].
Cross-domain transfer holds: RLP improves the eight-benchmark average by 7–9% across SFT-style corpora (OmniMath, OpenThoughts, Nemotron-Crossthink) and general-purpose corpora (academic papers, math textbooks, web-crawl) — i.e. it doesn’t require curated reasoning data [§5 Table 4].
Gains aren’t a compute-bookkeeping artifact: under FLOP-matched comparison (RLP on 170M tokens vs CPT on 6B tokens — 35× more data) RLP still leads by ~5.3% on the overall average [§5, Table 4 Nemotron-Crossthink rows].
RLP scales to a 12B hybrid Mamba-Transformer: on Nemotron-Nano-12B-v2, applying RLP for 250M tokens on top of a 20T-token checkpoint beats the 24T-token continued-pretraining baseline by 35% relative on the overall average (42.81 → 61.32), with science reasoning jumping 23 absolute points [§4 “Scaling Model Size and Architecture”, Table 2].
Ablations: more rollouts help up to ~8 (G=2 and G=4 underperform); longer completion length is the biggest lever (overall 41.05 at len=512 → 43.36 at len=2048); a KL anchor doesn’t help and slightly hurts (β=0 is best) [§5 “Ablations on rollout count, completion length, and KL weight”, Fig. 2].
RLP is positioned against the prior reinforcement-pretraining recipe RPT (Dong et al. 2025): RPT uses a sparse binary prefix-matching reward over auxiliary-model-filtered “high-entropy” tokens and ignores CoT content; RLP uses a dense continuous information-gain reward over all positions with no auxiliary filter, and updates the CoT itself [§1, §4 “RPT Comparison”].

Method

RLP rewires the last phase of pretraining into a GRPO-style RL loop on ordinary text. At each position $t$ in a sequence $x_{<t}$ , the model samples a short chain-of-thought $z_t \sim \pi_\theta(\cdot | x_{<t})$ , then scores the observed next token $x_t$ as $p_\theta(x_t | x_{<t}, z_t)$ . A second pass through an EMA-teacher copy of the network, $p_{\bar\theta}(x_t | x_{<t})$ , scores the same token without the CoT. The information-gain reward is $r_t = \log p_\theta(x_t | x_{<t}, z_t) - \log p_{\bar\theta}(x_t | x_{<t})$ — positive exactly when the sampled thought made the observed continuation more likely (§2.2, Eq. 5).

For each context, $G$ thoughts are sampled and a corrected inclusive-mean baseline produces low-variance group-relative advantages $\hat{A}_g = (r_g - \bar{r}) \cdot G/(G-1)$ (§2.4, Eq. 7). Updates apply only to the tokens inside the sampled thought, via a clipped PPO-style surrogate $L_{CLIP}$ on per-token importance ratios (Eq. 8). The reward itself is treated as a constant w.r.t. $\theta$ — no gradient through $p_\theta$ or $p_{\bar\theta}$ at reward computation time. The EMA teacher is initialized to the live network and updated post-step with momentum $\alpha$ , providing the one-step-lagged counterfactual that keeps the log-ratio informative without inviting degenerate “match the teacher exactly” strategies. Critically, the next-token-prediction loss (Eq. 1) is not part of the RLP objective during the RLP phase — only the information-gain surrogate is optimized.

Results

On qwen3-1.7b-base with 1B tokens of RLP on a general PT data mix, RLP averages 42.13 across the eight-benchmark math+science suite vs 34.19 for the base model and 35.60 for compute-equivalent continued pretraining (CPT) — a 19% relative lift over base, 17% over CPT [§4 Table 1, §5 Table 4]. With identical SFT-on-OpenThoughts + RLVR-on-MATH post-training, RLP+Post reaches 50.7 overall vs CPT+Post at 47.2 and base+Post at 47.5 — a +7–8% relative margin that grows in the science benchmarks (MMLU-Pro, GPQA-Diamond) [§4 Table 1].

Head-to-head against RPT (the prior reinforcement-pretraining recipe, Dong et al. 2025) on the OmniMATH dataset at matched data and compute: RLP improves Math Avg from 47.50 → 49.62 (+2.12), Science Avg from 35.88 → 37.07 (+1.19), and Overall from 41.69 → 43.35 (+4.0% relative). On Nemotron-Nano-12B-v2, RLP applied for 250M tokens on top of a 20T-token checkpoint beats the 24T-token continued-pretraining baseline by 35% on the overall average (42.81 → 61.32), with Science Avg jumping from 34.51 to 57.26 — i.e. a 23-absolute-point gain on science reasoning using ~0.125% of the baseline’s token budget [§4 “Scaling Model Size and Architecture”, Table 2].

Across data domains (§5 Table 4, all at 170M tokens), RLP-on-Nemotron-Crossthink reaches 43.36 overall, RLP-on-Web-Crawl reaches 42.13, RLP-on-OmniMath reaches 41.43 — all above CPT on the same data (35.01 on Nemotron-Crossthink-170M) and above the FLOP-matched CPT on Nemotron-Crossthink-6B (38.04). Ablation defaults: G=8 rollouts, completion length 2048, KL weight β=0.

Why it’s interesting

This is the most concrete answer to date to the question “what is the right RL objective for pretraining?” — and it directly extends the Reasoning RL cluster, which until now has only documented GRPO applied during post-training. RLP keeps the group-relative-advantage primitive but moves the entire recipe one phase earlier, swaps the reward from external-verifier-binary to internal-information-gain-continuous, and operates on raw text instead of curated reasoning traces. The information-gain framing — using an EMA-lagged copy of the model itself as a counterfactual — is a clean instance of the “model-as-its-own-reward” pattern that also shows up in Training-Free Group Relative Policy Optimization (Training-Free GRPO uses the LLM’s own semantic introspection as the policy update) and contrasts with Human detectors are surprisingly powerful reward models (which uses an external zero-shot perception stack as the reward). The “RL is the last phase of pretraining, not the first phase of post-training” framing is also the structural mirror of History May Repeat Itself: RSI Seen from a Previous AI Era‘s “is RL even the right RSI primitive?” question — both papers are pushing on the assumption that NTP-then-RL is the natural decomposition.