Demystifying Reinforcement Learning Post-Training of Language Models

TL;DR

A pedagogical deconstruction of RLVR (RL with Verifiable Rewards) post-training in a controlled and simplified environment, isolating how four axes shape outcomes: the base model’s prior distribution, reward-signal granularity, prompt-distribution diversity, and model scale. Uses the entropy of the policy’s output distribution as a common lens to compare pretraining, SFT, and RL-post-trained checkpoints, and shows mechanistically why “spurious rewards” only work under specific prompt distributions, why RL success depends on the base model already placing sufficient probability mass on the desired behavior, and why RL is best viewed as an exploration problem in the classical RL sense. Positioned explicitly as a primer for NLP practitioners incorporating RL into their toolbox.

Key claims

RL post-training success depends on the base model’s prior placing sufficient probability mass on the desired behavior, linking RL post-training to the classical RL notion of exploration [Abstract].
The effect of so-called “spurious rewards” depends on the prompt distribution used for post-training — same reward signal can help or fail depending on which prompts the RL loop sees [Abstract].
Output-distribution entropy is a useful common lens for comparing pretraining, SFT, and RL post-training — each stage shapes model certainty differently and the differences are visible in the entropy of the policy [Abstract].
The four factors interact rather than acting independently: base prior × reward granularity × prompt diversity × model scale jointly determine RL outcomes [Abstract].
RLVR is framed as a “black box” for many practitioners; the paper is positioned as a primer that opens the box by isolating each mechanic in a simplified controlled setting [Abstract].

Method

The paper deconstructs the RLVR pipeline into isolated mechanics and studies each in a controlled and simplified environment. Four factors are varied independently: (i) the base model’s prior distribution (what it puts probability mass on before RL), (ii) reward-signal granularity (dense vs sparse; correct vs partial vs spurious signals), (iii) prompt-distribution diversity (which set of prompts the RL rollouts draw from), and (iv) model scale. The comparison across pretraining / SFT / RL-post-trained checkpoints uses output-distribution entropy as a common lens — a single measurement that lets the three stages be placed on the same axis and shows how each stage sharpens or spreads the policy’s certainty. Building on this scaffolding, the paper revisits the “spurious rewards” phenomenon (originally shown for Qwen2.5-Math on math benchmarks) and links its effect to the prompt distribution, and revisits the “RL amplifies base behavior” framing by connecting it explicitly to exploration in the classical RL sense.

Results

Quantitative headline numbers are not stated in the abstract; the paper’s contribution is a set of controlled ablations rather than a new SOTA. The mechanistic claims are: (a) spurious rewards behave differently under different prompt distributions, (b) base-prior probability mass on the desired behavior is a necessary condition for RL post-training to succeed, and (c) the entropy of the policy’s output distribution differs systematically across pretraining, SFT, and RL stages in ways that reveal how each stage shapes certainty [Abstract]. The paper’s positioning explicitly targets the NLP practitioner community as a primer/resource rather than a benchmark leaderboard entry.

Why it’s interesting

The paper is a primary-source distillation of two threads this wiki tracks in scattered form. It restates the “RLVR at current scales elicits pretraining behaviors, does not install new ones” result that Spurious Rewards: Rethinking Training Signals in RLVR found empirically (Qwen2.5-Math’s Lang→Code transition under any weak reward) and The Path Not Taken: RLVR Provably Learns Off the Principals approached from the off-principal-subspace angle — but reframes it as an exploration problem tied to base-prior probability mass, which is the cleanest single-sentence framing on the wiki so far. It also connects to the SFT-for-RL Readiness cluster: the entropy-based comparison of pretraining vs SFT vs RL-post-trained distributions is exactly the axis Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning (PEAR) and TailSFT: Filtered Fine-Tuning Improves Post-Training Performance (TailSFT) study from the SFT side (why stronger-pass@1 SFT checkpoints underperform weaker ones after identical RL). Finally, the prompt-distribution dependency of spurious rewards adds a fifth degree of freedom to Shao et al.’s spurious-rewards mechanism — the Incorrect Baseline Evaluations Call Into Question Recent LLM-RL Claims audit already flagged that the effect is Qwen-family-specific; this paper argues the prompt set is another sensitivity.