Spurious Rewards: Rethinking Training Signals in RLVR

TL;DR

RLVR with pathologically weak or outright wrong reward signals — random rewards, format-only rewards, incorrect-label rewards — still produces ~+20-point MATH-500 gains on Qwen2.5-Math-7B, approaching the +29.1% gain from ground-truth rewards. The same rewards fail (or hurt) on Llama3 and OLMo2. The paper isolates the source of the Qwen-specific gain as code reasoning: Qwen2.5-Math emits Python-code-like solution traces 65% of the time before RLVR and >90% after training on any of the weak/spurious rewards, and code-reasoned answers are much more accurate than natural-language ones (60.9% vs 35.0%). RLVR here is not teaching new reasoning — it is amplifying a pretraining behavior. Practical implication: RLVR gains reported on Qwen alone should not be trusted without cross-family validation.

Key claims

On Qwen2.5-Math-7B, weak/spurious rewards produce large MATH-500 gains that approach the ground-truth-reward gain of +29.1%: random +21.4%, format +13.8%, incorrect-label +24.1%, 1-shot +26.0%, majority-vote +27.1% [§1, §2.3].
Random rewards work by design as pure noise — reward = 1 with fixed probability γ, else 0 — and the effect is monotone in γ (γ=0.5 is fastest, γ=0 collapses to no-signal as expected); this rules out any hidden supervision channel [§2.2, §4.4].
The spurious-reward effect is Qwen-family-specific: OLMo2 and Llama3 variants show flat or negative accuracy changes under the same rewards, and only ground-truth rewards yield gains for them [§3, Fig. 3].
The mechanism is a pretraining behavior called code reasoning — Qwen2.5-Math emits Python-like solution traces without any code-execution environment, 65.0% of pre-RLVR answers on MATH-500 [§4.1].
Code-reasoned answers are much more accurate than natural-language ones on Qwen2.5-Math-7B (60.9% vs 35.0% MATH-500 accuracy), and this “code helps” pattern is Qwen-specific — Qwen2.5-7B and OLMo2-7B-SFT also emit code frequently (“Bad-Code” models) but their code answers are less accurate than their language answers [§4.1, Table 4.2].
During RLVR with any weak/spurious reward, Qwen2.5-Math-7B’s code-reasoning frequency jumps to ≥90% within the first 15 steps and tracks the accuracy curve; random reward reaches the same endpoint more slowly [§4.2, Fig. 6].
Attributing the accuracy gain by transition class (Code↔Code, Code→Lang, Lang→Code, Lang→Lang) shows the Lang→Code subset contributes 58.3% of the total gain on Qwen2.5-Math-7B and 78.7% on Qwen2.5-Math-1.5B [§4.2, Table 2].
Explicit interventions on code frequency causally validate the mechanism: prompting the model to start with “Let’s solve this using Python” gives +24.2% / +15.0% / +10.0% on the three Qwen models that already do code reasoning, and degrades Llama and Qwen-2.5-7B (Bad-Code) by up to −28.6% [§4.3, Table 3].
A “Python reward” (reward = 1 iff response contains “python”) drives code frequency >99% within 20 steps on Qwen2.5-Math-7B and yields MATH-500 gains matching or exceeding prompt-based induction [§4.3].
The contrapositive holds too: compound “spurious reward AND no-python” rewards eliminate the Qwen2.5-Math-7B gain from format reward, reduce the incorrect-reward gain, and improve the Bad-Code models (OLMo2-7B-SFT: +8.9 on format, +5.5 on incorrect) [§4.3, Fig. 9].

Method

The setup is standard GRPO fine-tuning on DeepScaleR data, with the ground-truth binary correctness reward replaced by one of five progressively weaker signals: (1) ground truth, (2) majority-vote pseudo-labels from the pre-RLVR model, (3) format reward (any non-empty \boxed{}), (4) random reward with fixed γ, (5) “incorrect” reward that only rewards outputs matching the majority-vote pseudo-label restricted to items where that label is wrong. Everything else — optimizer, KL, batch — is held constant. Evaluation is pass@1/avg@8 on MATH-500 and AMC, with AIME24/25 as supplementary. Ten models are trained per reward: Qwen2.5-Math-{1.5B, 7B}, Qwen2.5-{1.5B, 7B}, Llama3.{1-8B, 2-3B} (base + instruct), OLMo2-7B, OLMo2-7B-SFT.

The mechanistic analysis has three parts. First, code-reasoning prevalence is computed by string-matching “python” in the response and manually verified. Second, per-item transitions between reasoning strategies (Code vs Lang, before vs after RLVR) are tabulated, and a Partial Contribution Score attributes total accuracy gain to each of the four transition subsets. Third, causal interventions in both directions: a prompt-prefix intervention forces code reasoning, an RL-reward intervention rewards responses containing “python”, and a compound reward that requires both the original spurious condition and the absence of “python” tests whether removing code reasoning removes the gain.

Results

Qwen2.5-Math-7B MATH-500: ground-truth +29.1%, incorrect +24.1%, random +21.4%, majority-vote +27.1%, format +13.8%, 1-shot +26.0%; all within 50 GRPO steps [§2.3, Fig. 2].
AMC gains follow the same pattern (format +6.3, incorrect +11.6, random +7.5 vs GT +12.4 on Qwen2.5-Math-7B, approximate) [§2.3].
AIME24 gains: format +10.3%, incorrect +10.2%, random +10.2%, GT +15.3%; AIME25 (post-cutoff) is where GT breaks away — spurious rewards yield −0.4% to +4.5% only [Appendix C].
Non-Qwen models: format/incorrect/random rewards produce flat or negative changes across all four Llama variants and both OLMo2 variants on MATH-500 and AMC [§3, Fig. 3].
Code-reasoning frequency on Qwen2.5-Math-7B jumps from 65% → >90% within 15 GRPO steps under format, incorrect, majority-vote, and (more gradually) random rewards; the trajectory closely tracks accuracy [§4.2, Fig. 6].
Compound “reward AND no-python”: on Qwen2.5-Math-7B, format-reward gains fully disappear (0 net change), incorrect-reward gains persist on MATH but shrink on AMC/AIME; GT-reward gains persist, confirming code reasoning is not the whole story for ground-truth [§4.3, Fig. 9].

Why it’s interesting

This paper is the empirical primary source behind Incorrect Baseline Evaluations Call Into Question Recent LLM-RL Claims‘s audit — Goel et al. list “Spurious Rewards” as their first case study, arguing separately that even the +21.4% headline number should be checked against the correct Qwen2.5-Math-7B baseline (they claim it is underreported). Filing the paper itself now completes the pairing: the Goel critique is on the wiki, but the finding it audits was not. This is exactly the kind of Qwen-centric RLVR result Reasoning RL currently accumulates uncritically, and it hardens the “cross-family validation is mandatory” caveat.

It also sharpens the parameter-level story from The Path Not Taken: RLVR Provably Learns Off the Principals. That paper argues RLVR concentrates updates on off-principal, low-magnitude weights and preserves the singular-value spectrum — mechanistically consistent with “RLVR does not install new reasoning, it re-routes to existing behaviors”. Spurious Rewards is the behavioral counterpart: even no reward signal amplifies a pretraining behavior (code reasoning) whose presence and quality is fixed by the base model. Together the two papers point at a consistent story where RLVR-at-current-scales is closer to a selective-elicitation procedure than to teaching new capabilities — a framing also raised by History May Repeat Itself: RSI Seen from a Previous AI Era‘s “Is RL the right primitive?” open question. The team-relevant takeaway is that reward shape may matter less than which pretraining behaviors the base model already ships with.