Incorrect Baseline Evaluations Call Into Question Recent LLM-RL Claims

TL;DR

A short Notion writeup auditing 7 widely-discussed LLM-RL papers — including “Spurious Rewards”, “RL from 1 example”, and three “intrinsic confidence rewards” papers — and showing that across all of them the pre-RL baseline numbers are significantly underreported relative to the official Qwen releases and standardized evaluations (e.g. Sober Reasoning). In several cases the headline “RL gain” shrinks dramatically or disappears once the corrected baseline is used, and “sometimes the pre-RL model even beats post-RL.” Identifies three recurring evaluation-methodology failure modes — gains explained by format-following, temperature-0 greedy decoding degenerating on long outputs, and lack of transparency about eval settings — and proposes that paper releases come with HuggingFace checkpoints + sample-level outputs.

Key claims

Across 7 popular LLM-RL papers (100+ to 3000+ likes, 50k+ to 500k+ views on X), the reported pre-RL baseline accuracies on MATH-500 are significantly below the official numbers in the Qwen releases and the standardized Sober Reasoning evaluations [§Snapshot, Snapshot table].
“Spurious Rewards” claims RLVR with random / format / incorrect-label / 1-shot / majority-voting rewards on Qwen2.5-Math-7B yields +21.4% / +16.4% / +24.6% / +24.4% / +26.5% on MATH-500, approaching the +28.8% from ground-truth rewards — but the audit re-runs raise doubts about whether the baseline in this comparison is the correct one [§Spurious Rewards].
Three common evaluation-methodology failure modes recur across the audited papers: (1) post-RL gains can be explained by better format-following rather than improved reasoning; (2) using temperature 0 / low temperatures amplifies (1), since greedy decoding is known to degenerate over long outputs; (3) papers do not transparently report eval settings [§Common Issues].
Both Sober Reasoning (0-shot) and the Qwen2.5 official numbers (few-shot) use simple, standard prompts — no prompt engineering — so the discrepancy is not explained by exotic baseline tuning [§Methodology].
For “RL from 1 example” and the three “intrinsic confidence reward” papers, individual per-paper breakdowns show that with corrected baselines the RL improvement either shrinks substantially or reverses [§per-paper sections].
The proposed mitigation is procedural rather than methodological: paper releases should ship open-weight checkpoints on HuggingFace plus sample-level outputs of the model on each reported eval, so claims can be audited at the rollout level [§Going Forward].

Method

The audit is observational, not a re-implementation. For each of the 7 audited LLM-RL papers, the authors compare the pre-RL baseline accuracy reported in the paper to (a) the official Qwen release numbers for the same checkpoint and (b) the standardized 0-shot evaluation in the Sober Reasoning paper (arXiv 2504.07086). Where the paper’s pre-RL number is lower than both reference numbers, they treat it as underreported and recompute the implied RL gain against the higher reference. They then read each paper’s evaluation section for plausible mechanical causes — temperature, decoding length, prompt template, format scoring — and identify which of the three failure modes (format-only gains, greedy-decoding degeneration, eval-setting opacity) likely produced the gap. The blog is intentionally short and concrete: each paper gets a per-claim block with the disputed number and the corrected comparison; no new RL training is performed.

Results

The headline result is qualitative: for every one of the 7 audited papers, the pre-RL baseline is reported lower than the standardized number, and the magnitude of the gap is in some cases large enough to fully erase or invert the claimed RL improvement. The “Spurious Rewards” section is the most cited: its central finding that RLVR with random/format/incorrect rewards still yields ~20+ point MATH-500 gains for Qwen2.5-Math-7B is restated alongside the observation that the paper itself notes these rewards “often fail to yield gains with other model families like Llama3 or OLMo2” [§Spurious Rewards] — i.e. the effect is Qwen-specific. The authors do not claim every audited paper’s headline is wrong; the explicit framing is that without corrected baselines and shared sample-level rollouts, the direction and magnitude of the RL gain cannot be assessed from the published numbers. The followup ask — checkpoints + per-sample outputs — is offered as the minimum-viable transparency contract.

Why it’s interesting

This blog is the cleanest filed example on the wiki of an evaluation-methodology critique aimed directly at the reasoning-RL literature that Reasoning RL has been documenting. The wiki’s reasoning-RL page accumulates “GRPO + shaped reward → +X points” results uncritically across Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning, MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling, Human detectors are surprisingly powerful reward models, and others; this writeup is the reminder that the “+X” almost always assumes the right baseline, and that the LLM-RL literature has systematically gotten that wrong. It contrasts usefully with Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling, which also picks at RL/synthetic-data assumptions but does so via controlled re-experiments rather than a baseline audit. The “format-following explains the gain” failure mode is also the empirical underbelly of the structured-CoT trend tracked by Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning — if reward shape rewards a tag schema, the headline gain may just be tag adherence.