JustRL: Scaling a 1.5B LLM with a Simple RL Recipe

TL;DR

JustRL is a stripped-down GRPO recipe for 1.5B reasoning models: single stage, fixed hyperparameters, 16K context cap, DAPO rule-based verifier, DAPO-Math-17k without filtering, no curriculum, no length penalty, no KL loss, no entropy regularization — just “clip higher” for stability. Trained for ~4K steps from DeepSeek-R1-Distill-Qwen-1.5B and OpenMath-Nemotron-1.5B with the same hyperparameters, it reaches 54.87% and 64.32% average across nine math benchmarks (AIME, AMC, MATH-500, Minerva, OlympiadBench, HMMT, BRUMO, CMIMC), matching or exceeding ProRL-V2’s nine-stage pipeline and QuestA’s curriculum at 2× less compute. The headline contribution is methodological: smooth monotonic training curves over 4,000+ steps with no collapses or oscillations, and ablations showing that adding “standard tricks” (overlong penalty, robust verifier) actively degrades performance.

Key claims

A single-stage GRPO recipe with fixed hyperparameters matches or exceeds nine-stage / curriculum-based RL approaches at 1.5B scale: 54.87% avg on DeepSeek-R1-Distill-Qwen-1.5B (vs ProRL-V2’s 53.08%, leading on 6/9 benchmarks) and 64.32% on OpenMath-Nemotron-1.5B (vs QuestA’s 63.81%, leading on 5/9) [Table 3, Table 5].
The same hyperparameters transfer between the two base models without per-model tuning, supporting a robustness claim beyond lucky single-model fit [§4.2, Table 2].
JustRL uses ~2× less compute than the closest competitors at comparable performance: 1.4×10⁸k tokens vs ProRL-V2’s 2.8×10⁸k (DeepSeek base), and 1.1×10⁸k vs QuestA’s 2.6×10⁸k (Nemotron base); BroRL needs 4.9× more compute via 512 rollouts/example [Table 4, Table 6].
Training dynamics are smooth and monotonic over 4,000+ steps without the collapses or plateaus typically cited as motivation for multi-stage / scheduled interventions: entropy oscillates 1.0–1.6 with no systematic drift, mean reward climbs -0.6 → +0.4 without extended plateaus, and response length naturally compresses from ~8K to 4–5K tokens with no explicit length penalty [§4.3, Fig. 2].
Adding a DAPO-style overlong-response penalty degrades AIME24 performance from ~55% to ~50% and collapses policy entropy to 0.5–0.6 vs the 1.2–1.4 baseline range, evidence that the penalty forces premature exploration collapse [§4.4, Fig. 3].
Adding a “robust” model-based verifier on top of the overlong penalty degrades AIME24 further to ~45%; the authors hypothesize the stricter rule-based verifier creates a richer learning-signal spectrum and pressures the model to develop more robust internal computations than a forgiving verifier would [§4.4].
The minimal stack used: GRPO with binary outcome reward from DAPO’s rule-based verifier (no SymPy), batch size 256, LR 1e-6 constant, temperature 1.0, 8 rollouts, clip ratio [0.8, 1.28], 16K max response length, no KL loss, no entropy regularization [Table 2].

Method

JustRL is an explicitly negative-design RL recipe — defined by what it removes from the accumulated 2025 toolkit. The authors enumerate nine techniques (Entropy Coefficient, Token-level Hyperparameters / Loss Aggregation, Truncated Importance Sampling, Reverse KL, Length Control, Adaptive Temperature, Reference Resets, Dynamic Sampling, Staged Training) used across STILL-3, DeepScaleR, FastCuRL, ProRL-V1/V2, e3, POLARIS, QuestA, and BroRL [Table 1]. JustRL uses exactly one of these — clip higher (their EC column) — and treats it as part of the GRPO baseline rather than an “added technique.”

Training is single-stage from a distilled checkpoint (DeepSeek-R1-Distill-Qwen-1.5B or OpenMath-Nemotron-1.5B) on DAPO-Math-17k for 4,380 / 3,440 steps respectively on 32× A800-80GB GPUs for 15 days each. The reward is binary correctness from DAPO’s string-matching verifier (no SymPy, no learned reward model). Length is controlled only by a 16K max-response cap. Evaluation uses the POLARIS protocol with CompassVerifier-3B for false-negative correction; Pass@1 is averaged over N=32 samples (or N=4 for MATH-500/Minerva/OlympiadBench) at temperature 0.7, top-p 0.9.

The ablation methodology is the load-bearing argument: the authors start from the JustRL baseline and add (1) DAPO’s overlong penalty over the last 4K tokens and (2) DeepScaleR’s robust verifier. Both interventions plateau lower than the simple baseline by 5 and 10 AIME24 points respectively [Fig. 3], directly contradicting the assumption that more sophisticated reward shaping is monotonically beneficial.

Results

Final numbers vs the closest competitors at each base model:

	Avg / 9	AIME24	AIME25	AMC23	MATH	Minerva	Olympiad	HMMT	BRUMO	CMIMC
DeepSeek-1.5B base	37.65	29.90	22.40	63.82	84.90	34.65	45.95	13.44	30.94	12.89
ProRL-V2	53.08	51.87	35.73	88.75	92.00	49.03	67.84	19.38	47.29	25.86
JustRL-DeepSeek	54.87	52.60	38.75	91.02	91.65	51.47	67.99	21.98	52.71	25.63
Nemotron-1.5B base	56.74	58.75	48.44	90.55	92.40	26.93	71.70	30.10	61.67	30.08
QuestA-Nemotron	63.81	71.56	62.08	93.44	92.95	32.08	72.28	40.94	67.50	41.48
JustRL-Nemotron	64.32	69.69	62.92	96.02	94.15	30.24	76.59	40.63	66.88	41.72

Compute: JustRL-DeepSeek uses 1.4×10⁸k tokens (vs ProRL-V2’s 2.8×10⁸k and BroRL’s 6.8×10⁸k) [Table 4]. JustRL-Nemotron uses 1.1×10⁸k tokens (vs QuestA’s 2.6×10⁸k) [Table 6]. The compute calculation assumes a 50% filter ratio for dynamic-sampling baselines; the authors note their numbers remain comparable or lower even at 0% filtering.

Ablations [Fig. 3]: JustRL baseline plateaus at ~55% AIME24 with entropy in 1.2–1.4. Adding DAPO’s overlong penalty drops the plateau to ~50% AIME24 with entropy collapsing to 0.5–0.6. Adding both overlong penalty and DeepScaleR’s robust verifier drops the plateau further to ~45% AIME24.

Why it’s interesting

This is the cleanest version of the “simpler baseline first” methodological critique to land in the wiki — it sits directly alongside Incorrect Baseline Evaluations Call Into Question Recent LLM-RL Claims, which argued that recent LLM-RL claims rest on under-tuned baselines, and provides the constructive counterpart: a single-stage recipe that beats the multi-stage ones outright at half the compute. Read together, the two papers argue that the 2025 reasoning-RL toolkit has been climbing complexity that the field hasn’t shown is necessary. For Reasoning RL this is the first filed result that explicitly subtracts from the GRPO recipe rather than adding to it; it contrasts with the trajectory of Training-Free Group Relative Policy Optimization (which removes the parameter update entirely) and the architecture-side bets in IQuest-Coder-V1 Model Family (which adds looped-transformer compute on top of RL). The negative-ablation finding — that the DAPO overlong penalty and a “robust” verifier hurt performance via entropy collapse — is the kind of result that should make reviewers of any future RL-for-reasoning paper ask whether the proposed technique is solving a problem the simpler recipe doesn’t have.