Training-Free Group Relative Policy Optimization

TL;DR

Training-Free GRPO ports GRPO’s group-relative advantage from parameter space to context space: a frozen LLM rolls out a group of trajectories per query, then introspects on the group to distill a natural-language “semantic advantage” that is committed to an evolving experience library via Add/Delete/Modify/Keep operations. The library is plugged in as a token prior on subsequent rollouts, replacing gradient updates. On AIME24/25 and WebWalkerQA with DeepSeek-V3.1-Terminus, three “training steps” over ~100 OOD samples (≈ $18 of API spend) outperforms 32B-scale fine-tuned baselines (ReTool, AFM) that cost ~$ 10K to train. Notable failure mode: gains collapse on weaker base models (QwQ-32B, Qwen2.5-72B-Instruct), suggesting the recipe is a capability amplifier, not a substitute.

Key claims

A 3-step, 100-sample “training-free” run on DeepSeek-V3.1-Terminus + ReAct lifts AIME24 from 80.0 → 82.7 (+2.7) and AIME25 from 67.9 → 73.3 (+5.4), surpassing 32B RL-fine-tuned baselines like ReTool (67.0 / 49.3) and AFM (66.7 / 59.8) [Table 3].
The semantic advantage — natural-language reasons distilled from groups with both winners and losers — is doing the work, not the experiences themselves: directly-generated experiences of the same quantity degrade the baseline (79.8 / 67.3 vs 80.0 / 67.9), while the GRPO-style group introspection lifts it [Table 2].
Group computation is load-bearing: setting group size G=1 (single rollout, no within-group comparison) drops AIME24/25 to 80.4 / 69.3, vs 82.7 / 73.3 at G=5 [Table 2].
The recipe works without ground-truth labels (relying on implicit majority voting / self-discrimination), reaching 80.7 / 68.9 — below the GT-supervised variant but still above ReAct baseline [Table 2].
Frozen-model + plug-in experience library preserves cross-domain performance: the same DeepSeek-V3.1-Terminus reaches SOTA on both math (82.7 / 73.3 on AIME) and web-search (67.8 on WebWalkerQA) by swapping experience libraries — whereas ReTool (math-trained) collapses to 18.3 on WebWalker and MiroThinker (web-trained) drops to 43.5/36.8 on AIME [Table 6].
Effectiveness scales with base-model capability: applied to QwQ-32B on WebWalkerQA, it underperforms the model’s own ReAct baseline (25.5 vs 27.5 pass@1) [Table 5]; on Qwen2.5-72B-Instruct gains are marginal (+1.4 / +1.8 on AIME) compared to DeepSeek-V3.1-Terminus (+2.7 / +5.4) [Table 3].
Cost asymmetry: replicating ReTool on Qwen2.5-32B-Instruct costs ~20,000 GPU-hours ≈ $10,000; Training-Free GRPO costs ~38M input + 6.6M output tokens ≈$ 18 at DeepSeek pricing — two orders of magnitude cheaper, no GPU procurement [§4.2].
Training-Free GRPO also reduces tool-call count during both training and OOD evaluation, suggesting the learned experiences encode shortcuts and avoid redundant tool invocations [§3.1, Fig. 4].

Method

Take vanilla GRPO and replace every gradient step with an inference-only operation. Initialize an empty natural-language experience library E. For each query q in a batch: (1) Rollout — generate G outputs o_i from the frozen LLM conditioned on (q, E); (2) Reward — score each o_i with a reward model (or ground-truth checker) to get r_i; (3) Semantic advantage — for groups with both clear winners and losers (i.e. non-uniform rewards), prompt the same LLM to (a) summarize each o_i, (b) compare summaries against current E, and (c) extract a concise natural-language “experience” A_i that articulates why the winners won and losers lost. This A_i is the semantic analogue of vanilla GRPO’s numerical advantage. (4) Optimization — prompt the LLM to apply a list of operations over E given the batch of {A_i}: each operation is one of Add (append new experience), Delete (remove a low-quality entry), Modify (refine an existing entry), or Keep (no-op). The updated E shifts the conditional distribution on subsequent batches/epochs, mimicking a policy update via a context shift. The frozen base model functions as an implicit KL anchor — the equivalent of vanilla GRPO’s KL penalty against a reference policy — because the policy cannot deviate further than what a prompt prefix can induce.

Defaults in the experiments: G=5, T=0.7 during learning, 3 epochs × 1 batch (so 3 optimization steps total), 100 training samples (DAPO-100 for math, AFM-100 for web search). Evaluation uses T=0.7 with Mean@32 on AIME.

Results

Math (AIME24/25, Mean@32): DeepSeek-V3.1-Terminus + ReAct + Training-Free GRPO reaches 82.7 / 73.3, vs 80.0 / 67.9 ReAct baseline [Table 3]. Surpasses 32B RL methods trained on thousands of samples: ZeroTIR (56.7 / 33.3), SimpleTIR (59.9 / 49.2), ReTool (67.0 / 49.3), AFM (66.7 / 59.8) [Table 3]. Direct-prompting variant (no tool use) also improves: 68.6→72.6 / 52.9→54.0 [Table 3]. Smaller-model results are weaker — Qwen3-32B Non-Thinking gains +4.4 / +5.9; Qwen2.5-72B-Instruct gains only +1.4 / +1.8 from a low baseline [Table 3].

Web search (WebWalkerQA): 63.2 → 67.8 pass@1 on DeepSeek-V3.1-Terminus with AFM-100 [Table 4]. Ablation on a stratified 51-sample subset (pass@1 / pass@3): ReAct 66.7 / 74.5 → full Training-Free GRPO 68.6 / 78.4 [Table 5]. Without ground truths: 66.7 / 78.4 (matches baseline pass@1 but improves consistency at pass@3) [Table 5]. Directly-generated experiences degrade pass@1 (64.7) [Table 5]. QwQ-32B base: 27.5 → 25.5 pass@1 (regression) [Table 5].

Cross-domain: Math-trained ReTool drops from AIME 67.0 to WebWalker 18.3; web-trained MiroThinker drops from WebWalker 53.6 to AIME 43.5 / 36.8 [Table 6]. Same DeepSeek-V3.1-Terminus + appropriate experience library: 82.7 / 73.3 / 67.8 across all three [Table 6].

Learning dynamics: Mean@32 improves monotonically across the 3 optimization steps on AIME24/25; average tool-call count decreases on both training and OOD evaluation [§3.1, Fig. 4].

Why it’s interesting

This is a sharp counterfactual to the Reasoning RL orthodoxy on the wiki: every other filed reasoning-RL paper updates parameters via GRPO (InfTool, MiroThinker, Video-Thinker, vlm-gym, HuDA). This one keeps the same group-relative comparison primitive but moves the optimization target into a natural-language experience library that lives in the prompt. For a team paying ~ $10K per RL run on 32B models, the "match-or-beat at$ 18, no GPU procurement, no specialized deployment” claim is worth stress-testing. The capability-dependence finding (works on DeepSeek-V3.1-Terminus, fails on QwQ-32B) is the most useful caveat: the technique amplifies an existing reasoning + tool-use prior rather than installing one. It also opens a clean question for the wiki’s parametric-vs-context-memory thread (cf. Deep sequence models tend to memorize geometrically; it is unclear why, Recursive Language Models): if a few-paragraph token-prior can stand in for thousands of gradient steps when the base model is strong enough, the marginal value of fine-tuning a frontier model on domain data may be small and rapidly shrinking with base-model scale.