Agent² RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?

ListenCan an Agent Actually Run RL? Mostly Not (But Watch ALFWorld)10:27

Show notes

Microsoft Research Asia's **Agent² RL-Bench** asks whether LLM agents can autonomously design, implement, and run a closed-loop online RL post-training pipeline under a twelve-hour budget. The headline: agents can sometimes pull it off — ALFWorld jumps from under six to ninety-three — but stable agent-driven RL stays rare, supervised fallbacks dominate, and the driver LLM turns out to be the load-bearing variable on interactive tasks. We dig into what the benchmark measures, where the gains are real versus inside the noise, and what it means for anyone evaluating agentic training systems. **Sources** - Agent² RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

Microsoft’s Agent² RL-Bench is the direct next step past PostTrainBench: Can LLM Agents Automate LLM Post-Training?: instead of asking whether a CLI coding agent can do supervised post-training on a base LLM, it asks whether the agent can autonomously design, implement, and run a closed-loop online RL pipeline (trajectory collection, rollouts, trajectory-level rewards) under a 12h budget. Six tasks span three progressive levels — L1 static rule-based (GSM8K, HumanEval), L2 static judge-based (AlpacaEval 2.0), L3 interactive rollout (ALFWorld, WebShop, DeepSearchQA). Across 5 agent systems × 6 driver LLMs on Qwen2.5-7B-Instruct and Qwen3-8B-Base, the headline finding is that agents produce dramatic gains on some interactive tasks (ALFWorld: 5.97 → 93.28 via SFT warm-up + GRPO with online rollouts under RL-only mode) but fail to engineer online RL reliably in general (DeepSearchQA: +2.75 within 3pp evaluation noise across every stack). The paper’s most transferable finding is that within a fixed Claude Code scaffold, switching drivers swings ALFWorld improvement from near-zero to +78pp — driver choice, not scaffold or algorithm, is the load-bearing variable on interactive tasks.

Key claims

The best agent (Claude Code + Opus 4.6) achieves ALFWorld 5.97 → 95.52 (Free mode) and 5.97 → 93.28 (RL-only via SFT warm-up + GRPO with online rollouts) on Qwen2.5-7B-Instruct, the cleanest filed demonstration of an agent stack producing a working online-RL pipeline; the RL-only 93.28 is the only case in the benchmark where online RL succeeds as the final best route [Table 2, §3.2, §3.4].
Driver choice dominates scaffold choice on interactive tasks: within Codex CLI, switching from GPT-4o to GPT-5.2 changes ALFWorld improvement from +0.74 to +78.35pp; within Claude Code, Opus 4.6 vs Sonnet 4.5 is +88.80 vs +6.71 [Table 2 Panel B, §3.2].
Task-dependent driver sensitivity: on GSM8K the six 8B-Base stacks span only 19pp (70.13–89.46), but on ALFWorld they span 89pp (9.70–97.76) — interactive tasks amplify stack quality differences [§3.2].
Static-benchmark success is a poor predictor of interactive success: Claude Code’s ALFWorld improvement is ~35× its GSM8K improvement; agents attempted RL methods first but fell back to SFT-based pipelines on 5 of 6 tasks [§3.3].
Operating-mode ordering reverses between L1 and L3: SFT-only is near-best for GSM8K (84.00) but worst for ALFWorld (67.91); RL-only is worst on GSM8K (81.88) but best pure-mode on ALFWorld (93.28); the best-to-worst mode spread exceeds 80 points on ALFWorld [Table 16, §3.4].
Multi-task joint optimization wins only on GSM8K and severely degrades interactive tasks on both 7B-Instruct and 8B-Base (Multi on 8B-Base: ALFWorld 85.82, WebShop 13.00, DeepSearchQA 7.00 vs Free 97.76 / 67.00 / 15.00) [Table 16, §3.4].
DeepSearchQA is structurally intractable across every filed system: best score 15.0 (+2.75 over baseline), 145 submissions in the best run, every method tried (DPO/GRPO/KTO/ORPO/model merging); a 20-seed sweep confirms 3pp evaluation variance — the reported improvement falls inside noise [§3.3, §2.6].
Benchmark has genuine discriminative floor: Gemini CLI + 2.5-flash regresses below baseline on GSM8K (70.13 vs 83.02) and HumanEval (10.98 vs 62.20) on unaligned Qwen3-8B-Base — poor stacks actively harm the base model rather than being no-ops [Table 2 Panel B, §3.2].
Alignment-gap closure from unaligned Qwen3-8B-Base is stack-dependent: strongest stacks (CC+Opus 4.6, Codex+GPT-5.2) approach Instruct-level performance; the weakest regresses below baseline [Panel B Table 2, §3.5].
Runtime instrumentation is treated as a first-class benchmark feature: every submission, code revision, and model artifact recorded per run, with an automated post-hoc analysis module that generates structured case studies — designed for behavioral diagnosis rather than only leaderboard scores [§2.4, §2.5].

Method

The benchmark instance is (task_description, base_model, training_data, evaluation_oracle, time_budget=12h). Each run creates an isolated workspace with read-only links to base model + training data, writable directories for generated code and model outputs, and a grading server the agent may submit candidate models to iteratively — best-so-far scoring within the run. Test sets are not mounted; the grading server returns only a scalar and best-so-far to limit adaptive overfitting.

Six tasks are grouped into three levels: L1 static rule-based (GSM8K, HumanEval with deterministic verifiers), L2 static judge-based (AlpacaEval 2.0 with LLM judge, ~0.5–1pp noise), L3 interactive rollout (ALFWorld with sparse binary rewards over 134 fixed episodes, WebShop with dense continuous rewards over 100 fixed instructions, DeepSearchQA with judge-based ReAct-style search — highest noise at 3pp). Each level adds a structural requirement the previous level does not impose, making progression diagnostic.

Two experiment studies: (1) a 7B-Instruct scaffold comparison (OpenHands / OpenCode driven by GPT-5.4, Claude Code driven by Opus 4.6 in four modes: SFT-only, RL-only, Free, Multi); (2) a Qwen3-8B-Base CLI-agent study pairing three CLI scaffolds (Claude Code, Codex CLI, Gemini CLI) with six driver LLMs (Opus 4.6, Sonnet 4.5, GPT-5.4, GPT-5.2, GPT-4o, gemini-2.5-flash), yielding seven system configurations. Mode constraints in Claude Code are enforced via system-prompt + verified by exhaustive post-hoc workspace audit. The 8B study is partial factorial, single-run — described in the paper as “suggestive evidence” not conclusive proof.

Results

Top three on the 7B-Instruct scaffold panel (12h budget, single run per system):

System	Driver	GSM8K	HumanEval	AlpacaEval	ALFWorld	WebShop	DeepSearchQA
Baseline	—	77.70	75.61	18.90	4.85	69.00	11.75
OpenHands	GPT-5.4	81.24	78.05	18.77	9.41	40.00	7.80
OpenCode	GPT-5.4	82.27	79.27	20.33	19.33	54.50	21.00
Claude Code	Opus 4.6 (best mode)	84.00	81.71	30.76	95.52	84.00	15.00

Claude Code wins 5 of 6 tasks; the ALFWorld gap over OpenCode is +76.19pp, the WebShop gap is +29.50pp. OpenCode wins only on DeepSearchQA (21.00 vs 15.00), where task difficulty near-saturates the ceiling. On the 8B-Base panel, the strongest stack (Claude Code + Opus 4.6) reaches ALFWorld 97.76 (baseline 8.96), while the weakest (Gemini CLI + gemini-2.5-flash) regresses to GSM8K 70.13 (baseline 83.02) and HumanEval 10.98 (baseline 62.20). Operating-mode analysis on Claude Code shows Free mode’s ALFWorld 95.52 comes from a composite supervised pipeline combining trajectory collection with training-evaluation format alignment — described as not strictly online-RL engineering; RL-only’s 93.28 (via SFT warm-up + GRPO with online rollouts) is the cleanest online-RL success in the benchmark [§3.4].

Time-budget behavior mirrors PostTrainBench: Can LLM Agents Automate LLM Post-Training?: interactive tasks benefit most from extended budgets; static tasks plateau earlier. A pass@k analysis (Appendix) confirms that rare runs discover qualitatively better pipelines rather than gains being uniform across attempts.

Why it’s interesting

Agent² RL-Bench slots directly into the Agentic Software Engineering and Reasoning RL concept pages as the next structural rung beyond PostTrainBench: Can LLM Agents Automate LLM Post-Training?. PostTrainBench found that agents can partially close the base→instruct gap via supervised recipes on static benchmarks; this paper isolates the interactive rung and finds that agents mostly cannot do it — with one striking exception (ALFWorld under strong stacks). Two contributions transfer to internal practice: (a) driver sensitivity is 4-5× larger on interactive tasks than static ones within the same scaffold, which is the strongest filed evidence that scaffold-vs-driver disentanglement matters for agentic RL evaluation; (b) SFT-fallback remains the dominant behavior even when agents nominally understand they should implement RL — matching the Reasoning RL page’s claim that GRPO is used almost universally as stage-two refinement after SFT rather than from-scratch RL. The DeepSearchQA structural-intractability finding (+2.75pp inside 3pp noise across every stack + method combination) is a useful stress-test datapoint for future search-agent RL claims — it complements ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents‘s rubric-graded evaluation of deep-research agents and suggests the same ceiling appears when agents try to train deep-research policies themselves. The paper’s operational-mode analysis (SFT-only vs RL-only vs Free vs Multi rank ordering reversing between L1 and L3) is the first filed quantification that “which paradigm the agent picks” is a hidden variable that mode-blind benchmark scores hide — argues for reporting the full model-agent-mode stack.