Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning

TL;DR

AWM is an open-source pipeline that synthesizes 1,000 executable, SQL-database-backed tool-use environments from scratch — given only 100 seed scenario names — and exposes them through a unified MCP interface for large-scale agentic RL. The pipeline decomposes synthesis into scenario → tasks → SQLite schema + sample data → MCP toolset → code-augmented LLM-as-Judge verifier, each gated by execution-based self-correction (~1.13 retries on average). Total synthesis cost is $57 per environment with GPT-5; the released dataset spans 1,000 environments × 35 tools × 10,000 tasks × 1,985 lines of code average. Qwen3-thinking 4B/8B/14B trained with GRPO + a hybrid format-correctness + outcome reward on a 526-environment / 3,315-task subset generalize out-of-distribution to BFCLv3, τ-bench, and MCP-Universe, with the 8B reaching 65.94 BFCLv3 overall (vs. 53.83 base) and the 14B reaching 12.29 on MCP-Universe (vs. 8.38 base).

Key claims

AWM synthesizes 1,000 environments at ~$57 each in API cost, with 88.3% / 88.2% / 86.8% success rates on the database / sample-data / env-code stages and ~1.13 average self-correction iterations per stage [§3.3.1, Table 1].
The released environment set is the largest reported open tool-use environment collection: 1,000 environments vs. EnvScaler’s 191, AutoForge’s 10, MCP-Universe’s 11, τ-bench’s 2-3; 35.1 tools per environment vs. EnvScaler’s 18.6; 1,984.7 lines of environment code per scenario [Table 3].
Database-backed (SQLite) state grounds every transition: the LLM infers required entities/attributes/relations from the task set, generates a schema, and populates initial data so every task is executable from start state — distinguishing AWM from LLM-simulated environments (hallucination-prone, expensive per-step) and concurrent NoSQL/key-value-store approaches (AutoEnv, EnvScaler) [§3, §3.3.1].
Verification is code-augmented LLM-as-a-Judge, not pure code or pure LLM: code-based checks extract structured signals (DB-state diffs before/after rollout); an LLM-as-Judge combines those signals with trajectory context to assign one of {Completed, Partially Completed, Agent Error, Environment Error}. The paper argues pure code-driven verification is brittle to “transient failures, partial executions, or infrastructure issues” common to even real services [§3.3.1].
GRPO training uses a hybrid reward: step-level format check (any tool-call-format violation triggers early termination with negative reward); task-level outcome from the code-augmented Judge, broadcast to all steps on successful termination [§4.1, Eq. 1].
History-aware training addresses a training/inference distribution mismatch: most RL frameworks optimize over full histories in one forward pass, but inference uses framework-truncated histories. AWM applies the same truncation during optimization, splitting each trajectory into sub-trajectories each conditioned on its own truncated history [§4.2, Eq. 3].
Headline results, evaluated out-of-distribution (the synthesis pipeline targets no benchmark): on BFCLv3 the 8B improves overall 53.83 → 65.94 (+12.11), beating Simulator (52.53) and EnvScaler (36.83); on MCP-Universe the 8B improves 6.70 → 11.17 with large gains in Financial (22.5 → 35.0) and Location (0.00 → 8.57); on τ-bench Pass@1 the 8B improves 26.44 → 33.45 [Table 4].
LLM-simulated environments (using GPT-5 as the transition function for the same tasks/tools) consistently underperform AWM’s executable environments on BFCLv3 and MCP-Universe, isolating the “executable vs. simulated” axis [Table 4, Simulator vs. AWM rows].
EnvScaler (concurrent, 191 env, task-set-conditioned synthesis) regresses on BFCLv3 (–8.93 avg) and MCP-Universe (–1.39 avg) relative to base, while AWM improves over base on all three benchmarks — the paper attributes this to EnvScaler’s reliance on existing task sets potentially overlapping with τ-bench [§5.2].
Training was done on a 526-env / 3,315-task subset with 1,024 isolated environment instances per RL step, max interaction length 20, sliding window 4K — scale that depends on the environments being fully parallel-isolated and resettable [§5.1].

Method

The pipeline mirrors how software is built: starting from a high-level scenario description (e.g. “online shopping platform”), it (1) generates likely user tasks in that scenario as functional requirements, (2) generates a SQLite database schema + sample data sufficient to make every task feasible, (3) generates an MCP-exposed Python toolset (action space + observation space, ~35 tools per env), and (4) generates per-task verification code that inspects DB state before/after agent execution. Each stage runs in an isolated sandbox; if execution fails, the error message is fed back to the LLM for up to 5 self-correction iterations.

At training time, AWM frames each environment as a POMDP and runs GRPO with the hybrid format + outcome reward described above. The history-aware variant of GRPO (§4.2, Eq. 3) decomposes a full rollout into sub-trajectories each conditioned on the truncated history a real-time framework would feed at inference — explicitly closing the train/inference distribution gap that arises from sliding-window context management at deployment time.

Results

BFCLv3 overall (8B): 53.83 → 65.94 (+12.11); Non-Live 59.58 → 80.44; Live 58.03 → 72.39; Multi-Turn 43.88 → 45.00; modest regression on Hallucination 76.42 → 70.80 (attributed to the format reward always incentivizing tool use vs. refusal) [Table 4].
BFCLv3 overall (4B): 54.92 → 64.50; (14B): 61.25 → 70.18, beating both Simulator (67.68) and Base.
τ-bench Pass@1 (8B): Airline 26.50 → 38.50; Telecom 18.42 → 23.47; overall 26.44 → 33.45. AWM is “competitive with EnvScaler and consistently exceeds Simulator” but does not target conversational multi-turn dialogue, which τ-bench stresses [§5.2].
MCP-Universe overall: 4B 6.15 → 6.70, 8B 6.70 → 11.17 (best in class), 14B 8.38 → 12.29 (best in class). Largest gains on Financial and Location. AWM excludes browser automation and information retrieval, both central to MCP-Universe.
EnvScaler regresses below the base model on BFCLv3 (e.g. 8B 53.83 → 36.83) — a strong negative result against task-set-conditioned synthesis when the synthesized tasks may overlap with the eval distribution.
Pipeline cost breakdown: Scenario $0.43, Task$ 0.56, Database $3.59, Sample Data$ 13.75, Toolset Schema $23.74, Env Code$ 12.81, Verification $2.21 — total$ 57.09 per environment [Table 1]. Toolset schema dominates because it must produce the documentation surface agents actually consume.

Why it’s interesting

This is the strongest filed datapoint for environment synthesis as distinct from trajectory synthesis or task synthesis — the three threads already on Synthetic Training Data target labels/trajectories, while AWM targets the entire executable RL substrate. The closest comparison is Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing (InfTool), which synthesizes trajectories against given MCP servers using a three-agent role-play loop; AWM operates one layer down, synthesizing the MCP servers themselves. Combined with InfTool, the open agentic-RL stack is now end-to-end synthesizable from seed scenario names through trained agent, with no human task curation. The history-aware GRPO formulation (§4.2) is also a useful artifact: it names and addresses a train/inference mismatch that other long-horizon agent training papers — including MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling which uses streaming rollouts under similar sliding-window inference — paper over with infrastructure rather than the loss.