Self-Play Data Generation
Self-play data generation produces post-training data by running model instances against each other (or against role-defined siblings) in a structured environment, then using the environment outcomes — votes, tool returns, downstream task success — as supervision. Across the three filed papers the recipe spans from cooperative role-playing pipelines that synthesize tool-use trajectories (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing) through multi-paradigm trajectory synthesis using mixed LLM drivers (MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling) to strict zero-sum competitive self-play with role-asymmetric reward shaping (Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play). The shared bet: a well-designed environment + a verification or game-outcome signal can replace human-annotated trajectories at multiple orders of magnitude lower cost.
Key claims
Section titled “Key claims”- Closed data→model→data loops over agentic environments can lift a 32B base from 19.8% → 70.9% on BFCL using no human labels, with the trained model re-used as the synthesizer in the next round (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing Abstract).
- Strict competitive self-play (a single model plays both sides of a zero-sum game) reaches 71% win-rate against a frozen reference and lifts Qwen2.5-VL-7B by ~3% on six reasoning benchmarks using only 2k–1k image pairs and no human labels (Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play §3.1, Fig. 6, Table 1).
- The verification gate / game-outcome signal is the load-bearing piece: InfTool uses gated rewards over verified trajectories (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing Abstract); MiroThinker filters both noisy-correct (API failures, action loops) and trivially-incorrect (format failures) samples (MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling §5.3); Vision-Zero uses the spy-identification vote as the verifiable reward signal (Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play §2.2, Eq. 4).
- Pure self-play reaches local equilibrium without external supervision: Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play §3.2 ablates this directly — pure clue-stage self-play underperforms Iterative-SPO by ~2% on LogicVista because the decision-maker’s discrimination quality bottlenecks the zero-sum reward signal. The recipe answer is to alternate self-play with verifiable-reward RL, not to replace self-play with it.
- Synthetic trajectories from multi-paradigm pipelines (ReAct + multi-agent + multiple driver LLMs) can avoid single-source overfit and support 256K-context, 600-tool-call rollouts (MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling §4.2, §3.3).
- Self-play data scales to extreme data-efficiency regimes: Vision-Zero trains on 2k CLEVR or 1k chart image pairs (Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play §2.1) vs 155k (R1-OneVision) and 72k (ViGaL) for curated-data baselines on similar tasks (Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play Table 3).
- The recipe transfers across architectures: Vision-Zero gains +1.8% / +1.6% on reasoning average for InternVL3-8B / InternVL3-14B respectively using the same CLEVR data the Qwen2.5-VL recipe used (Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play Table 4) — the environment, not the model, is the load-bearing element.
- Self-play can mitigate cross-capability negative transfer that hurts RLVR baselines: MM-Eureka loses ~10% on ChartQA when math-trained, while Vision-Zero (CLEVR) is within 0.2% on chart/OCR while gaining on vision-centric tasks (Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play §3.1, Table 2). The argument is that strategic multi-role gameplay forces integrated visual+linguistic reasoning rather than task-specific shortcuts.
Recent contributions
Section titled “Recent contributions”- [2026-06-30] ASPIRE — self-evolving sensorimotor skill library that compounds across robots and the sim-to-real gap (Jim Fan / NVIDIA GEAR): Jim Fan announces ASPIRE (NVIDIA GEAR), the second filed embodied datapoint after RATs (Playful Agentic Robot Learning) for the data→model→data loop closing through robot rollouts — coding agents run evolutionary search over control programs against multimodal sensory traces from sim and real robots, with successful programs distilled into a persistent code skill library. Adds evolutionary search over programs as the inner-loop mechanism (distinct from RATs’ proposer/planner/diagnoser and from ENPIRE’s coding-agent-driven policy RL).
- [2026-06-20] Playful Agentic Robot Learning: RATs is the first filed embodied instance of the data→model→data loop tracked on this page: a Code-as-Policy agent self-proposes novel-yet-learnable manipulation tasks, executes them on robots in LIBERO-PRO / MolmoSpaces, and distills successful executions into a persistent code skill library. The verification gate is a physical/simulated robot rollout (goal verifier + per-step verifier + diagnoser) rather than a string match or a game-outcome vote, and the resulting skill library yields +20.6 / +17.0 pp on held-out downstream tasks and transfers cross-agent without finetuning.
- [2026-05-25] Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play: First strict-self-play datapoint on this page. A single VLM plays a 4-player “Who Is the Spy?” image-difference game where one player gets a subtly-modified image and the others get the original. Iterative-SPO alternates zero-sum self-play in the Clue Stage with GRPO RLVR in the Decision (voting) stage, gated by EMA-smoothed accuracy and “n/a” rate thresholds to avoid equilibrium collapse. Reaches ~3% reasoning-benchmark gain on Qwen2.5-VL-7B and beats curated-data baselines (MM-Eureka, ViGaL) trained on 10–100× more data. Also: cross-capability negative-transfer mitigation, the first reported datapoint where the self-play paradigm itself appears to suppress shortcut learning.
- [2026-05-22] MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling: Multi-paradigm trajectory synthesis (ReAct + MiroFlow multi-agent, function calling + MCP, multiple driver LLMs) feeds a three-stage SFT → DPO → online-GRPO pipeline with streaming rollouts to handle long-tailed agentic trajectories of up to 600 tool calls. Trajectory curation filters both noisy-correct and trivially-incorrect samples. 72B model reaches 47.1% BrowseComp / 81.9% GAIA-Text-Only.
- [2026-05-22] Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing: Three-role pipeline (User Simulator + Tool-Calling Assistant + MCP Server) synthesizes verified tool-use trajectories. After each GRPO training round with gated rewards, the improved model becomes the next round’s Tool-Calling Assistant — explicitly targeting prior failure modes. 32B base: 19.8% → 70.9% on BFCL.
Open questions
Section titled “Open questions”- Does the data→model→data loop in Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing genuinely transfer off-distribution, or do BFCL gains reflect the synthesizer’s own distribution? No filed paper isolates this for the tool-use setting.
- Vision-Zero’s cross-capability transfer claim (Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play §3.1) is the strongest argument that self-play differs in kind from curated-data RLVR — but the comparison is against post-trained models on different data, not against Vision-Zero with shuffled/non-game training. A clean within-recipe ablation would isolate whether the game environment or just the data scale drives the result.
- At what task difficulty does pure self-play work without an alternating RLVR stage? Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play §3.2 reports pure self-play plateaus on Vision-Zero’s environment — but doesn’t isolate whether the bottleneck is the reward signal’s weakness (decision-maker discrimination) or a more fundamental equilibrium-collapse phenomenon.
- Multi-paradigm synthesis vs single-recipe synthesis: MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling uses ReAct + multi-agent + multiple drivers explicitly to avoid single-source overfit, but doesn’t ablate the per-source contribution. Is the diversity what matters, or just the volume?
- Does “self-play as data generator” generalize beyond text/VLM agents to generative models (image, video, audio)? Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play gestures at this — image-difference reasoning enables vision-centric improvements — but no filed paper applies self-play to a generative model’s output (e.g. two generators competing on a discriminator’s judgment, beyond a single GAN-style game).
- The three filed recipes differ on a key axis: are the players cooperative (InfTool, MiroThinker — synthesize good trajectories together) or competitive (Vision-Zero — zero-sum)? Both routes work; no filed paper directly compares them on the same task.
Papers
Section titled “Papers”2026-01
Section titled “2026-01”- SETA: Scaling Environments for Terminal Agents (env repo) — SETA: Scaling Environments for Terminal Agents (env repo) (published 2026-01-09)
2025-12
Section titled “2025-12”- Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing — Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing (published 2025-12-29)
2025-11
Section titled “2025-11”- DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning — DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning (published 2025-11-26)
- MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling — MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling (published 2025-11-14)
2025-09
Section titled “2025-09”- Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play — Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play (published 2025-09-29)