Self-Play Data Generation

TL;DR

Self-play data generation produces post-training data by running model instances against each other (or against role-defined siblings) in a structured environment, then using the environment outcomes — votes, tool returns, downstream task success — as supervision. Across the three filed papers the recipe spans from cooperative role-playing pipelines that synthesize tool-use trajectories (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing) through multi-paradigm trajectory synthesis using mixed LLM drivers (MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling) to strict zero-sum competitive self-play with role-asymmetric reward shaping (Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play). The shared bet: a well-designed environment + a verification or game-outcome signal can replace human-annotated trajectories at multiple orders of magnitude lower cost.

Key claims

• Closed data→model→data loops over agentic environments can lift a 32B base from 19.8% → 70.9% on BFCL using no human labels, with the trained model re-used as the synthesizer in the next round (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing Abstract).
• Strict competitive self-play (a single model plays both sides of a zero-sum game) reaches 71% win-rate against a frozen reference and lifts Qwen2.5-VL-7B by ~3% on six reasoning benchmarks using only 2k–1k image pairs and no human labels (Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play §3.1, Fig. 6, Table 1).
• The verification gate / game-outcome signal is the load-bearing piece: InfTool uses gated rewards over verified trajectories (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing Abstract); MiroThinker filters both noisy-correct (API failures, action loops) and trivially-incorrect (format failures) samples (MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling §5.3); Vision-Zero uses the spy-identification vote as the verifiable reward signal (Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play §2.2, Eq. 4).
• Pure self-play reaches local equilibrium without external supervision: Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play §3.2 ablates this directly — pure clue-stage self-play underperforms Iterative-SPO by ~2% on LogicVista because the decision-maker’s discrimination quality bottlenecks the zero-sum reward signal. The recipe answer is to alternate self-play with verifiable-reward RL, not to replace self-play with it.
• Synthetic trajectories from multi-paradigm pipelines (ReAct + multi-agent + multiple driver LLMs) can avoid single-source overfit and support 256K-context, 600-tool-call rollouts (MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling §4.2, §3.3).
• Self-play data scales to extreme data-efficiency regimes: Vision-Zero trains on 2k CLEVR or 1k chart image pairs (Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play §2.1) vs 155k (R1-OneVision) and 72k (ViGaL) for curated-data baselines on similar tasks (Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play Table 3).
• The recipe transfers across architectures: Vision-Zero gains +1.8% / +1.6% on reasoning average for InternVL3-8B / InternVL3-14B respectively using the same CLEVR data the Qwen2.5-VL recipe used (Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play Table 4) — the environment, not the model, is the load-bearing element.
• Self-play can mitigate cross-capability negative transfer that hurts RLVR baselines: MM-Eureka loses ~10% on ChartQA when math-trained, while Vision-Zero (CLEVR) is within 0.2% on chart/OCR while gaining on vision-centric tasks (Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play §3.1, Table 2). The argument is that strategic multi-role gameplay forces integrated visual+linguistic reasoning rather than task-specific shortcuts.

Recent contributions

• [2026-10-20] Self-Improving Video-to-Action Models: Mirror (Ryu, Chen, Wilcoxson, Sitzmann; MIT) adds a verifier-is-a-video-planner-not-a-reward-or-judge row to the data→model→data loop pattern tracked on this page: a frozen pre-trained video planner proposes plans, the current V2A model tries to execute them, and the resulting (observed-video, executed-action) pairs train the next V2A iteration. Verification source is structurally new vs the existing cells — InfTool uses gated rewards over verified trajectories, MiroThinker uses trajectory-completeness filtering, Vision-Zero uses a zero-sum game outcome, RATs/ASPIRE use robot rollout success — Mirror uses the frozen video planner’s own output distribution as the exploration-pressure and verification source simultaneously. First filed strict embodied instance where the generator-side state (the planner) retains non-trivial in-context memory across attempts: dropped in a Minecraft lava lake, 76% cumulative escape over 30 attempts with history retained vs 46% with context reset. Author framing: “neither RL (no reward) nor conventional behavior cloning (no action-labeled demonstrations)” — a cleaner self-labeling story than the embodied neighbours RATs/ASPIRE which still rely on environment success as the gate.
• [2026-10-16] ASPIRE: Agentic /Skills Discovery for Robotics: ASPIRE paper is the technical report behind the June Fan tweet already filed on this page, and sharpens the “which inner-loop exploration mechanism” question vs RATs (Playful Agentic Robot Learning): evolutionary search over control-program populations, conditioned jointly on top performers and their remaining failure modes, yields +77pp on LIBERO-Pro Object and 4% → 31% zero-shot on held-out LIBERO-Pro Long. Validated-repair admission is the gate (coordinator audits actor-reported repair patterns for API-policy compliance + reusability); failed attempts contribute “remaining failure modes” to the next-round condition rather than being discarded, directly addressing the data-recycling question this page tracks on the text side with InfTool.
• [2026-09-23] Skild AI Messinator — self-play soccer robot from 140 years of simulated World Cup: Skild AI’s Messinator demo — first tabula-rasa physical-robot datapoint on this page: a humanoid soccer policy trained purely by competitive self-play in physics simulation (140 simulated years, each generation vs. progressively stronger prior self), then deployed on real hardware. No paper, video-only, no ablations — but the first explicit AlphaGo-style self-play → embodied deployment claim, and it skips the imitation-learning warm start used by other embodied self-improving loops (RATs, ASPIRE).
• [2026-06-30] ASPIRE — self-evolving sensorimotor skill library that compounds across robots and the sim-to-real gap (Jim Fan / NVIDIA GEAR): Jim Fan announces ASPIRE (NVIDIA GEAR), the second filed embodied datapoint after RATs (Playful Agentic Robot Learning) for the data→model→data loop closing through robot rollouts — coding agents run evolutionary search over control programs against multimodal sensory traces from sim and real robots, with successful programs distilled into a persistent code skill library. Adds evolutionary search over programs as the inner-loop mechanism (distinct from RATs’ proposer/planner/diagnoser and from ENPIRE’s coding-agent-driven policy RL).
• [2026-06-20] Playful Agentic Robot Learning: RATs is the first filed embodied instance of the data→model→data loop tracked on this page: a Code-as-Policy agent self-proposes novel-yet-learnable manipulation tasks, executes them on robots in LIBERO-PRO / MolmoSpaces, and distills successful executions into a persistent code skill library. The verification gate is a physical/simulated robot rollout (goal verifier + per-step verifier + diagnoser) rather than a string match or a game-outcome vote, and the resulting skill library yields +20.6 / +17.0 pp on held-out downstream tasks and transfers cross-agent without finetuning.
• [2026-05-25] Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play: First strict-self-play datapoint on this page. A single VLM plays a 4-player “Who Is the Spy?” image-difference game where one player gets a subtly-modified image and the others get the original. Iterative-SPO alternates zero-sum self-play in the Clue Stage with GRPO RLVR in the Decision (voting) stage, gated by EMA-smoothed accuracy and “n/a” rate thresholds to avoid equilibrium collapse. Reaches ~3% reasoning-benchmark gain on Qwen2.5-VL-7B and beats curated-data baselines (MM-Eureka, ViGaL) trained on 10–100× more data. Also: cross-capability negative-transfer mitigation, the first reported datapoint where the self-play paradigm itself appears to suppress shortcut learning.
• [2026-05-22] MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling: Multi-paradigm trajectory synthesis (ReAct + MiroFlow multi-agent, function calling + MCP, multiple driver LLMs) feeds a three-stage SFT → DPO → online-GRPO pipeline with streaming rollouts to handle long-tailed agentic trajectories of up to 600 tool calls. Trajectory curation filters both noisy-correct and trivially-incorrect samples. 72B model reaches 47.1% BrowseComp / 81.9% GAIA-Text-Only.
• [2026-05-22] Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing: Three-role pipeline (User Simulator + Tool-Calling Assistant + MCP Server) synthesizes verified tool-use trajectories. After each GRPO training round with gated rewards, the improved model becomes the next round’s Tool-Calling Assistant — explicitly targeting prior failure modes. 32B base: 19.8% → 70.9% on BFCL.

Open questions

• Does the data→model→data loop in Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing genuinely transfer off-distribution, or do BFCL gains reflect the synthesizer’s own distribution? No filed paper isolates this for the tool-use setting.
• Vision-Zero’s cross-capability transfer claim (Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play §3.1) is the strongest argument that self-play differs in kind from curated-data RLVR — but the comparison is against post-trained models on different data, not against Vision-Zero with shuffled/non-game training. A clean within-recipe ablation would isolate whether the game environment or just the data scale drives the result.
• At what task difficulty does pure self-play work without an alternating RLVR stage? Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play §3.2 reports pure self-play plateaus on Vision-Zero’s environment — but doesn’t isolate whether the bottleneck is the reward signal’s weakness (decision-maker discrimination) or a more fundamental equilibrium-collapse phenomenon.
• Multi-paradigm synthesis vs single-recipe synthesis: MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling uses ReAct + multi-agent + multiple drivers explicitly to avoid single-source overfit, but doesn’t ablate the per-source contribution. Is the diversity what matters, or just the volume?
• Does “self-play as data generator” generalize beyond text/VLM agents to generative models (image, video, audio)? Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play gestures at this — image-difference reasoning enables vision-centric improvements — but no filed paper applies self-play to a generative model’s output (e.g. two generators competing on a discriminator’s judgment, beyond a single GAN-style game).
• The three filed recipes differ on a key axis: are the players cooperative (InfTool, MiroThinker — synthesize good trajectories together) or competitive (Vision-Zero — zero-sum)? Both routes work; no filed paper directly compares them on the same task.

Papers

2026-01

• SETA: Scaling Environments for Terminal Agents (env repo) — SETA: Scaling Environments for Terminal Agents (env repo) (published 2026-01-09)

2025-12

• Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing — Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing (published 2025-12-29)

2025-11

• DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning — DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning (published 2025-11-26)
• MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling — MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling (published 2025-11-14)

2025-09

• Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play — Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play (published 2025-09-29)