Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play

TL;DR

Vision-Zero is a self-play framework for VLM post-training that requires no human annotations: a single model plays a “Who Is the Spy?” social-deduction game over subtly-edited image pairs, with civilians describing one image and a spy with a different image trying to blend in. A new optimization algorithm, Iterative-SPO, alternates between zero-sum self-play on the clue stage and GRPO RLVR on the decision (voting) stage to avoid the equilibrium collapse common in pure self-play. Trained on 2k CLEVR / 1k chart / real-world image pairs with no human labels, Qwen2.5-VL-7B picks up ~3% absolute gains on reasoning benchmarks (MathVista, MathVision, LogicVista) and ~1.4% on vision-centric tasks (MMVP, BLINK, MuirBench), outperforming RLVR baselines like MM-Eureka and ViGaL trained on tens to hundreds of thousands of curated samples. Accepted to ICLR 2026.

Key claims

A “Who Is the Spy?” image-pair game is the proposed self-play environment: 4 players are issued one image, one is a “spy” whose image has subtle edits (added / removed / modified object), and players alternate verbal clue rounds followed by a private vote to identify the spy [§2.1].
The environment is designed to satisfy four conditions for productive self-play: skills transfer to target tasks, difficulty scales as agents improve, multimodal complexity is sufficient, and input data is label-free or near-free [§1].
Data is domain-agnostic and label-free: CLEVR scenes are procedurally rendered (~6 A100 GPU-hours for 2k pairs), Chart pairs are produced by swapping numerical attributes via Gemini 2.5-Flash on 1k ChartQA images, and Real-World pairs use Nano Banana / ChatGPT image editors — total dataset cost on the order of tens of dollars [§2.1].
Iterative-SPO alternates two distinct RL objectives keyed off live training signal: zero-sum self-play with Role-Advantage Estimation in the clue stage, and GRPO with a discrete (correct vote / “n/a” / wrong) reward in the decision stage [§2.2, Eq. 1, Eq. 4].
Stage switching uses hysteresis thresholds over EMA-smoothed decision-stage accuracy and “n/a” rate, with a minimum dwell time per stage to prevent chattering — equilibrium triggers a switch to clue training; difficulty triggers a switch back to decision training [§2.2, Eq. 9-10].
VisionZero-Qwen-7B (CLEVR) reaches 44.1 average across 6 reasoning benchmarks vs 41.1 for the Qwen2.5-VL-7B base and 43.0 for ViGaL-Snake+Rotation (the strongest curated-data baseline) [Table 1].
All three Vision-Zero variants (CLEVR/Chart/Real-World) outperform GPT-4o on chart/OCR tasks: 84.4 vs 81.1 (Vision-Zero Chart vs GPT-4o average across 7 chart/OCR benchmarks), driven by +4.2% on InfoVQA and +1.3% on AI2D over the Qwen2.5-VL-7B base [§3.1, summary table].
The framework mitigates cross-capability negative transfer that hurts RLVR baselines: MM-Eureka-Qwen-7B drops ~10% on ChartQA when trained on math/reasoning data, while Vision-Zero (Chart) improves chart/OCR and vision-centric tasks simultaneously (+1% on the vision-centric average) [§3.1, Table 2].
Iterative-SPO outperforms both pure clue-stage self-play and pure decision-stage RLVR in ablation: pure self-play plateaus early due to weak verifiable rewards, and Iterative-SPO adds ~2% over pure self-play and ~1% over pure RLVR on LogicVista [§3.2, Fig. 7].
The recipe generalizes across architectures: VisionZero-InternVL3-8B gains +1.8% and VisionZero-InternVL3-14B gains +1.6% on the 6-task reasoning average using only the CLEVR dataset, despite InternVL3 having a different vision encoder from Qwen2.5-VL [Table 4].
Average token length in the Decision stage grows from ~250 to ~400 tokens over training for InternVL3-8B and InternVL3-14B, suggesting Iterative-SPO induces longer reasoning chains rather than just shifting policy distributions [§3.1, Fig. 6].
Total training data is 2k image pairs (CLEVR) — multiple orders of magnitude less than curated-data baselines (R1-OneVision: 155k; ViGaL: 72k; MM-Eureka: 15k) [Table 3].

Method

The training environment is a 4-player image-difference variant of social-deduction games like “Who Is the Spy?”. For each episode, the environment samples an image pair (original, modified) from one of three datasets: 2k procedurally-rendered CLEVR scenes (modified version has two objects changed in color+shape), 1k chart pairs (modified chart has numerical attributes swapped via Gemini 2.5-Flash), or real-world pairs edited with Nano Banana / ChatGPT. Three civilians receive the original image; one spy receives the modified image. In the Clue Stage, players take turns emitting natural-language descriptions of their image (with private chain-of-thought hidden from others, public clues visible to subsequent players). After 2 clue rounds the Decision Stage runs: each civilian privately reasons over all clues + their image and votes for the suspected spy (or “n/a” if uncertain). The spy does not vote.

Iterative-SPO assigns a different RL objective to each stage. The clue stage uses a zero-sum reward (Eq. 1): the spy’s reward is positive when civilians receive more votes than average; civilians receive penalties for receiving votes and a behavioral-inconsistency term. Role-Advantage Estimation (RAE) maintains EMA-smoothed per-role baselines that get subtracted from the reward to neutralize asymmetric win-probability from role information. The decision stage uses a discrete GRPO reward: +1 for correctly voting the spy, 0 for “n/a”, -1 for an incorrect vote — explicitly designed to encourage uncertainty acknowledgement over false confidence (Eq. 4). Group normalization over per-round difficulty is applied (Eq. 5) and the policy is updated with the standard GRPO advantage-weighted log-likelihood + KL regularizer (Eq. 6). Stage switching is gated by EMA-smoothed decision-stage accuracy and “n/a” rate (Eq. 7-8): when accuracy is high and “n/a” rate low (decision stage saturated → clues too revealing), training switches to the clue stage to make clues harder; conversely, when accuracy drops or “n/a” rises (clues too vague), training switches back to decision (Eq. 9-10), with a minimum dwell time M_min to prevent oscillation. Models are trained for 100 iterations at batch size 1024 using the VLM-R1 framework, with 4 civilians per round, 2 clue speeches per stage, and clue-reward coefficients (α, β) = (0.5, 0.2).

Results

On reasoning benchmarks (MathVista, MathVision, WeMath, MathVerse, LogicVista, DynaMath), VisionZero-Qwen-7B variants reach 43.9–44.1 average vs 41.1 (base Qwen2.5-VL-7B) and 43.0 (ViGaL-Snake+Rotation, the strongest baseline) [Table 1]. LogicVista shows the largest individual gain (47.2 → 50.1, +2.9 absolute for CLEVR variant). On chart/OCR (AI2D, ChartQA, OCRBench, SEED-2), VisionZero-Qwen-7B (Chart) hits 85.8 / 87.2 / 89.0 / 70.9 vs base 84.7 / 86.1 / 88.3 / 70.4, and beats GPT-4o on all four [§3.1 table]. On vision-centric (RealWorldQA, MMVP, BLINK, MuirBench, CRPE), Vision-Zero (Real-World) reaches 68.0 average vs 66.6 base, with +2.7 on MMVP and +2.3 on BLINK. Cross-capability transfer: where MM-Eureka-Qwen-7B drops ~10% on ChartQA when math-trained, Vision-Zero (CLEVR) is within 0.2% on chart/OCR average while gaining on vision-centric. Win-rate vs frozen reference grows monotonically from 50% to 71% on Qwen2.5-VL-7B over training [Fig. 6]. Ablations (Fig. 7): on LogicVista, Iterative-SPO outperforms pure clue-stage self-play by ~2% and pure decision-stage RLVR by ~1% — pure self-play plateaus early due to the decision-maker lacking the discrimination needed to produce useful zero-sum rewards. Average Decision-stage token length grows from ~250 to ~400 tokens through training. Architecture transfer: VisionZero-InternVL3-8B improves +1.8% and VisionZero-InternVL3-14B improves +1.6% on reasoning average using only CLEVR data.

Why it’s interesting

Vision-Zero is the cleanest filed example of self-play as a VLM post-training recipe: prior reasoning-RL entries on the wiki (vlm-gym: RL gym for vision language models (JAX) for VLM GeoGuessr, Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning for video MLLMs) used GRPO against environment-defined ground truth, and the multi-agent “self-play” papers (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing, MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling) are really synthetic-trajectory pipelines with role-playing rather than competitive policy copies. Vision-Zero is the first wiki entry where the game is the supervision signal and a single model competes against copies of itself across asymmetric roles. The Iterative-SPO alternation is also a direct answer to the “pure self-play reaches local equilibrium” failure mode flagged on Reasoning RL — by routing the verifiable reward through the decision stage and using stage-saturation signals to switch training targets, the recipe avoids the premature-equilibrium pathology without abandoning the self-play primitive. It complements Synthetic Training Data‘s gallery of label-free recipes by adding a new flavor: the “data” is the gameplay itself, generated on the fly from arbitrary image pairs at tens-of-dollars cost. The cross-capability negative-transfer mitigation result is the most surprising claim — RL post-training that avoids breaking other capabilities is a known hard problem, and the implication that strategic multi-role gameplay forces the model to develop integrated visual+linguistic reasoning rather than task-specific shortcuts is testable.