SWE-Vision: A Minimal Agent for Advancing Visual Intelligence

TL;DR

SWE-Vision is a minimal multimodal agentic workflow that wraps a frontier VLM (GPT-5.2-xhigh or Seed-2.0-Pro) with a Docker-based Jupyter kernel so the model can think with images by writing executable code that manipulates them — crop, OCR, draw axis lines, measure pixel gaps, parse tables — and append the transformed image back into its context across iterations. The system prompt is deliberately under-specified: it advertises tool availability without prescribing when to invoke code or which algorithms to use, letting the base model’s reasoning drive strategy. Across five vision benchmarks (BabyVision, MathVision, Zero-Bench-Sub, OmniSpatial, plus one more), SWE-Vision consistently lifts the wrapped frontier model — +2.3 to +18.2 points with GPT-5.2 and +1.6 to +3.8 with Seed-2.0-Pro — claiming state-of-the-art on all five. It is the open-source counterpart to Gemini 3 Flash’s “Agentic Vision” feature.

Key claims

A minimal agent loop (VLM + Jupyter REPL + image I/O) on a frontier base model achieves consistent improvement across five diverse vision benchmarks — BabyVision (perception), MathVision (math), Zero-Bench-Sub (multi-step), OmniSpatial (spatial), plus a chart-reasoning benchmark — claiming SOTA on all five [§Benchmarks].
Gains over the base model are larger for GPT-5.2-xhigh (+2.3 to +18.2 points) than for Seed-2.0-Pro (+1.6 to +3.8 points), suggesting the wrapper’s lift is base-model-dependent and not a fixed bonus [§Results].
Reported absolute numbers include 64.4 on BabyVision, 94.0 on MathVision, 50.1 on Zero-Bench-Sub, ~69.0 on a fourth benchmark [§Benchmarks].
The system prompt is intentionally under-specified — it lists available tools but does not prescribe when to invoke code or which algorithms to apply; problem-solving strategy emerges from the base model’s own reasoning [§Method].
The case study (which subplot has the largest gap between two curves at Quarters=15) demonstrates the substrate’s value proposition: the agent first excludes a candidate subplot by inspection, then locates the two curves at the target x-coordinate in each remaining subplot and uses code to measure pixel gaps precisely — replacing the unreliable “eyeballing” common in pure-VLM evaluation [§Case study].
Evaluation uses GPT-5 as LLM-as-judge for answer correctness across all benchmarks, with max reasoning effort and a 100-iteration budget on the agent loop [§Evaluation setup].

Method

SWE-Vision is a VLMToolCallAgent loop: the model receives an image + query, can call a small toolset (Python REPL inside a Docker-isolated Jupyter kernel, image I/O, file manager), and writes code to crop / annotate / measure / plot the image. The kernel state persists across turns, so derived artifacts (crops, computed numbers, generated plots) accumulate as the agent works. The transformed images and code outputs are appended back to the model’s context, producing a Think-Act-Observe loop where “Act” is arbitrary Python and “Observe” is a re-rendered image plus stdout. The library is small (agent.py, kernel.py, image_utils.py, config.py, trajectory.py, plus a Flask web app and a trajectory viewer) and runs against any OpenAI-compatible API endpoint (OpenRouter, etc.), so the same wrapper drops onto different frontier bases without retraining.

Results

Headline numbers across the five benchmarks: 64.4 BabyVision, 94.0 MathVision, 50.1 Zero-Bench-Sub, ~69.0 OmniSpatial (additional benchmark not fully visible). Per-base lifts: GPT-5.2-xhigh sees +2.3 to +18.2 points, Seed-2.0-Pro sees +1.6 to +3.8 points. Three benchmarks (BabyVision, ZeroBench-Sub, OmniSpatial) are reported with the agent configuration; judge is GPT-5 as LLM-as-judge for answer correctness; agent runs with max reasoning effort and a 100-iteration cap.

Why it’s interesting

SWE-Vision is the open-source open-recipe sibling of Introducing Agentic Vision in Gemini 3 Flash — both productize the same “MLLM writes Python to manipulate the image, transformed image re-enters context” substrate that Thinking with Modalities names as position (c) on its substrate axis. Gemini 3 Flash bakes the loop into a frontier API with implicit-zoom trained in; SWE-Vision is a thin agent wrapper any team can fork and point at any frontier base via OpenAI-compatible endpoints. It also adds a fifth datapoint to Tool-Use Agents where the action space is code that manipulates an image rather than web/MCP API calls — extending the design space beyond text-only agents like MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling and Training-Free Group Relative Policy Optimization and beyond visual-primitive agents like VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents (where actions are typed swap/rotate/move primitives, not arbitrary Python). The reported per-base asymmetry (+18.2 max gain on GPT-5.2 vs +3.8 max on Seed-2.0-Pro) is consistent with Training-Free GRPO’s capability-cliff finding — context-space techniques need a strong base.