Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence?

TL;DR

Agentic-MME is a process-verified benchmark for multimodal agentic capability — i.e. MLLMs that act on images (crop, rotate, enhance, flip, resize, …) and also call open-web search, in one workflow. 418 tasks across 6 domains and 3 difficulty levels are paired with ~2,000 stepwise human-annotated checkpoints, dual-axis (Visual / Search) sub-scores, and an “overthinking” efficiency metric measured against human reference trajectories. Each agent is run under two unified interfaces — sandboxed Python code (Code mode) and structured function-calling (Atomic mode) — letting the harness compare tool competence across interaction styles. The headline finding: even Gemini 3 Pro tops out at 56.3% overall and 33.3% on Level-3 vs. 82.3% for humans, with frontier multi-step planning and reliable tool execution still the bottleneck rather than perception.

Key claims

The benchmark explicitly couples Visual Expansion (active image manipulation) with Knowledge Expansion (open-web search) in a single workflow, with 13 visual operations and 4 retrieval tools; previous benchmarks evaluate the two modes in isolation [§1, Table 1].
Tasks are stratified by interaction complexity: Level 1 = a single visual op (48.6% of tasks, ~2.89 checkpoints), Level 2 = visual + search chain (32.1%, ~4.64 checkpoints), Level 3 = synergistic interleaved visual↔search loops (19.4%, ~6.67 checkpoints, avg 4.07 tool calls) [Table 2, §2.2].
Process verification is implemented along two complementary axes: the S-axis scores per-hop search keywords / reference URLs / intermediate answers; the V-axis is split into intent-to-invoke and faithfulness-of-intermediate-artifact, with the latter judged via an MLLM-as-judge against human ground-truth intermediate images [§2.2, §3.1].
Data is constructed via model-in-the-loop backward drafting: annotators first prompt a SOTA model (Gemini 3 Pro) to describe the raw image, target details the model misses or hallucinates, apply visual tools to surface the evidence, and verify the same model then perceives it correctly — so the question is guaranteed to require active manipulation [§2.3].
Quality control combines stepwise oracle testing (walking SOTA models through the human reference trajectory with ground-truth intermediate observations; tasks are retained only if the model then reaches the correct final answer) with two independent human verifiers per instance [§2.4].
A unified execution harness normalizes code-mode and tool-mode into a single auditable event stream, with an AST-based tracer extracting canonical visual operations from executed code traces so checkpoint matching is consistent across coding styles [§2.5].
Best model (Gemini 3 Pro, Atomic mode) reaches 56.3% overall, 70.6% Level-1, 47.4% Level-2, 33.3% Level-3 vs. human 93.8 / 99.0 / 92.6 / 82.3 [Table 3].
Tools are essential on hard tasks: Gemini 3 Pro without tools scores 7.5% on Level-3; with tools (Atomic) it reaches 33.3%, a ~4.4× lift, but is still ~50 points below humans [Table 3, §3.2 Finding 1].
Closed-source vs. open-source gap is concentrated in search/planning, not perception: open-source agents (Thyme-rl, Deepeyesv2) collapse to S-axis scores below 5% — i.e. they invoke tools but fail to formulate useful search queries [§3.2 Finding 2, Table 3].
Structured tool APIs (Atomic) match or beat code generation (Gen) across all evaluated models; GPT-5.2 in Gen mode rarely emits image-processing code (V₁<10%), and switching to Atm raises V₁ above 70% [§3.2 Finding 3, Table 3].
Final-answer scoring is made deterministic by constraining the output space at task-design time (specific units, categorical options, short strings) with an accepted-variant list — explicitly avoiding LLM-as-a-judge for the final score, even though MLLM-as-judge is used for the V₂ artifact axis and search-hop verification [§2.3, §3.1].

Method

Agentic-MME’s pipeline has three pieces. (1) Data: 430 high-res images sourced across 6 domains × 35 sub-categories, deliberately curated so the answer requires cropping, rotation, photometric enhancement, flipping, or resizing. Tasks are drafted via the model-in-the-loop backward protocol so that the targeted SOTA model fails passively but succeeds after the annotator’s manipulation; stepwise annotation records, for every action, a natural-language description of intent, the structural tool/operation, intermediate ground-truth images paired with a targeted verification question, and — for search hops — keywords, human-verified URLs, and expected intermediate answers. (2) Harness: A unified execution layer runs each agent either as sandboxed Python (instrumented with save-path redirection and AST tracing) or via structured function calls (atomic visual ops + retrieval APIs), producing a fully logged replayable trace of actions, observations, and artifacts. (3) Scoring: Final-answer Acc is regex-deterministic. S-axis is the fraction of search-hop checkpoints passed (LLM-judged against URLs / keywords / intermediate answers). V-axis splits into V₁ (proactive intent to invoke at the right step) and V₂ (MLLM-judged faithfulness of the intermediate visual artifact against human ground truth). The Overthink metric penalizes redundant tool calls relative to the human minimal trajectory.

Results

Overall ranking on Acc (Atm mode unless noted): Gemini 3 Pro 56.3, Gemini 3 Flash 52.2, Qwen3.5-plus 47.1, Kimi-k2.5 44.0, GPT-5-mini 37.7, GPT-5.2 35.0; best open-source Qwen3-VL-235B 34.9, Deepeyesv2 25.2, Thyme-rl 23.0; Human 93.8 [Table 3].
Level-3 is where the gap blows up: best model Gemini 3 Pro 33.3% vs. human 82.3%; Thyme-rl 2.5%, Deepeyesv2 3.8% [Table 3].
For Gemini 3 Pro the Atm vs. Gen lift on Level-3 is 23.0 → 33.3 (Acc), with V₂ jumping from 54.1 → 87.5 — the structured API surface makes intermediate artifacts much more faithful to ground truth [Table 3].
S-axis is the dominant blocker for open-source: Qwen3-VL-235B reaches ~20.5 (Atm) vs. ~29 for closed-source frontier; Thyme-rl/Deepeyesv2 sit below 5 [§3.2 Finding 2, Table 3].
43.1% of instances place decisive visual evidence in regions covering <1.5% of image area; >40% require recovering localized information — i.e. passive perception over the full image is structurally insufficient [Table 2, §2.4].

Why it’s interesting

Sits at the exact intersection of three live concept pages on the wiki. (i) For Tool-Use Agents, it’s the first filed benchmark that unifies visual tool use and open-web search in one workflow with stepwise process verification — directly probing the “interaction depth” / S-axis the page currently debates between MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling and Training-Free Group Relative Policy Optimization, and it answers the page’s open question on Atomic vs. code interfaces with a clear empirical “Atomic ≥ code across all models”. (ii) For Thinking with Modalities, Agentic-MME is the first benchmark explicitly targeting substrate (c) — the external-tool Think-Act-Observe loop productized by Introducing Agentic Vision in Gemini 3 Flash — with process verification that decouples intent to use the tool from faithfulness of the resulting artifact, exposing GPT-5.2’s Gen-mode failure to even invoke image ops as a different bottleneck from execution quality. (iii) For VLM-as-Evaluator, it adopts the per-sample, prompt-decomposed MLLM-as-judge shape (same family as IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation and RISE-Video: Can Video Generators Decode Implicit World Rules?) for the V₂ artifact axis, while deliberately keeping final-answer scoring rule-based — landing on a hybrid position between the A Very Big Video Reasoning Suite “rules are better” stance and the LMM-as-judge consensus.