AdaTooler-V: Adaptive Tool-Use for Images and Videos

TL;DR

AdaTooler-V is a Qwen2.5-VL-7B-derived MLLM post-trained to adaptively invoke vision tools (CropImg, FrameAt, VideoClip, PathTracer) only when they actually help, instead of the “blind tool-use” pattern that current “Thinking with Images/Videos” models exhibit. The core contribution is AT-GRPO: a GRPO variant whose tool-use reward is scaled by a per-sample Tool Benefit Score measuring the empirical accuracy gain from tool-use, so the policy is penalized for redundant invocations on text-CoT-solvable problems and rewarded for productive ones. A two-stage SFT (on a new 100k CoT-trajectory dataset) → AT-GRPO RL (on a new 300k single-image / multi-image / video dataset with verifiable rewards) recipe yields a 7B model that hits 89.8% on V* (beating GPT-4o, Gemini 1.5 Pro, Pixel Reasoner, DeepEyes, Mini-o3) and posts gains across 12 image and video reasoning benchmarks.

Key claims

Existing open-source multimodal interleaved-CoT models exhibit “blind tool-use”: invoking vision tools even when they aren’t needed, which causes overthinking, deviates from the optimal reasoning path, weakens reliance on the original visual input, and inflates inference cost [§1, Fig. 1].
AT-GRPO defines a Tool Benefit Score (S = \text{Acc}\text{tool} - \text{Acc}\text{no-tool}) per sample, estimated by running Qwen2.5-VL-72B-Instruct 8× with and 8× without tools, and uses it to scale the adaptive-tool reward (R_\text{tool} \propto S \cdot \exp(-(n/n_\max)^2/\sigma^2)) so redundant invocations on (S \le 0) samples are penalized and productive ones on (S > 0) samples are rewarded [§3.3, Eqs. 1–3].
The SFT cold-start is load-bearing: skipping SFT (GRPO-from-scratch) drops average accuracy from 69.9 → 65.9 across V*/MathVista/VSI-Bench/MVBench, attributed to the lack of structured priors for tool-interaction reasoning during early RL [§4.3.2, Tab. 3].
AT-GRPO beats vanilla GRPO on top of the same SFT cold-start by +2.3 average points (69.9 vs 67.6) across the same four benchmarks, isolating the adaptive-tool reward shaping as the source of the gain [§4.3.1, Tab. 3].
Tool-use itself is necessary: a text-CoT-only RL variant (“RL-wo-tool”) on the same training data drops 5.4 average points (65.5 vs 69.9) and 6.8 points on VSI-Bench (39.9 vs 46.7), so the adaptive-tool reward isn’t just teaching the model to skip tools [§4.3.4, Tab. 5].
AdaTooler-V-7B reaches 89.8% on V* (vs Qwen2.5-VL-7B base 78.5%, GPT-4o 65.2%, Mini-o3 88.2%) and posts gains on MathVista (74.5), MMSI-Bench (36.8), SPAR-Bench (40.3) [§4.2, Tab. 3.2].
On video benchmarks (32-frame setting), AdaTooler-V-7B reaches 46.7 VSI-Bench / 54.6 VideoMMMU / 68.4 MVBench / 62.5 Video-MME / 55.6 Video-Holmes — Video-Holmes is roughly 2× the Qwen2.5-VL-7B base (27.8) [§4.2, Tab. 3.3].
Performance is robust to the adaptive-tool reward weight λ in the 0.4–0.8 range (68.5–69.9 avg.) and degrades only modestly at λ=0.2, suggesting low hyperparameter sensitivity [§4.3.3, Tab. 4].

Method

The pipeline is a ReAct-style Think → Action → Observation loop where the policy can either emit a single thought and answer (text-CoT path) or iteratively invoke one of four vision tools — CropImg (zoom on a bounding box), FrameAt (extract a video frame at time t), VideoClip (extract a sub-clip), PathTracer (draw a trajectory between two points) — appending each observation back into context. Two new datasets back the training: AdaTooler-V-300k for RL with rule-based verifiable rewards (multiple choice / numerical / OCR-WER / free-form ROUGE) across single-image, multi-image, and video modalities, and AdaTooler-V-CoT-100k, a Qwen2.5-VL-72B-distilled and rule-filtered CoT-trajectory corpus for SFT.

The RL stage uses AT-GRPO: the per-sample Tool Benefit Score (S) (an offline empirical estimate of how much tool-use helps that sample) scales an exponential-decay term over tool-use frequency (n/n_\max), giving (R_\text{tool} = S \cdot \exp(-(n/n_\max)^2/\sigma^2)) with σ=2. The total reward is a weighted sum of (R_\text{tool}) (λ=0.6) plus the standard correctness + format reward from DeepSeek-R1, and the group-relative advantage drives the GRPO policy update. Initialization is Qwen2.5-VL-7B-Instruct; training is 8× H100 with verl-tool / vLLM.

Results

Headline numbers (7B, vs all baselines reported in the paper):

V* (high-res perception): 89.8% — vs GPT-4o 65.2, Gemini 1.5 Pro 71.7, Qwen2.5-VL-7B base 78.5, Pixel Reasoner 84.3, DeepEyes 85.6, Mini-o3 88.2 [Tab. 3.2].
MathVista: 74.5 vs Qwen2.5-VL-7B 68.2 (+6.3) [Tab. 3.2].
MMSI-Bench / SPAR-Bench (multi-image spatial reasoning): 36.8 / 40.3, best in table [Tab. 3.2].
Video-Holmes: 55.6 at 32 frames, 58.3 at 128 frames — vs Qwen2.5-VL-7B 27.8, Video-R1 36.5, Video-Thinker 43.2 [Tab. 3.3].
VSI-Bench: 46.7 (32f) → 49.5 (128f) vs Qwen2.5-VL-7B 29.8/34.8 [Tab. 3.3].
Ablations (V*/MathVista/VSI-Bench/MVBench avg.): GRPO 65.9 → SFT+GRPO 67.6 → SFT+AT-GRPO 69.9; text-CoT RL (no tools) 65.5; λ sweep flat over 0.4–0.8 [Tabs. 3–5].

Why it’s interesting

AdaTooler-V is the cleanest filed datapoint that blind tool-use is itself a failure mode of “Thinking with Images/Videos” recipes — it complements Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning (which trains a video MLLM to emit <grounding>/<caption> tags via SFT→GRPO but does not penalize redundant invocations) and Introducing Agentic Vision in Gemini 3 Flash (productized Think-Act-Observe with Python code execution, where Google flags “deciding when to invoke” as roadmap rather than trained-in behavior). The Tool Benefit Score is a concrete, computable instantiation of the “reward shape, not RL objective” thesis on Reasoning RL — it sits next to InfTool’s gated rewards and HuDA’s perception-stack reward as a third example of GRPO doing the heavy lifting only after the reward is engineered. The result is also a counterweight on Tool-Use Agents: where MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling argues interaction depth (~600 calls) is an independent scaling axis, AdaTooler-V argues the opposite for vision — productive interaction depth is sample-conditional and should be bounded down by the reward when no benefit is expected.