Skywork-R1V4: Toward Agentic Multimodal Intelligence through Interleaved Thinking with Images and DeepResearch

TL;DR

Skywork-R1V4 is a 30B (3B-active) multimodal agentic model that unifies four capabilities in a single ReAct loop: structured planning, image manipulation via Python sandbox, multi-modal web search (Google Lens + text + webpage retrieval), and — critically — interleaved alternation between visual operations and search within the same trajectory. The model is trained with supervised fine-tuning on fewer than 30K trajectories, validated by stepwise reasoning–image consistency filtering and o3-mini answer agreement. It scores 66.1 on MMSearch and 67.2 on FVQA, beating Gemini 2.5 Flash on all 11 reported metrics, and reaches roughly 4× the inference throughput of Gemini 2.5 Flash on single-round evaluation [§3.1, §3.3].

Key claims

Sophisticated agentic multimodal intelligence is achievable through supervised fine-tuning alone — no reinforcement learning is required when the SFT data is carefully curated for planning-execution consistency [§1, Abstract].
Skywork-R1V4 beats the Qwen3-VL 30B-A3B baseline by +47.4 on MMSearch (66.1 vs 18.7), +13.9 on FVQA (67.2 vs 53.3), and +8.4 on BrowseComp-VL (38.4 vs 30.0), and outperforms Gemini 2.5 Flash on all 11 reported metrics [§3.1, Table 2].
Step-wise consistency filtering is the load-bearing curation gate: trajectories are discarded when the final answer contradicts the last thinking step, or when an image produced at step k (e.g. a blank crop) does not match the subsequent reasoning that claims to use it [§2.1, “Outcome and Step-wise Filtering”].
Removing data with sandbox errors or “re-cropping” (correction-after-mistake) steps improves SFT performance, because the model otherwise learns to reproduce the erroneous-then-fix pattern; the trained model still recovers gracefully from such errors at inference time despite never seeing them in training [§2.5, “Low-Quality Data Removal”].
Enhanced multimodal search data is generated via a constrained random walker over an encyclopedia knowledge graph: a seed entity yields a verifiable answer, then questions are iteratively rewritten to refer to entities indirectly via their linked neighbours, with an LLM uniqueness evaluator filtering generic referents [§2.2.2, “Enhanced Text Query Generation”].
Text-to-multimodal reformulation grounds the final entity in an image (e.g. rewriting “Loïc Féry” to “the person in this picture”) via property-conditioned image search + LLM rewrite, ensuring the visual modality is load-bearing rather than decorative [§2.2.2, “Text-to-Multimodal Query Reformulation”].
Visual perception is the bottleneck when synthesizing interleaved trajectories: Claude-4-Sonnet’s image-crop misalignment dominates trajectory failure, and an additional VLM-based consistency filter is required on top of the search/answer-consistency filters used for pure-search and pure-image data [§2.3].
Mix-mode training across the four task types yields mutual promotion — training the planner improves search-task performance; general VQA data improves attribute/relation judgment in image manipulation [§2.5, “Mix-Mode Training”].
3B active parameters give a roughly 4× inference speed advantage over Gemini 2.5 Flash and ~15× over Gemini 2.5 Pro on single-round MMSearch / FVQA evaluation; in end-to-end multi-round mode with code+search tools the model maintains ~2× tokens-per-second [§3.3, Fig. 3].

Method

The training pipeline assembles four data flavors, each with its own curation gate, then fine-tunes (no RL) under task-specific system prompts. (1) Think-with-Image: 1024² images from Thyme-RL / Fine-Vision are paired with questions; GLM-4.5V and Claude-4 propose Python code (crop / rotate / contrast / pixel inspection), executed in a sandbox; 4 rollouts per query are retained when the final answer matches ground truth, then filtered for reasoning–answer consistency and step-image–subsequent-thinking consistency [§2.1]. (2) Basic search: FVQA queries, Claude-4-Sonnet generates ReAct trajectories using Serper Image Search / Text Search / Webpage tools (tagged <image_search>, <text_search>, <web_content>), with format filtering and o3-mini answer-agreement filtering; Qwen3-32B summarizes long webpages [§2.2.1]. (3) Enhanced search: a constrained random walker over an offline encyclopedia builds multi-hop QA, rewritten iteratively so the question refers indirectly to entities via their relations; text queries are converted to multimodal by image-grounding the final entity [§2.2.2]. (4) Interleaved trajectories: 3K LiveVQA samples, Claude-4-Sonnet generates traces that alternate image manipulation and search, with an additional VLM-based filter for image-crop validity [§2.3]. The Multi-Modal Agentic Planner is trained on structured re-renderings of all of the above, where each step is (natural-language sub-task, tool name, parameter field) and inter-step dependencies are encoded as symbolic placeholders like “[Result from Step 3]” [§2.4]. Final SFT mixes all four flavors with task-specific system prompts and in-house non-think attribute/relation/VQA data [§2.5].

Results

On perception, Skywork-R1V4 gains +3.3 → +14.4 over the Qwen3-VL 30B-A3B baseline across HRBench-4K / HRBench-8K / MME-Real / MME-Real-CN / MME-Real-Lite / V* / TreeBench / Visual Probe, e.g. 91.8 (+3.3) on HRBench-4K FSP, 90.4 (+8.7) on V* Attribute, 88.0 (+5.8) overall V*, 59.4 (+14.4) on MME-Real-CN Reasoning [Table 2]. On deep multimodal search the gains are dramatic: 66.1 MMSearch (+47.4), 67.2 FVQA (+13.9), 38.4 BrowseComp-VL (+8.4), beating Gemini 2.5 Flash on 11/11 metrics and Gemini 2.5 Pro on 5/11 including V* (88.0 vs 79.1) [Table 2, §3.1]. Inference speed: roughly 4× Gemini 2.5 Flash and 15× Gemini 2.5 Pro on single-round MMSearch/FVQA; ~5× on BrowseComp-VL (subject to repetition behaviour on the harder benchmark); ~2× tokens-per-second sustained in multi-round end-to-end mode with code+search tools [§3.3, Fig. 3]. The model orchestrates 10+ tool calls per task in long-horizon scenarios [Abstract].

Why it’s interesting

This is a clean SFT-only counterpoint to the GRPO-heavy tool-use orthodoxy filed at Tool-Use Agents — MiroThinker / InfTool / Training-Free GRPO all carry their performance via RL or in-context-update equivalents, while Skywork-R1V4 explicitly argues “data quality is all you need” and matches or beats Gemini 2.5 Flash on multimodal search at 30B (3B active). For Thinking with Modalities this paper is the second filed instance of substrate (c) — external-tool image manipulation appended back to context — after Introducing Agentic Vision in Gemini 3 Flash, and the first that releases the data-curation recipe rather than the productized loop. The interleaved-thinking axis (image-op → search → image-op within one trajectory) is the genuinely new design point this paper adds to the wiki.