ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation

TL;DR

ToolArtist is a post-trained Unified Multimodal Model (UMM) that places the entire open-world image-generation pipeline — reasoning, external tool calls (web search etc.), and native pixel generation — under a single agent policy rather than a fixed workflow or partially-scripted scaffold. Training is two-stage: SFT converts trajectories from a teacher agent equipped with search + external image-generation tools into a UMM-compatible format where the external image tool is hidden but the produced images are kept as targets; then RL uses Reason-Act-Draw GRPO (RAD-GRPO) with complementary intent and quality rewards to jointly optimize reasoning, tool invocation, and generation. The paper argues, and reports experiments showing, that end-to-end agent control beats fixed-pipeline and partial-agent baselines on tasks needing multi-step reasoning or external knowledge, and releases training data plus the post-training infrastructure.

Key claims

Full-pipeline agent control (reasoning + tool use + native generation under one UMM policy) consistently outperforms fixed-workflow and partially agent-controlled baselines on open-world image generation [Abstract].
The SFT recipe distills a teacher agent that uses both search tools and a separate image-generation tool into a UMM by concealing the image tool in the trajectory while retaining the images it produced — turning external tool-generated pixels into native-generation targets [Abstract].
RAD-GRPO (Reason-Act-Draw GRPO) uses two complementary reward channels — an intent reward (does the trajectory address the prompt’s semantic intent?) and a quality reward (image quality) — to jointly train the reasoning, tool-invocation, and generation components in a single policy update [Abstract].
The paper releases training data and the complete post-training infrastructure for agentic RL over UMMs [Abstract].

Method

ToolArtist takes a pretrained Unified Multimodal Model and post-trains it in two stages. Stage 1 (SFT): a teacher agent is equipped with search tools plus a separate external image-generation tool and rolls out trajectories for open-world image tasks; those trajectories are then rewritten into a UMM-compatible format in which the image-generation tool call is hidden and the resulting image is retained as the target output of the UMM’s own native generation head. Stage 2 (RL): the SFT model is optimized with Reason-Act-Draw GRPO — a GRPO variant that treats the whole (reason, act via tool, draw) sequence as a single trajectory and computes group-relative advantage from two rewards, an intent reward capturing whether the trajectory addressed the user’s semantic intent (via reasoning + tool use) and a quality reward on the produced image. The design point is that a single policy dynamically decides whether to think, when to invoke a tool, and how to draw — as opposed to prior work that hardcodes some subset of these into a fixed pipeline.

Results

The paper reports that end-to-end agent control outperforms both fixed-pipeline and partially-agent-controlled baselines on open-world image generation tasks that require complex semantic understanding, multi-step reasoning, or external world knowledge [Abstract]. Specific benchmark numbers are not stated in the retrieved abstract; the load-bearing empirical claim is comparative rather than absolute.

Why it’s interesting

ToolArtist is a rare single-paper crossover between three concept clusters currently on the wiki. It extends Unified Multimodal Models beyond the “understanding + generation share weights” pattern into “understanding + tool use + generation share weights and a single RL policy” — a strictly stronger version of the UMM thesis than the E2E / Decoupled / Agentic taxonomy from UniG2U-Bench: Do Unified Models Advance Multimodal Understanding? currently accommodates, since the agentic branch there still assumes tool orchestration is external to the UMM. It complements Masked Visual Actions for Unified World Modeling‘s “one policy over reasoning + visual output” framing but on the text-to-image rather than the video-world-modeling side. Against Tool-Use Agents, RAD-GRPO is a natural extension of the GRPO-with-shaped-reward recipe that already dominates the tool-use post-training literature (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing, MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling) to a setting where the terminal action isn’t a tool call or a text answer but a generated image — sibling in spirit to InterleaveThinker: Reinforcing Agentic Interleaved Generation (which used a frozen image generator inside an agent loop), but with the generator fine-tuned jointly with the tool-use policy. Against Reasoning RL, the intent-plus-quality reward split is one of the cleanest “two orthogonal reward channels for a multi-stage trajectory” designs filed to date; comparable in structure to Pref-GRPO’s pairwise + point rewards for T2I (Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learning) but with a semantic-intent axis that pairwise preference alone can’t isolate.