Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty

TL;DR

Underspecified T2I prompts get resolved unilaterally by the model, forcing users into a trial-and-error loop of prompt refinement. This paper (Google DeepMind, Dec 2024) proposes wrapping an off-the-shelf T2I model (Imagen 3) in a proactive agent that (a) parses the prompt into an editable belief graph — entities, attributes, relations, each with LLM-imagined value distributions and importance scores — and (b) asks the user clarification questions targeted at high-uncertainty × high-importance items before generating. Three question-asking variants are prototyped on top of Gemini 1.5. Evaluated with a simulated-user self-play protocol (a second LLM answering from a held-out ground-truth prompt+image) on DesignBench (a new hand-curated benchmark), COCO-Captions, and ImageInWords; the agent reaches ≥2× the VQAScore of single-turn Imagen 3 within 5 turns. The paper is best read as a design-space exploration for interactive T2I, not as a new model.

Key claims

Underspecified T2I prompts are the failure mode being targeted: users struggle to articulate precise intent, forcing repeated prompt refinement to steer a passive model toward the desired output [§1].
The proposed agent architecture is modular: (i) an LLM-produced belief graph over entities / attributes / relations with per-value probabilities and per-item importance scores, (ii) a question-asking strategy that consumes the belief, (iii) a transition step that re-parses the belief from an updated conversation summary after each user response, (iv) an off-the-shelf T2I model for pixel synthesis [§4, §4.2, §4.3].
Belief graphs differ from classical POMDP belief states in dropping the pre-defined predicate set: the LLM proposes entity/attribute/relation names on the fly per prompt, including implicit and background entities not mentioned in the prompt (pet-owner for a pet scene; style, time of day, location) [§3, §4.2].
Question generation is instantiated three ways: Ag1 rule-based over the belief (heuristics over importance + likelihood), Ag2 belief-guided (belief + history fed to LLM), Ag3 direct prompt (question-asking principles written into an LLM prompt); all three use the same belief-graph substrate [§4.1.2].
Question-asking is governed by four principles — Relevance, Uncertainty Reduction, Easy-to-Answer, No Redundancy — with Easy-to-Answer operationalized by having the agent offer answer options drawn from likely values in its own belief [§4.1.1].
Automatic evaluation is done via simulated-user self-play: an oracle LLM (Ag2-style) answers agent questions using a held-out ground-truth prompt and belief graph constructed from it; the agent’s final image is scored against the ground-truth prompt with VQAScore over 15 turns [§5.1, Fig. 2].
Reported result: on DesignBench, COCO-Captions, and ImageInWords the agent achieves ≥2× the VQAScore of a single-turn Imagen-3 baseline within 5 turns of simulated interaction [Abstract, §5.1].
Human study reports ≥90% of subjects found the agents helpful, ~85% found the belief graphs helpful, 58% expected the question-asking feature to deliver value soon or immediately, and preferred agent images over single-turn T2I in >80% of 550 image pairs [Abstract, §1].
DesignBench is a new hand-curated benchmark for the artist/designer use case: aesthetic scenes with multiple entities and interactions, both short and long captions per image, with photo-realistic, animation, and multi-style diversity — targeted at robust multi-turn testing [§1, §5.1.2].

Method

Given an underspecified prompt, the agent runs an LLM in-context (Gemini 1.5, 32K context) to emit a structured belief graph: (a) mentioned entities, (b) implicit entities likely to appear, (c) background entities (style, lighting, location, time of day); each entity carries attributes with distributions over possible values and estimated probabilities, plus importance scores. Pairwise relations between entities are handled the same way. The agent then picks its next question by scoring belief items on (importance × entropy) and asking about the top item, either directly or through an LLM that phrases the question; likely-value options are offered where feasible to keep answers cheap. On receiving a user response, the agent summarizes the full interaction into a single expanded prompt and re-parses the belief from that updated prompt (no incremental Bayesian update — the belief graph is regenerated). At any turn the current prompt is passed to Imagen 3 for image generation. Users can also directly edit belief items — flip existence probabilities, prune attribute values, add relations — through the graph UI without going through the question channel.

Three question strategies are prototyped: Ag1 (rule-based over the parsed belief), Ag2 (belief text + history → LLM), Ag3 (principles-in-a-prompt → LLM). The same belief-graph substrate is shared across all three.

Results

Simulated-user self-play (15-turn cap, VQAScore against held-out ground-truth prompt) on DesignBench + COCO-Captions + ImageInWords: agent reaches ≥2× the VQAScore of a single-turn Imagen-3 baseline within 5 turns [Abstract, §5.1].
Human study (550 prompt-image pairs): agent-generated images preferred over single-turn baseline in >80% of pairs; ≥90% of subjects expected proactive clarification to be helpful; ~85% found belief graphs helpful; 58% thought the question-asking feature could deliver value very soon or immediately [Abstract, §1].
Ablations across Ag1 / Ag2 / Ag3 question strategies and belief-editing vs question-only interaction modes are reported (paper body truncated at fetch; not quantified here).
No comparison to prior multi-turn T2I systems (Mini DALLE 3, Instruct-Imagen, interactive prompt-refinement setups from Vodrahalli & Zou) on a shared metric — the baseline is single-turn Imagen 3 only.

Why it’s interesting

Sits in a mostly-empty region of the wiki’s design space: it’s an agent wrapping a frozen T2I model with a structured belief-graph memory, rather than a new generator or a new reward model. The closest filed analog is The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation‘s argument that CoT state-transition modeling beats atomic-skill mixes for multi-turn planning — this paper commits to the state-transition side, materialized as an editable symbolic graph over the imagined scene, and does the same “regenerate the state representation from a full-history summary” transition The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation lands on. It contrasts sharply with the concurrent multi-turn image-editing benchmarks — EdiVal-Agent: An Object-Centric Framework for Automated, Scalable, Fine-Grained Evaluation of Multi-Turn Editing and WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation score editing trajectories over already-generated images, whereas this paper’s multi-turn loop happens before any pixels are produced, using an LLM belief to elicit constraints. The simulated-user self-play protocol (oracle LLM answers from a held-out ground-truth prompt) is also a data point for VLM-as-Evaluator — it’s the T2I-side sibling of LLM-as-a-Verifier: A General-Purpose Verification Framework‘s continuous-logit verification, and predates most of the filed VLM-judge work by ~a year.