VLM-Guided Adaptive Negative Prompting for Creative Generation

TL;DR

VLM-Guided Adaptive Negative-Prompting is a training-free, inference-time method that steers a frozen text-to-image diffusion model toward “creative” outputs (a new type of pet, an unconventional building) by querying a vision-language model at each denoising step to identify which conventional concept the current intermediate output most resembles, then accumulating that concept into the negative-prompt list for the remaining steps. The closed-loop VLM → negative-prompt feedback drives the denoising trajectory into unexplored regions of CLIP embedding space while preserving categorical validity. Quantitatively wins on novelty (Total Variance 0.318, Vendi 5.794) and validity (GPT-Score 0.917 with Qwen2.5) against ConceptLab and other creative-generation baselines, with negligible compute overhead.

Key claims

Existing creative-generation methods either interpolate within predefined category embeddings (no out-of-category exploration) or require embedding optimization / fine-tuning (slow), motivating a training-free inference-time alternative [§1, Abstract].
The denoising process is augmented with an adaptive negative-prompt list that grows over the trajectory: at each step a frozen VLM analyzes the intermediate latent (decoded to an image) and identifies the dominant conventional concept (e.g. “cat”), which is appended to the running negative-prompt set used by the standard CFG formula [§3.2, Fig. 3].
Per-generation adaptation (the list is rebuilt for each sample) and dynamic accumulation (concepts persist across steps once detected) are both ablated as load-bearing — neither alone matches the full method [§Ablation].
Novelty/diversity gains are statistical: Total Variance 0.318 and Vendi Score 5.794 (with BLIP-2), broader than ConceptLab (TV 0.289 / Vendi 5.119) and “Creative Prompting” (TV 0.191 / Vendi 3.139), evaluated in CLIP embedding space [Quick Review, Table 1].
Validity (does the generated object still belong to the prompted category?) is preserved: GPT-Score 0.917 with Qwen2.5 as the VLM, vs ConceptLab 0.862 [Quick Review, Table 1].
The method generalizes from single-object creative generation to (a) coherent sets of creative items (tea sets, chess sets) and (b) creative content embedded inside elaborate compositional prompts (“imaginary pet surfing on a board near an island”), neither of which the optimization-based prior art handles [project page; §5].
The compute overhead is “negligible” — one extra VLM forward pass per denoising step, against tens of diffusion steps, with no extra training [Abstract; §6].

Method

Given a prompt like “a new type of pet”, VLM-GANP samples Gaussian noise and runs the standard CFG denoising loop with one modification: the negative-prompt set used by the unconditional branch is rebuilt across timesteps. At each step t:

Decode the intermediate. The latent x_t (or x_0|t estimate) is decoded to an RGB image.
VLM query. A frozen VLM (Qwen2.5, BLIP-2, or similar) is prompted to identify the dominant conventional concept present in the intermediate — e.g. “this looks like a cat”.
Accumulate. The detected concept token is added to a running negative-prompt list N_t = N_{t-1} ∪ {concept}.
Standard CFG step. The next denoising step uses the original positive prompt against the accumulated N_t via the usual CFG extrapolation.

The result is a closed-loop sampling process: as soon as the partially-denoised image starts to look like any known category, that category is added to the suppression set, pushing the remaining steps into the gaps between categories. Because the suppression is per-sample and accumulates, two seeds with the same prompt can land on entirely different novel concepts (one sample becomes a non-cat non-dog non-rabbit pet, another lands somewhere else in the gap).

This is geometrically the opposite of what most CFG variants do: instead of strengthening the positive signal or scheduling the conditioning content, it actively grows the unconditional branch’s specification during sampling, with the VLM as the controller deciding what to push away from.

Results

Headline numbers from the project page and review summaries (the wiki’s fetch of the arxiv body was blocked by upstream rate limiting at filing time, so detailed table/equation pointers default to §-level anchors):

Novelty/diversity (with BLIP-2 VLM). Total Variance 0.318, Vendi Score 5.794 — vs ConceptLab (TV 0.289 / Vendi 5.119) and “Creative Prompting” (TV 0.191 / Vendi 3.139) [Table 1].
Validity (with Qwen2.5 VLM). GPT-Score 0.917 — vs ConceptLab 0.862 [Table 1]. So novelty does not come at a category-coherence cost.
Categories demonstrated. Pets, jackets, buildings (Fig. 1); plant species, fruit shapes, set-level coherence (tea/chess/cutlery/luggage sets); compositional scenes embedding the creative object [project page].
Compute. “Negligible overhead” relative to baseline sampling, because one extra VLM forward per step is dominated by the diffusion step itself for any non-trivial T2I backbone [Abstract].

Why it’s interesting

This adds a genuinely new axis to the Classifier-Free Guidance variants cluster: every entry there so far either (a) redesigns what enters the unconditional branch at training time (Continuous Adversarial Flow Models, Random noise augmentation during diffusion training eliminates the need for CFG, SoFlow: Solution Flow Models for One-Step Generative Modeling), (b) redesigns how the formula combines branches at inference (Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models moves to attention space; Improving Motion in Image-to-Video Models via Adaptive Low-Pass Guidance schedules the conditional content; SparkVSR: Interactive Video Super-Resolution via Sparse Keyframe Propagation swaps the unconditional channel), or (c) decomposes per-modality (LTX-2: Efficient Joint Audio-Visual Foundation Model, HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning). VLM-GANP is the first filed instance that grows the negative prompt itself during sampling, with a VLM in the loop deciding what to suppress — orthogonal to all of the above and trivially composable with them.

It also extends the VLM-as-Evaluator cluster into a new shape: prior entries use the VLM as a static rubric (ERNIE-Image-Aes: Robust Image Aesthetics Scoring with Balanced Category Generalization), per-prompt GRPO reward (Unified Personalized Reward Model for Vision Generation), discrete pick over candidates (DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models), or benchmark-time judge (RISE-Video: Can Video Generators Decode Implicit World Rules?). VLM-GANP is the first filed use of a VLM as an inner-loop controller of the sampling process — its output is not a reward or a pick but a token added to the suppression set, consumed by the very next denoising step. The “VLM proposes, generator disposes” pattern that DiffThinker × MLLM-verifier and UnifiedReward-Flex both demonstrate at the outer loop here happens per denoising step.