The Unreasonable Effectiveness of Text Embedding Interpolation for Continuous Image Steering

TL;DR

A training-free recipe for continuous, slider-style control of text-conditioned image/video diffusion models that operates entirely in the text-encoder embedding space. For a target concept (smile, photorealism, age, …), an LLM auto-generates a small set of “debiased” contrastive prompt pairs; pooled embeddings of the concept-bearing tokens are averaged on each side, and the normalized difference of the two centroids becomes a steering vector that is added to the prompt representation at inference. An elastic-range search procedure auto-calibrates the usable magnitude interval, avoiding under- and over-steering. Because the intervention lives in the text encoder, the same vector transfers across any backbone sharing that encoder — demonstrated on Qwen-Image, FLUX/FLUX2, and Wan2.1 / Wan VACE for both image and video.

Key claims

A single normalized difference-of-means direction in the text encoder’s output space is sufficient for continuous attribute control across diffusion image and video generators; no LoRA, no auxiliary module, no model finetuning [project page §“Method”, Step 2].
“Debiasing” the contrastive prompt pair by holding the rest of the prompt fixed and varying only the concept-bearing tokens (e.g. “frowning”↔“smiling”), then selecting only those token spans before pooling, isolates the target axis from incidental scene attributes [project page §“Method”, Step 1].
The construction is text-encoder-agnostic in practice: identical (re-derived) steering vectors transfer across Qwen-Image-Edit, FLUX2, Wan2.1 and Wan VACE — direct evidence that the relevant continuous-control axis lives in the text-conditioning representation rather than the diffusion backbone [project page §“Image” / §“Video”].
An elastic-range search procedure picks the usable magnitude interval per concept automatically (initialize → gap evaluate → midpoint expand → left/right move → similarity-filter), removing the steering-scale hyperparameter that prior slider methods leave to the user [project page §“Method”].
Frame-level edits applied on the first frame of a video pipeline can be propagated consistently through the rest of a generation via VACE — i.e. the text-embedding steering composes with image-to-video propagation rather than being limited to per-frame steering [project page §“Video”].
Method fails when the underlying generative model cannot produce one end of the concept spectrum (e.g. six-fingered hands) — the recipe assumes the target attribute exists somewhere in the model’s distribution [arxiv §“Limitations”].

Method

The pipeline has three stages, all at inference time. (1) Debiased contrastive prompt construction. An LLM is prompted to generate a small set of positive/negative prompt pairs for the target concept; pairs are matched on everything except the concept-bearing token spans. Only those concept-bearing token spans are kept for embedding extraction, so the resulting direction does not pick up background nuisance variation. (2) Difference-of-means direction. For each pair, the selected token embeddings are pooled (mean over the kept tokens) within the text encoder’s output to give $e_i^+$ and $e_i^-$ . The centroids mean(e⁺) and mean(e⁻) are taken across all pairs, their difference s is computed, and d_s = s / ‖s‖₂ is the final unit steering direction. (3) Steering at inference. For a runtime prompt, the prompt embedding is offset by α · d_s, and α is chosen from the elastic-range search interval. Because everything lives in the text-encoder output, the same d_s plugs into any diffusion model that consumes the same encoder — including video models via shared backbones like Wan VACE.

Results

The project page shows continuous steering across smile/frown, photorealism, age, lighting, style, and pose-like axes, demonstrated on Qwen-Image-Edit, FLUX2 (image-edit and image-to-image), Wan2.1, and Wan VACE. No quantitative metrics are reported on the project page beyond the qualitative slider walkthroughs and an animated visualization of the elastic-range search loop. The headline experimental claim is generalization across backbones — the same direction working unmodified on multiple downstream generators that share a text encoder — rather than a benchmark number against a fixed baseline.

Why it’s interesting

The construction is the diffusion-text-encoder analogue of the activation-steering / “Golden-Gate-Claude” line in LLM mech-interp: a normalized difference of two centroids in a learned representation space, added to the runtime representation, controls a behavioral axis continuously without any training. That matters for the Luma stack because text-conditioning is shared across Luma’s image and video generators, so a single steering vector can in principle deploy to multiple downstream models — the cross-backbone transfer demonstration is the actually-interesting result, not the slider construction itself. The method is a deliberate counterpoint to the LoRA-based slider literature (Spanning the Visual Analogy Space with a Weight Basis of LoRAs, Flux-Slider) and to the trained-module SAEdit line: it argues that for many continuous axes, training the LoRA was unnecessary in the first place. It also generalizes the CFG-variants story tracked in Classifier-Free Guidance variants: there, the “guidance branch” is rethought (training-time CFG in SoFlow, reference-vs-zero in SparkVSR); here, the prompt embedding itself is rethought as a vector that admits direct additive control, with the steering scale playing the role of guidance strength.