Exploring MLLM-Diffusion Information Transfer with MetaCanvas

TL;DR

MetaCanvas is a lightweight framework for using a multimodal LLM (Qwen2.5-VL) as a spatial planner — not just a global text encoder — for a downstream diffusion transformer. Learnable 2D/3D “canvas tokens” are appended to the MLLM input, processed with multimodal RoPE, and then fused with the DiT’s noisy latents through a small ControlNet-style connector with zero-init projections. Implemented on three backbones (SANA, FLUX.1-Kontext, Wan2.2-5B) across six tasks — T2I, T2V/I2V, image/video editing, in-context video — it consistently beats global-text-conditioning baselines and a MetaQuery-style 1D-query baseline, with the canvas alone (no text to the DiT) providing enough structural signal to generate coherent images. The recipe is a concrete answer to “how do you cash in an MLLM’s spatial reasoning inside a diffusion generator without retraining either model end-to-end.”

Key claims

Using a frozen MLLM only as a global text encoder under-uses it; appending learnable spatial canvas tokens to the MLLM input and fusing them with the noisy latents transfers structural/planning information that pure text conditioning cannot [§1, Fig. 1].
The canvas tokens carry layout/spatial-prior information on their own: SANA trained with only MetaCanvas tokens and no text input to the DiT still produces semantically coherent images, and PCA visualizations of the connector features look like rough spatial sketches [§4, Fig. 5].
Combining text (T5) as global guidance with MetaCanvas as a visual prior yields the fastest GenEval convergence and best final score; this beats both the SANA T5 baseline and a MetaQuery-style 256 1D-token alternative [Fig. 4 left].
Even with no text conditioning at all, adding 2D canvas tokens on top of the noisy latents beats the no-text MetaQuery baseline — evidence that the spatial 2D structure of the canvas (vs 1D queries) is what carries the prior [Fig. 4 right].
Connector design ablations: a vanilla Transformer block + a DiT block with AdaLN and zero-init projection, conditioned on the diffusion timestep, and not projecting canvas tokens into the low-D VAE space early, gives the best GenEval result [Table 1].
Adding MetaCanvas to FLUX.1-Kontext [Dev] for image editing improves ImgEdit-Bench and GEdit-Bench consistently across training, with only lightweight connector overhead [Tables 2, 3, Fig. 6].
For video editing, MetaCanvas on Wan2.2-5B matches recent SOTA (InsViE, Ditto, Lucy-Edit-Dev) on VBench/GPT-4o quality scores but wins editing-accuracy / semantic-faithfulness by a large margin and tops human-preference win rates [Table 5].
The “no-canvas” controlled variant — i.e. swapping the T5 text encoder for the MLLM-based condition encoder but without canvas tokens — already matches or beats text-encoder baselines, so a non-trivial fraction of the gain comes from the MLLM-as-encoder swap itself; the canvas adds further gains on top [Table 5].
For in-context video (reference-image-conditioned video generation), MetaCanvas matches Wan-VACE 1.3B/14B and is particularly strong on human-object-interaction tasks [Table 6].

Method

MetaCanvas inserts a learnable “canvas” between an MLLM (Qwen2.5-VL) and a DiT generator. Text is tokenized and embedded by the MLLM’s text embedder; images/videos are encoded by both the MLLM’s visual encoder and the DiT’s VAE. A set of learnable 2D (image) or 3D (video) canvas tokens is appended to the MLLM input and processed with the MLLM’s multimodal RoPE, so the MLLM sees them as another modality and can write into them during its forward pass.

The resulting MLLM text embeddings pass through a lightweight MLP connector and become the DiT’s text-conditioning input. The canvas embeddings go through a separate canvas connector: a vanilla Transformer block (aligns canvas tokens to the DiT latent space) followed by a ControlNet-style DiT block where the transformed canvas tokens and noisy latents are summed and processed jointly under AdaLN conditioned on the diffusion timestep. Both blocks end in zero-initialized linear projections so the canvas has no effect at the start of training. Linear-Attn and Mix-FFN (from SANA) keep the connector cheap.

For video, the Wan2.2-5B input layer is modified: reference/condition latents are concatenated with noisy latents along the channel dim, then patch-embedded, then summed with interpolated MetaCanvas keyframe canvas tokens (light-purple in their Fig. 3). Canvas keyframes are not applied to the reference frames themselves — only to the noisy-latent stream.

Results

GenEval (T2I): MetaCanvas + text outperforms SANA T5 baseline and a MetaQuery-equivalent (256 1D Qwen2.5-VL query tokens) on the GenEval convergence curves [Fig. 4 left]. The no-text MetaCanvas variant outperforms no-text MetaQuery [Fig. 4 right].
Image editing (FLUX.1-Kontext [Dev] + MetaCanvas): consistent gains over the vanilla model on ImgEdit-Bench [Table 2] and GEdit-Bench [Table 3]; training curves show steady improvement throughout [Fig. 6].
T2V/I2V (Wan2.2-5B + MetaCanvas): comparable to CogVideoX-5B, HunyuanVideo, Wan2.1-14B, baseline Wan2.2-5B, while additionally enabling strong video editing [Table 4].
Video editing: matches InsViE, Ditto, Lucy-Edit-Dev on VBench / GPT-4o quality scores; wins editing accuracy by a large margin; highest human-preference win rate on accuracy, spatio-temporal consistency, and overall preference [Table 5].
In-context video (reference-conditioned): competitive with Wan-VACE 1.3B and 14B, particularly strong on character + object multi-ID human-object-interaction tasks [Table 6].

Why it’s interesting

This sits at the intersection of two threads on the wiki. First, it’s another data point for frozen LLM as text encoder : like LTX-2’s Gemma3-12B + multi-layer feature extractor + thinking-tokens connector (LTX-2: Efficient Joint Audio-Visual Foundation Model), MetaCanvas freezes a powerful pretrained LM and reads it out through a learned lightweight connector — but goes a step further by letting spatial tokens flow back through the LLM, so the LLM is doing layout planning, not just sentence embedding. The “no-canvas-but-MLLM-encoder-swap already helps” datapoint (Table 5) is the cleanest evidence so far that MLLM features beat T5 features for video-editing conditioning even before you exploit the spatial channel.

Second, it’s a clean comparison against MetaQuery-style 1D learnable queries (Pan et al. 2025). Holding token count roughly constant, the 2D-canvas variant wins — suggesting the spatial structure of the query tokens (not just their existence as learnable LLM outputs) is what carries the planning signal. That’s a non-obvious architectural prior worth carrying into any internal MLLM-conditioned diffusion work.

Third, the ControlNet-style zero-init connector with AdaLN-on-timestep is the kind of recipe that drops cleanly onto an existing pretrained DiT without retraining the backbone — exactly the integration profile that makes it cheap to try internally on Luma’s video stack.