Transfer between Modalities with MetaQueries

TL;DR

MetaQuery is a Meta + NYU recipe for unified multimodal models: keep the MLLM completely frozen, prepend a set of learnable query tokens to its input, and read out the resulting hidden states as conditioning for a separately-trained diffusion image decoder. The whole system trains with only the standard denoising objective on 25M public image-caption pairs, no joint AR+diffusion balancing. The MetaQuery-XL configuration (Qwen2.5-VL 7B + Sana-1.6B) matches Janus-Pro-7B on GenEval (0.80) while preserving SOTA understanding scores (MMB 83.5, MMMU 58.6) and beating Janus-Pro on MJHQ-30K FID. It is the canonical “frozen-MLLM + DiT via learnable queries” baseline that several follow-ups on this wiki (MetaCanvas, OmniTransfer, DuoGen) are explicitly compared against.

Key claims

A small set (64–512) of learnable query tokens fed into a frozen MLLM produces image-generation conditioning that matches or beats using the MLLM’s last-layer input-token embeddings, while remaining compatible with in-context and interleaved multimodal output [§3.2, Table 1].
A fully frozen MLLM with a learnable-query bridge matches fully-tuned MLLM image-generation performance: 7.43 vs 7.75 MJHQ-30K FID, 0.56 vs 0.58 GenEval, 75.35 vs 78.97 DPG-Bench [Table 2].
Adding diffusion-decoder finetuning on top of the frozen MLLM further improves both FID (6.06) and GenEval (0.61), matching end-to-end joint tuning of MLLM+DiT [Table 2].
The Enc-Proj connector design (transformer encoder at MLLM hidden width, then linear projection to DiT condition width) beats Proj-Enc (project first, then encode at DiT width) at 4× fewer parameters: 7.43 FID / 0.56 GenEval / 75.35 DPG vs 7.41 / 0.51 / 73.75 for the larger Proj-Enc [Table 3].
Token-count scaling is asymmetric: visual quality saturates around 64 query tokens but prompt alignment (GenEval, DPG-Bench) keeps improving past 256; the paper picks 256 as a quality/efficiency tradeoff for the final models [§3.3 “Number of tokens”].
MetaQuery-XL (Qwen2.5-VL 7B + Sana-1.6B, 256 queries) reaches MJHQ-30K FID 6.02 (SOTA among unified models), GenEval 0.80 with rewriting, DPG-Bench 82.05, while keeping MMB 83.5 and MMMU 58.6 from the underlying VLM [Table 4].
The frozen-MLLM bridge transfers world knowledge: SOTA on CommonsenseT2I and WISE, including correctly resolving “national flag of the country where Yellowstone National Park is located” → US flag [§5.5, Fig. 9].
Instruction tuning with 2.4M naturally-occurring image pairs mined from mmc4 (SigLIP-clustered captions, MLLM-generated transformation instructions) beats prior zero-shot subject-driven baselines on DreamBench: DINO 0.737 / CLIP-I 0.852 / CLIP-T 0.301 vs Kosmos-G’s 0.694 / 0.847 / 0.287 [Table 5].
The same frozen-MLLM model can be lightly finetuned (1000 steps) to perform image editing by transferring its image-reconstruction capability [§5.3].

Method

The architecture is three blocks. (1) An MLLM (LLaVA-OneVision 0.5B for Base, Qwen2.5-VL 3B/7B for Large/XL) takes the text prompt and N learnable query tokens appended after it, processed under the MLLM’s standard causal mask — no full attention enabled for the queries, the MLLM is not modified at all. (2) A 24-layer transformer connector with bi-directional attention, configured Enc-Proj: it operates at the MLLM hidden width (896 for LLaVA-OV) then projects once to the diffusion model’s condition width. (3) A standard text-to-image diffusion model (Sana-1.6B for most experiments, Stable Diffusion v1.5 for COCO FID) whose original text-condition input is replaced wholesale by the connector output. The MLLM is frozen throughout; the query tokens, connector, and (optionally) DiT are trained with the standard denoising objective.

Pretraining is 8 epochs on 25M public image-caption pairs at batch 4096, LR 1e-4 cosine to 1e-5 with 4k warmup. Instruction tuning uses 2.4M image pairs mined from mmc4’s core-fewer-faces subset: SigLIP clusters captions (up to 6 images per group, similarity ≥0.5), the image with minimum average within-cluster similarity becomes the target and the rest become sources, and Qwen2.5-VL 3B writes the source→target transformation instruction.

Results

Image generation (Table 4): MetaQuery-XL hits MJHQ-30K FID 6.02 (best filed unified-model result on the page at filing time), GenEval 0.80 with rewriting (ties Janus-Pro-7B), DPG-Bench 82.05. With SD-v1.5 as the head, COCO FID 8.69 sets a new SOTA among SD-v1.5-based unified models (vs MetaMorph 11.8, Emu 11.66).
Image understanding (Table 4): Preserves the base VLM’s scores — MetaQuery-XL on Qwen2.5-VL 7B gets MME-P 1685.2, MMB 83.5, SEED 76.9, MMMU 58.6, MM-Vet 66.6, beating Janus-Pro-7B’s 1567.1 / 79.2 / 72.1 / 41.0 / 50.0 across every understanding axis at matched generation quality.
Subject-driven generation (Table 5, DreamBench): MetaQuery-B-Instruct (LLaVA-OV 0.5B) reaches DINO 0.737 / CLIP-I 0.852 / CLIP-T 0.301 zero-shot, beating Kosmos-G (0.694 / 0.847 / 0.287) and approaching the real-image oracle on DINO (0.774).
Knowledge-augmented generation (§5.5): SOTA on CommonsenseT2I and WISE, both of which probe world-knowledge reasoning the diffusion-only baselines cannot do.
Token-count sweep (§3.3, Fig. 2): MJHQ-30K FID converges by 64 tokens; GenEval and DPG-Bench keep rising past 256 — more pronounced for long captions and image reconstruction.

Why it’s interesting

This is the canonical “frozen MLLM + learnable queries + DiT” recipe that several already-filed wiki papers explicitly compare against. Exploring MLLM-Diffusion Information Transfer with MetaCanvas (MetaCanvas) holds query-token count constant against MetaQuery and beats it with 2D-structured canvas tokens — making MetaQuery the 1D-baseline whose ablation point isolates spatial structure as the load-bearing element. OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer (OmniTransfer) directly cites MetaQuery’s “per-task learnable MetaQuery tokens” as the design pattern it imports for swapping Wan-2.1’s T5 encoder for Qwen-2.5-VL. DuoGen: Towards General Purpose Interleaved Multimodal Generation (DuoGen) and the Unified Multimodal Models taxonomy list MetaQuery as the AR+Diffusion exemplar of the Decoupled class.

Two implications for Luma. First, MetaQuery’s frozen-MLLM-equals-tuned-MLLM result (Table 2) is the empirical justification for treating powerful VLMs as drop-in conditioning encoders rather than co-finetuning targets — the same case LTX-2: Efficient Joint Audio-Visual Foundation Model (LTX-2) makes with Gemma3-12B. Second, the mmc4 image-pair curation pipeline is a concrete recipe for getting instruction-tuning data without expert-model-generated pairs, the same scalability bottleneck OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer flags for its reference-conditioned video work.