DuoGen: Towards General Purpose Interleaved Multimodal Generation

TL;DR

DuoGen is an interleaved image-text generation framework from NVIDIA Cosmos Lab (CVPR 2026) that fuses a pretrained MLLM (Qwen2.5-VL 7B) with a DiT initialized from a video generation model (Cosmos Predict 2.5 2B), avoiding from-scratch unimodal pretraining and the joint-objective balancing that early-fusion unified models require. The MLLM autoregressively emits a <Begin-of-Vision> (BOV) token to hand off to the DiT, which conditions on prior images stacked as “video frames” plus the MLLM’s hidden states. Training is decoupled into two stages — instruction-tune the MLLM first on 298k curated interleaved conversations, then freeze the MLLM and align the DiT on large-scale interleaved context (5M video first/last-frame pairs + open image-edit datasets). Across CoMM, InterleavedBench, and a new Cooking-200 / How-to-500 benchmark, DuoGen outperforms prior open-source interleaved models by large margins and also achieves SOTA on text-to-image (GenEval 0.88) and image editing (ImgEdit 4.19) among unified models.

Key claims

• A pretrained video DiT can be repurposed as the image-generation head of an interleaved MLLM+DiT system: stacking prior conversation images along the temporal axis (treating them as “video frames”) and concatenating with the noisy target latent gives the DiT both visual consistency and a single, native conditioning interface [§4, §4.1].
• The MLLM autonomously triggers image generation via a learned <BOV> special token, supervised in the assistant turn under standard next-token-prediction loss; this is what lets the model “switch modalities” without an external policy [§4].
• A two-stage decoupled training strategy — instruction-tune the MLLM on 298k high-quality interleaved conversations first, then freeze the MLLM and fine-tune only the connector + DiT on context-alignment data — is required to preserve the MLLM’s pretrained post-training behaviors; introducing the (linguistically uninformative) alignment data in stage 1 would harm the MLLM [§4.2].
• A data engine converts 347k raw how-to / story webpages (Instructables, eHow, StoryBird, CoMM raw) into 268k high-quality interleaved conversations via a multi-step LLM/MLLM pipeline: image validity filtering, text rewriting to strip HTML/links, image-text reordering, near-duplicate image removal, then dialogue conversion [§3.1].
• 30k high-quality synthetic interleaved samples are added on top of web data to fix resolution/aesthetic variance: human annotators write 1,500 seed prompts across 151 subcategories in 8 domains, OpenAI O3 expands to 15,270 instructions, plus 15k dish prompts from MM-Food-100k [§3.1].
• 5M raw videos are converted into interleaved (first-frame, transition-caption, last-frame) sequences via scene detection + Qwen2.5-VL-32B captioning, providing the bulk of stage-2 context-alignment data [§3.2].
• CFG for the generation step keeps visual conditions fixed but removes the final text chunk from the MLLM hidden-state sequence when computing the negative velocity — an asymmetric CFG specific to the interleaved setup [§4].
• Packed-sequence training is extended to interleaved samples with heterogeneous image resolutions by treating all images in a sample as a flat “video frame” sequence with per-image height/width/index, and by extending position embeddings so temporal indices increment per image and spatial RoPE uses per-image resolution [§4.1].
• On CoMM, DuoGen reaches IRS 7.76 vs 2.71 for MiniGPT-5 (~2.8×) [Table 2 area, §5.1]; on InterleavedBench, it reaches text-quality 4.28 vs Emu2’s 1.26 (~3.4×) [Table 3 area, §5.1].
• On GenEval, DuoGen scores 0.88 overall — above Qwen-Image 0.87, FLUX.1-dev 0.82, Bagel 0.82, OmniGen2 0.80, and the commercial GPT-4o-Image 0.84 [Table area, §5].
• On ImgEdit, DuoGen scores 4.19 (vs Qwen-Image-Edit 4.27, Nano Banana 4.23, GPT-4o-Image 4.20, OmniGen2 3.44, Bagel 3.20); on GEdit-EN it scores G_O 7.35 (vs Qwen-Image-Edit 7.56, Nano Banana 6.93, OmniGen2 6.41) [Table area, §5].
• GPT-5 is used as the MLLM judge for the new Cooking-200 / How-to-500 benchmark because GPT-4o is shown to miss fine-grained visual-semantic inconsistencies (e.g. headless-shrimp errors when the input image shows shrimp with heads) [§5.1].

Method

DuoGen is an MLLM + DiT pair with a one-token handoff. The MLLM (Qwen2.5-VL 7B) generates text autoregressively. When it emits the special <BOV> token, the system switches to image-generation mode: the DiT (Cosmos Predict 2.5 2B, pretrained on video) receives (a) a visual condition — all prior images in the conversation, encoded by the VAE and stacked along the temporal axis as if they were preceding video frames, concatenated with the noisy target latent; and (b) a language/semantic condition — the MLLM hidden states for all multimodal tokens preceding <BOV>, projected through a lightweight connector and cross-attended at every DiT decoder layer. The generated image is appended to the conversation history and the MLLM continues. The video-pretrained DiT is the key architectural bet: it already knows how to maintain object/scene consistency across a stack of “frames,” which is exactly the constraint an interleaved sequence imposes on consecutive images.

Training has two decoupled stages. Stage 1 trains only the MLLM under next-token-prediction loss (including the <BOV> token in the loss) on 298k curated interleaved instruction-tuning samples — the model learns when to switch modalities and how to continue conditioned on a newly generated image. Stage 2 freezes the MLLM and trains only the connector + DiT (flow-matching loss on a randomly sampled target image per sequence) on the much larger context-alignment corpus: 5M video first-last-frame pairs annotated with Qwen2.5-VL-32B transition captions, plus open-source image generation/editing data (ShareGPT-4o-Image, NHR-Edit, OmniGen1/2, UniWorld-V1, Echo-4o), with the stage-1 instruction data mixed back in. The instruction-tuning corpus itself is built by two complementary pipelines: a webpage data engine that filters 347k how-to / story pages down to 268k clean dialogues via cascaded LLM/MLLM rewriting + image filtering + dialogization; and a 30k synthetic subset where O3 expands 1,500 human-written seed prompts into 15,270 high-quality, aesthetics-controlled queries.

Results

On CoMM (Sty/Enti/Tren/Comp/ImgQ/IRS), DuoGen reports 9.22 / 9.22 / 9.24 / 9.66 / 9.53 / 7.76, with the headline gain on Illustration Relevance Score (image-text alignment): 7.76 vs 2.71 for MiniGPT-5, 1.46 for SEED-LLaMA, 2.02 for Emu2. On InterleavedBench (T-Q / I-Q / I-Co / IT-Co / Helpfulness / Avg), DuoGen reaches 4.28 / 3.65 / 3.70 / 3.69 / 4.06 / 3.87 vs MiniGPT-5 1.82 avg, GILL 1.84, Emu2 1.68. On the new Cooking-200, Cooking-200-Text-Input, and How-to-500 subsets, DuoGen consistently outperforms SEED-LLaMA, MiniGPT-5, and Zebra-CoT (the only open-source interleaved baselines that produce coherent outputs), with the gap to commercial Nano Banana narrowing substantially on the image-completeness and image-coherence axes but remaining on the text-completeness axis (Nano Banana 4.24 vs DuoGen 3.61 on Cooking-200 T-Com).

The unified-model evaluations are arguably the more striking result. On GenEval (text-to-image), DuoGen reaches 0.88 — above Qwen-Image (0.87), GPT-4o-Image (0.84), Blip-3o (0.84), Uni-CoT (0.83), FLUX.1-dev (0.82), Bagel (0.82), Janus-Pro-7B (0.80), MetaQuery-XL (0.80), UniWorld-V1 (0.80), OmniGen2 (0.80) — despite the DiT being only 2B and not having undergone any T2I-specific pretraining beyond the video-generation initialization. On ImgEdit, DuoGen scores 4.19, just below Qwen-Image-Edit 4.27 and Nano Banana 4.23, and well above all open-source unified models (OmniGen 2.96, Bagel 3.20, UniWorld-V1 3.26, OmniGen2 3.44). On GEdit-EN, DuoGen reaches G_SC/G_PQ/G_O = 7.68/7.76/7.35, again competitive with commercial Nano Banana and behind only Qwen-Image-Edit.

Why it’s interesting

Two things stand out for Luma. First: DuoGen is direct evidence that a video generation model can serve as the image-generation head of a unified text-image system at no apparent quality cost — in fact at SOTA-level T2I performance against systems with bespoke image-pretrained backbones. The frame-stacking conditioning recipe (treat all prior images as video frames) is mechanically tiny but ideologically large: it collapses the “interleaved image generation” problem into “next-frame video generation conditioned on text.” Anyone training a video DiT already has the right base model for this. Second: the decoupled two-stage strategy explicitly preserves a strong pretrained MLLM by gating the alignment data to stage 2 only, which is the cleanest reported recipe so far for “add image generation to an MLLM without breaking the MLLM” — relevant to anyone considering bolting a generator onto an existing perception stack. The data pipeline (298k instruction-tuning conversations via cascaded LLM/MLLM rewriting + O3-expanded synthetic prompts + 5M video transitions) is also the most detailed published recipe for general-purpose interleaved instruction data, and it makes the explicit argument that web-scale interleaved pretraining corpora (MMC4, OBELICS) provide insufficient instruction-style supervision — a point the Synthetic Training Data thread has been circling.

See also

• Synthetic Training Data — the 298k webpage→dialogue rewriting pipeline + 30k O3-expanded synthetic samples + 5M video-transition captions are a textbook instance of the “verification/aggregation gate as load-bearing piece” thesis on this concept page; the explicit framing is that existing pretraining corpora are insufficient and the curated synthetic corpus is what unlocks instruction-following.
• VLM-as-Evaluator — DuoGen uses GPT-5 as the MLLM judge for its new benchmarks and explicitly documents GPT-4o’s failure modes (missing headless-shrimp-style fine-grained inconsistencies), pushing the VLM-as-judge frontier one model generation forward.
• Unified Multimodal Models — DuoGen is one of the cleaner MLLM+DiT decoupled UMM recipes; see also Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation for the orthogonal “generation improves understanding” post-training direction.
• World Foundation Models — DiT initialized from Cosmos Predict 2.5 (a world-model lineage) is here repurposed for general image generation; the cross-pollination between video/world models and unified image-text systems is the architectural story.
• LTX-2: Efficient Joint Audio-Visual Foundation Model — LTX-2 is the dual-stream-DiT analog in joint audio-video; DuoGen is its analog for interleaved image-text with a decoupled (sequential) two-stream architecture rather than a parallel one.