UniVideo: Unified Understanding, Generation, and Editing for Videos

TL;DR

UniVideo extends the image-domain unified-multimodal-model recipe (MLLM understanding stream + MMDiT generation stream à la Qwen-Image) to the video domain. A frozen Qwen2.5-VL-7B reads text/image/video instructions and emits last-layer hidden states into HunyuanVideo-T2V-13B via a trainable MLP connector; the MMDiT additionally receives the same visual inputs through its VAE, giving it both semantic and fine-grained pixel-level conditioning. Trained jointly across T2I/T2V/I2V/image editing/in-context video generation/in-context video editing under a single instruction paradigm, it matches or beats task-specific SOTA on each task and — critically — generalizes zero-shot to free-form video editing (material/weather changes, green-screening) it was never trained on. Accepted at ICLR 2026.

Key claims

Two-stream architecture: an MLLM (Qwen2.5-VL-7B, frozen) processes text + image + video instructions and emits last-layer hidden states; a connector aligns these to an MMDiT (HunyuanVideo-T2V-13B) which also sees the visual inputs separately via the VAE — preserving fine detail the MLLM’s semantic encoder discards [§2.1].
Both streams of the MMDiT receive visual signals, not just the MLLM — the ablation in Table 7 shows that feeding visual inputs only to the MLLM drops subject-consistency on in-context generation from 0.88 to 0.32 and on in-context-insert editing from 0.92 to 0.18 [§3.4, Table 7].
A single multimodal instruction paradigm (with ID-tagged visual inputs) replaces task-specific bias embeddings (UNIC) and context adapters (VACE); 3D positional embeddings with temporal-axis offset for condition vs noisy latents outperform Qwen2-VL’s MRoPE for distinguishing them [§2.2].
Three-stage training: (1) MLP-connector alignment only, MLLM+MMDiT frozen, on T2I/T2V + image reconstruction; (2) connector + MMDiT fine-tuning on high-quality T2I/T2V, MLLM still frozen; (3) multi-task training adds image editing, image-to-video, in-context video generation, in-context video editing — MLLM frozen throughout [§2.4, Table 2].
The MLLM is never unfrozen — UniVideo is explicitly described as an “assembled multimodal generative system” rather than a native end-to-end one (contrast with BAGEL / Mogao) [§1].
Beats task-specific SOTA on in-context video generation: human-eval Overall 0.95 (single-ID) and 0.85 (multi-ID, 2-4 refs) vs Kling 1.6’s 0.88 / 0.95 and VACE’s 0.42 / 0.48 [Table 4].
Beats baselines on in-context video editing in the harder mask-free setting (all baselines require explicit editing masks): insertion CLIP-I 0.693 / DINO-I 0.398 vs Kling 1.6’s 0.632 / 0.287; swap CLIP-I 0.728 / DINO-I 0.427 vs UNIC’s 0.725 / 0.429 [Table 5].
Joint multi-task training beats matched-architecture single-task baselines, especially for editing: average human-eval scores PF 0.80 vs 0.64, SC 0.78 vs 0.67, Overall 0.85 vs 0.79 across six in-context editing/generation subtasks [Table 6].
Zero-shot task composition (combine in-context editing with style transfer, simultaneously delete one identity and add another) emerges from unified training without ad hoc designs [§3.3.1, Fig. 5].
Zero-shot transfer of editing capability from large-scale image editing data to free-form video editing instructions (material changes, weather changes, green-screening characters) — UniVideo handles these despite no free-form video-editing data in its training mix [§3.3.1, Table 1].
Visual-prompt understanding: a user can draw a story plan or motion annotations on a canvas, and the frozen MLLM’s autoregressive reasoning translates these into dense prompt tokens that condition the MMDiT directly, turning visual prompting into in-context video generation [§2.3, §3.3.2, Fig. 6].
Retains the underlying MLLM’s understanding scores: 83.5 MMBench / 58.6 MMMU / 66.6 MM-Vet alongside 82.58 VBench T2V — competitive with HunyuanVideo’s 83.24 VBench while doubling as a comprehension model [§3.2.1, Table 3].
Compute footprint: 32 H100 GPUs total, three stages of 15K + 5K + 15K steps with EMA 0.9999 from Stage 2 onward [§2.4, Table 2].

Method

UniVideo is a connector between two large frozen-ish components. The MLLM (Qwen2.5-VL-7B, kept frozen throughout) reads ID-tagged multimodal instructions and emits last-layer hidden states. An MLP up-projects these to HunyuanVideo-T2V-13B’s input dimension and feeds them into the MMDiT’s understanding stream. In parallel, every visual input the MLLM sees is also encoded by HunyuanVideo’s VAE, padded to a uniform shape, concatenated along the temporal axis with the noisy target latent, and fed into the MMDiT’s generation stream — this is what gives the MMDiT the fine-grained pixel detail the MLLM’s semantic encoder lost. 3D positional embeddings preserve spatial indices across condition vs target frames while only the temporal index increments, letting the MMDiT distinguish references from the noise it’s denoising.

Tasks differ only in which inputs go to which stream (Table 1 in the paper enumerates eleven training tasks ranging from T2I to in-context image stylization). Training proceeds in three stages of progressively more tasks, with the MLLM frozen at every stage and only the connector + MMDiT trainable after Stage 1. Visual-prompt understanding is unlocked entirely by the frozen MLLM’s existing reasoning: hand-drawn annotations are interpreted in autoregressive token space, dense prompt tokens are emitted, and these flow through the standard connector path into the MMDiT — no separate visual-prompt module.

Results

In-context video generation (Table 4): Single-ID Overall 0.95 vs Kling 1.6 0.88 / VACE 0.42 / Pika 2.2 0.15; multi-ID (2-4 refs) Overall 0.85 vs Kling 1.6 0.95 / VACE 0.48 — UniVideo wins on Subject Consistency at both (0.88, 0.81) and ties or wins on Prompt Following.
In-context video editing, mask-free (Table 5): Wins insertion (CLIP-I 0.693), swap (CLIP-I 0.728), and stylization (CSD 0.228, ArtFID 37.877, CLIP 0.226); ties or slightly trails on deletion. Baselines all require masks; UniVideo does not.
Visual understanding (Table 3): 83.5 MMBench / 58.6 MMMU / 66.6 MM-Vet — matches LLaVA-NeXT-tier scores without sacrificing T2V quality (VBench 82.58 vs HunyuanVideo backbone 83.24).
Multi-task vs single-task ablation (Table 6): Multi-task gains +0.16 PF / +0.11 SC / +0.06 Overall averaged across six subtasks. Largest gain on swap editing (+0.38 PF).
Streaming-to-MMDiT ablation (Table 7): Routing visual inputs to the MLLM only and not to the MMDiT collapses subject-consistency to 0.11-0.32 from 0.58-0.92 — the VAE pathway carries identity, the MLLM pathway carries semantics.
Generalization (Fig. 5, §3.3.1): Zero-shot task composition (compose editing + style transfer, multiple simultaneous edits) and zero-shot free-form video editing (material/weather changes, green-screening) — the latter transferred entirely from image-editing training data.

Why it’s interesting

UniVideo is the cleanest filed demonstration that the Unified Multimodal Models recipe transfers from images to video without changing the architectural shape: frozen MLLM + connector + MMDiT, joint training over a multi-task instruction soup. It complements OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning, which independently arrived at a frozen-MLLM-with-MMDiT recipe for video — but OmniWeaving unfreezes the MLLM in its Stage 3 for reasoning-augmented training, while UniVideo never does, making it the cleaner reference point for “assembled” vs “native” unified video systems. Architecturally it joins Dual-stream diffusion transformer as another point where the two streams come from different model families (an MLLM and an MMDiT) rather than two same-family DiT backbones — contrast with LTX-2: Efficient Joint Audio-Visual Foundation Model / SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model / Waver: Wave Your Way to Lifelike Video Generation where dual-stream means two coordinated DiT backbones. Three details worth lifting for the team’s own work: (1) routing the same visual inputs through both the semantic (MLLM) and pixel (VAE) pathways is non-optional for identity preservation — Table 7’s collapse is dramatic; (2) zero-shot transfer of free-form video editing from image editing data alone is a meaningful sample-efficiency result and resonates with the “image editing data is leverage for video” theme also seen in InstructX: Towards Unified Visual Editing with MLLM Guidance and Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset (Ditto / Editto); (3) the “Kling’s video + Kontext paper” framing in the Slack post is apt — the paper explicitly cites FLUX.1 Kontext as one of its inspirations and the Kling Team is the second affiliation.