OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning

TL;DR

OmniWeaving is Tencent Hunyuan’s open-source attempt to close the gap to proprietary “omni-capable” video systems like Seedance-2.0. It is an MLLM + MMDiT + VAE stack built on HunyuanVideo-1.5 where the MLLM is elevated from a passive text encoder into an active reasoner — activating “thinking mode” to emit an enhanced prompt whose hidden states condition the MMDiT, plus DeepStacked multi-layer hidden states injected into the first three MMDiT conditioning layers. Trained in three stages (modality alignment → multi-task free-form pretraining → reasoning-augmented fine-tuning) on a large synthetic+real corpus spanning foundational, compositional, and reasoning-augmented video tasks. Ships with IntelligentVBench, a Gemini-2.5-Pro-judged benchmark explicitly targeting reasoning and composition in unified video generation.

Key claims

OmniWeaving unifies T2V, first-frame-to-V, key-frames-to-V, V2V editing, reference-to-V, compositional multi-image-to-V, and text-image-video-to-V in a single architecture rather than a router over task-specific modules [§1, §4.1].
“Activating Thinking Mode of the MLLM” — the MLLM autonomously generates an intermediate reasoning trace that produces a semantically precise enhanced prompt; its hidden states are forwarded alongside the original MLLM features to condition the MMDiT [§4.1].
Hidden States DeepStacking extracts hidden states from a broader range of intermediate MLLM layers and adds them (via an MLP connector) to the corresponding hidden states in the first three layers of the MMDiT conditioning branch, injecting multi-granular semantic guidance [§4.1].
Three-stage training: Stage 1 freezes the MLLM and finetunes MMDiT + connector on T2V/I2V to align modalities; Stage 2 adds the full multi-task free-form pretraining (excluding reasoning-augmented tasks) with MLLM still frozen, split into a sub-stage without video inputs and a sub-stage with them; Stage 3 unfreezes the MLLM and co-trains the reasoning-augmented tasks with a next-token-prediction loss on reasoning traces alongside the diffusion loss [§4.2].
Training data is constructed via a dual-pipeline strategy (output-first: curate real videos, synthesize task-specific inputs; input-first: curate text/key-frames, synthesize target videos), with Qwen3-30B / Qwen3-VL-235B / Gemini-2.5-Pro / SAM3 / FLUX2 / Veo3 / HunyuanVideo-1.5 used as tools and Qwen3-VL acting as a quality-filter evaluator [§3.3].
For local object addition V2V the paper inverts the local-removal pipeline (post-removal video → original video, “removal” instruction → “addition”), producing physically-consistent training samples that direct addition pipelines fail to generate [§3.3].
IntelligentVBench is presented as the first benchmark explicitly evaluating multimodal composition and abstract reasoning in unified video generation, using a VLM-as-a-judge paradigm (Gemini-2.5-Pro) over four task categories with three-dimension 1–5 scoring and both mean and minimum aggregation [§2, §1; project README].
The paper claims SoTA among open-source unified video models on IntelligentVBench [Abstract, §1] — proprietary Seedance-2.0 is positioned as the gap-defining reference, not as a comparison baseline.

Method

OmniWeaving builds on HunyuanVideo-1.5 (8.3B-param video DiT with selective-and-sliding-tile attention) as the MMDiT backbone, paired with an MLLM (the paper uses Qwen3-VL for both encoding and evaluation) and a video VAE. The MLLM has two roles: as a semantic parser it projects free-form interleaved text+image+video inputs into a high-level semantic space (last-layer hidden states → MLP connector → MMDiT conditioning branch), and as an active reasoner it autonomously emits an enhanced prompt under “thinking mode” whose hidden states are forwarded alongside the original features. DeepStacking — inspired by Qwen3-VL — extracts hidden states from multiple intermediate MLLM layers and injects them additively into the first three MMDiT conditioning layers via a learned MLP connector. The VAE acts as a low-level visual tokenizer for the generative branch.

Training is staged. Stage 1 (modality alignment) keeps the MLLM frozen and finetunes MMDiT + connector on T2I and T2V to match the backbone’s text-only generation quality with MLLM features. Stage 2 (multi-task free-form pretraining) scales to all compositional/foundational tasks, MLLM still frozen, with an explicit sub-stage split: video-input tasks are introduced only after the image-text interleaved tasks have stabilized. Stage 3 (reasoning-augmented fine-tuning) unfreezes the MLLM and co-trains reasoning-augmented tasks with a next-token-prediction loss on the reasoning traces alongside the diffusion loss, yielding a “comprehend-then-generate” model that emits an explicit thinking trace before synthesis on ambiguous inputs.

Training data spans three competencies — Foundational (T2V/T2I/V2V editing/key-frames-to-V), Multimodal Composition (interleaved text-and-multi-image-to-V; text-image-video-to-V), and Reasoning-Augmented (T2V from ambiguous queries; intent-driven I2V; event-deductive multi-image-to-V). The output-first pipeline starts from real-world videos and tools an ensemble of MLLMs to extract inputs; the input-first pipeline starts from textual prompts/key-frames and uses Veo3 / HunyuanVideo-1.5 to synthesize targets. Critical design choice for local object addition: the V2V removal corpus is inverted (swap source/target, rewrite the instruction) to generate physically-grounded addition samples.

Results

IntelligentVBench is the paper’s headline evaluation surface — a Gemini-2.5-Pro-judged 1–5 / 3-dimension scoring suite over four task categories (Editing local_change / local_add / back_change; compositional and reasoning tasks). The paper claims SoTA among open-source unified video models [Abstract, §1].
Stage-1 ablation: after modality alignment alone, OmniWeaving matches the HunyuanVideo-1.5 backbone’s text-to-video quality using MLLM features in place of the native text encoder [§4.2].
Quantitative table numbers, ablations on DeepStacking depth, and per-task scores against VACE / UniVideo / VINO / OmniVideo / OmniVideo2 are in the truncated tail of the paper (§5–§6) and not surfaced in this filing — open for follow-up read.

Why it’s interesting

OmniWeaving is the third datapoint in the wiki’s unified-video-generation cluster after VACE/UniVideo/VINO (named here as the field’s prior generation): a unified model that pushes beyond “T2V + V2V in one checkpoint” toward genuine free-form input composition plus an explicit reasoning trace before generation. Three reasons to file it:

It is the first filed UMM where the generation modality is video and where the “thinking mode” pattern (CoT before synthesis) is operationalized in a video DiT rather than in an image UMM — directly extending the Unified Multimodal Models taxonomy (E2E / Decoupled / Agentic) into the video axis with a “comprehend-then-generate” recipe distinct from DuoGen: Towards General Purpose Interleaved Multimodal Generation (which freezes the MLLM end-to-end) and from NEO-unify: Building Native Multimodal Unified Models End to End (encoder-free MoT).
The IntelligentVBench evaluation choice (Gemini-2.5-Pro as the sole judge) directly aligns with IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation‘s finding that Gemini-2.5-Pro is the only judge clearing r=0.8 against humans for image-side verification — but extends the LMM-as-judge pattern into video reasoning, where A Very Big Video Reasoning Suite and RISE-Video: Can Video Generators Decode Implicit World Rules? disagree on whether VLM-judges should be the default at all. OmniWeaving lands squarely on the RISE-Video side of that debate.
The training-data construction reads as a maximalist instance of Synthetic Training Data — Qwen3-VL-235B simultaneously generates labels, filters quality, and verifies identity-alignment between extracted entities and target videos; entire task subsets (local object addition) are constructed by inverting an existing synthetic pipeline. The closest comparison is the data engineering in Summer-22B: A Systematic Approach to Dataset Engineering and Training at Scale for Video Foundation Model and HunyuanVideo-1.5’s own data curation; OmniWeaving extends that to interleaved multimodal inputs and reasoning traces.

The gap to Seedance-2.0 is asserted, not measured — the paper offers no direct comparison to closed proprietary systems and IntelligentVBench is not yet a community standard, so the SoTA claim is constrained to open-source unified models on a self-introduced benchmark.