DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation

TL;DR

DreamID-Omni (Tsinghua + ByteDance Intelligent Creation) unifies three controllable human-centric audio-video tasks — reference-based generation (R2AV), reference-conditioned video editing (RV2AV), and audio-driven animation (RA2V) — in a single dual-stream DiT initialized from Ovi. The core architectural contributions are a Symmetric Conditional DiT that injects identity-reference features by sequence concatenation while injecting structural conditions (source video, driving audio) by element-wise addition, and a Dual-Level Disentanglement mechanism (Syn-RoPE + Structured Captions) that resolves identity-timbre binding and speaker confusion in multi-person scenes. Trained via a three-stage progressive curriculum (in-pair reconstruction → cross-pair disentanglement → omni-task fine-tune), it reports SOTA across video, audio, and audio-visual consistency on the authors’ new IDBench-Omni benchmark, including beating Wan2.6.

Key claims

R2AV, RV2AV, and RA2V can be unified under a single conditional distribution by treating the difference as “which optional structural condition is provided” (source video for editing, driving audio for animation, neither for pure reference generation) [§3.1, Table 1].
The Symmetric Conditional DiT injects the two kinds of conditioning asymmetrically into both streams: reference features (identity image, voice timbre) are concatenated to the noisy latents to be processed by attention, while structural conditions (source video, driving audio) are added element-wise as a structural canvas [§3.2.1, Eqs. 2–3].
A null structural input collapses the additive term to zero, so task switching between R2AV / RV2AV / RA2V is “free” — no architectural change, single parameter set [§3.2.1, Table 1].
Multi-person confusion has two distinct failure modes: signal-level identity-timbre mismatch (subject A speaks in subject B’s voice) and semantic-level attribute-content misattribution (subject A inherits B’s visual attributes and dialogue) — and they need to be addressed separately [§3.2.2].
Syn-RoPE binds identities at the signal level by assigning each reference identity its own non-overlapping temporal RoPE segment of width D, with the same segment used for both the visual and acoustic features of that identity — yielding inter-identity decoupling (distinct rotational subspaces) and intra-identity synchronization (shared positional segment across modalities) [§3.2.2].
The audio stream’s RoPE frequency is scaled by Tv/Ta so that target video and target audio share a unified physical-time index, building on Ovi’s recipe [§3.2.2].
Structured Captions split the prompt into video / audio / joint fields, all using a shared per-identity anchor token Ai (initialized from a fine-grained attribute description) so that visual attributes, motions and dialogue are explicitly bound to specific subjects [§3.2.2].
The three-stage Multi-Task Progressive Training curriculum is: (i) in-pair reconstruction (R2AV only, masked-loss to prevent reference-copying), (ii) cross-pair disentanglement (reference and target drawn from different clips, full-stream loss), (iii) omni-task fine-tune on R2AV/RV2AV/RA2V mixed at 4:3:3 [§3.3].
The progressive ordering is essential: training without omni-task fine-tuning (“MT w/o OFT”) collapses ViCLIP to 9.518 and AQ to 4.449 (vs Ours 14.573 / 6.313); training with In-pair Reconstruction only causes severe copy-paste with ViCLIP 11.931 [Table 6].
On R2AV with multi-person dialogue, removing Structured Captions tripled speaker confusion (Spk-Conf. 0.08 → 0.26) and tanked ViCLIP from 13.613 to 11.381; removing Syn-RoPE dropped timbre similarity from 0.402 to 0.211 and crashed lip-sync (Sync-C 6.074 → 4.192) [Table 6].
On the R2AV benchmark, DreamID-Omni reports the lowest WER (0.052) and Speaker Confusion (0.080) among all baselines, including Wan2.6 (0.534 / 0.380), Qwen-Image + LTX-2, Qwen-Image + Ovi, Phantom, VACE, and HunyuanCustom [Table 2].
For RV2AV (video editing) and RA2V (audio-driven animation), DreamID-Omni reports SOTA video metrics (AES, ViCLIP, ID-Sim.) against VACE, HunyuanCustom and Humo, while also generating audio that the baselines cannot [Tables 4, 5].

Method

Two parallel DiT towers — video stream and audio stream — exchange information via bidirectional cross-attention at every block (initialized from Ovi). The architectural novelty is in how conditions enter the network. Reference identity images and reference voice timbres are encoded by the respective VAEs and concatenated along the sequence dimension to the noisy target latents, so they live in attention space and the DiT can disentangle high-level identity/timbre priors from them. Source video (for editing) and driving audio (for animation) are passed through the same VAEs and added element-wise to the noisy latents, acting as a structural canvas that enforces spatial/temporal consistency. The two channels are independent, and supplying a null tensor for the additive channel reduces the model to pure reference-conditioned synthesis.

Within attention, Syn-RoPE assigns each entity its own slot on the temporal axis. The target video and target audio latents occupy positions [0, T). The k-th reference identity is then placed at [T + (k-1)·D, T + k·D), with the same segment used for both that identity’s image latents and its voice timbre latents. Because RoPE is periodic, distinct slots project into distinct rotational subspaces, which empirically suppresses cross-identity attention; because each identity’s image and voice share a slot, the model gets an implicit cross-modal binding signal without needing an explicit pairing loss. The audio RoPE is additionally rescaled by Tv/Ta to put video and audio on a common physical-time index (the same trick used by SkyReels-V4 and MOVA).

At the semantic level, Structured Captions force the prompt into a script-like format with three fields (video caption, audio caption, joint caption), where every reference to a person uses a designated anchor token Ai that has been initialized from that identity’s attribute description. This eliminates the residual ambiguity that text alone leaves about which subject is doing/saying what.

Training proceeds in three stages: (i) In-pair reconstruction — references and target come from the same clip; a masked reconstruction loss zeroes the loss inside the reference regions to prevent trivial copying. (ii) Cross-pair disentanglement — references and target come from different clips, no mask, forcing the model to synthesize from abstract identity/timbre rather than low-level correlations. (iii) Omni-task fine-tune — mix R2AV / RV2AV / RA2V samples 4:3:3, where RV2AV provides a masked source video as Vs and RA2V provides driving audio as Ad. Inference uses multi-condition CFG independently on video and audio streams (chained, so identity/timbre guidance operates on a text-aligned basis).

Results

R2AV (IDBench-Omni, 100 prompts): Speaker Confusion 0.080 vs Wan2.6 0.380, Qwen-Image+Ovi 0.380, Qwen-Image+LTX-2 0.340; WER 0.052 vs Wan2.6 0.534; ID-Sim (single/multi) 0.674/0.603 vs Phantom 0.657/0.572, VACE 0.664/0.395; T-Sim (single/multi) 0.493/0.402 vs Wan2.6 0.391/0.217 [Table 2].
RV2AV (editing, 50 examples): AES 0.584 / ViCLIP 14.832 / ID-Sim 0.635 vs VACE 0.560/14.353/0.565 and HunyuanCustom 0.538/14.576/0.590, and additionally generates audio (WER 0.065, T-Sim 0.513, Sync-C 6.241) which neither baseline supports [Table 4].
RA2V (audio-driven animation, 50 examples): ViCLIP 16.618 vs Humo 14.859, ID-Sim 0.623 vs Humo 0.609, Sync-C 6.325 vs Humo 6.114 [Table 5].
Ablation, multi-person R2AV: removing Structured Captions tripled speaker confusion 0.08 → 0.26 and dropped ViCLIP 13.613 → 11.381; removing Syn-RoPE collapsed timbre similarity 0.402 → 0.211 and Sync-C 6.074 → 4.192 [Table 6].
Ablation, training curriculum: In-pair-only gives ID-Sim 0.692 / T-Sim 0.504 but only ViCLIP 11.931 (copy-paste failure); cross-pair-only gets ViCLIP 14.044 but ID-Sim crashes to 0.543; skipping the omni-task stage gives ViCLIP 9.518 (worst). Full curriculum: ViCLIP 14.573, ID-Sim 0.674, AQ 6.313 [Table 6].
Initialized from Ovi; learning rate 1e-5; global batch 32; RoPE margin D = Tv + 1. Schedule: 10k steps in-pair → 20k cross-pair → 20k omni-task fine-tune [§4.1].
Code, demos and a project page are promised at guoxu1233.github.io/DreamID-Omni/ (code release announced in the abstract) [Abstract].

Why it’s interesting

This is the cleanest filed instance of unified controllable human-centric generation built on the dual-stream A+V recipe: it sits on top of Ovi (the same baseline LTX-2 and SkyReels-V4 are measured against in Joint audio-video generation) and shows the dual-stream DiT is flexible enough to absorb three task types (R2AV / RV2AV / RA2V) without architectural changes — the Symmetric Conditional DiT’s “concatenate references, add structural canvas” split is a concrete recipe other dual-stream-DiT builders could copy. Two of its tricks are especially relevant cross-references against the wiki’s existing dual-stream cluster: the audio-RoPE rescaling by Tv/Ta is the same alignment idea as MOVA’s Aligned RoPE and SkyReels-V4’s 10.4× audio rescaling (Dual-stream diffusion transformer), but Syn-RoPE’s per-identity slot mechanism extends it from “align two modalities on one time axis” to “bind N (identity, voice) pairs in attention space at once” — which none of the filed joint-A+V baselines tackle. Compare to Wan-Animate: Unified Character Animation and Replacement with Holistic Replication: Wan-Animate also unifies animation + replacement on a single backbone via region-flagged input tokens, but is video-only and uses skeleton + implicit face features rather than identity-image + voice-timbre; DreamID-Omni’s contribution is essentially “the same unification move, but with audio”. The IDBench-Omni speaker-confusion metric (LMM-judged via Gemini-2.5-Pro) is also worth tracking as a candidate primitive for future audio-visual eval pipelines.