Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction

TL;DR

A dual-tower T2SV (text-to-sounding-video) recipe that attacks two pieces of the open dual-stream-DiT design space directly: (1) the conditioning problem — a single shared caption injects modal interference into pretrained T2V and T2A backbones — and (2) the interaction problem — what cross-modal fusion primitive to use between the two towers. Fix for (1) is HVGC, a three-stage visually-grounded captioning pipeline that produces disentangled, modality-pure video and audio captions. Fix for (2) is BridgeDiT, four bidirectional Dual Cross-Attention blocks bridging frozen Wan-2.1 (1.3B) and Stable Audio Open (1.0). Beats Wan+MMAudio cascade and JavisDiT/SSVG/MTV/CoDi joint baselines on FVD, FAD, CLAP, and AV-Align across AVSync15, VGGSound-SS, and Landscape.

Key claims

• Sharing a single caption between video and audio towers in a dual-tower T2SV model causes modal interference: text relevant to one modality is out-of-distribution noise to the other (the example “a red car emits a sharp honk” — “sharp honk” leaks into the video tower, “red” into the audio tower) [§1, §3.2].
• Direct audio-LLM captioning (Qwen2-Audio) of raw audio yields a disentangled audio caption but hallucinates because raw audio is information-sparse — e.g. “rhythmic drumming” misread as “high heels clicking on pavement” [§3.2].
• HVGC fixes this with a three-stage visually-grounded pipeline: (i) Qwen2.5-VL-72B produces a detailed visual description of the clip; (ii) an LLM extracts auditory-event tags from that description (chain-of-thought filter); (iii) Qwen2.5-72B writes the final audio caption conditioned on both, restricted to non-visual auditory language [§3.2].
• BridgeDiT consists of two frozen pretrained backbones (Wan 2.1 1.3B video + Stable Audio Open 1.0) bridged by 4 BridgeDiT Blocks uniformly inserted between corresponding layers of the two towers [§3.3, §4.1].
• The Dual Cross-Attention (DCA) block runs two symmetric streams per block: Audio-to-Video where video is the query and audio supplies K/V, and Video-to-Audio where audio is the query and video supplies K/V, each with its own residual and MLP path; both streams are AdaLN-conditioned on the per-modality diffusion timesteps [§3.3, Fig. 3b, Eqs. 7–8].
• On AVSync15, BridgeDiT beats every baseline on FVD (765.7), KVD (21.3), FAD (5.34), KL (1.30), CLAP (35.95), and AV-Align (0.275); it is also competitive on CLIPSIM and IB-VA (lost only to JointDiT and SeeingHearing respectively, with explanations: JointDiT uses Qwen-Image as a T2I prior, SeeingHearing uses ImageBind-Score as classifier guidance) [Table 1].
• On VGGSound-SS, BridgeDiT achieves FVD 615.3, FAD 6.01, AV-Align 0.362; on Landscape, FVD 628.1, FAD 4.78, AV-Align 0.258 — best on most metrics on both [Table 2].
• The Wan+SDA ablation (same architecture, interaction modules removed) confirms that the BridgeDiT Blocks themselves — not the pretrained backbones — drive the synchronization gains: removing them collapses AV-Align from 0.275 to 0.205 and FAD from 5.34 to 11.90 [Table 1].
• Caption ablation: HVGC with training beats Audio-LLM-disentangled, shared-video-only, and shared-Omini-caption configurations on every audio/sync metric; zero-shot (no interaction-module training) HVGC also beats every shared-caption zero-shot configuration on FAD and AV-Align [Table 3].
• Fusion-mechanism ablation: Dual Cross-Attention beats Full Attention (JointDiT-style), Additive Fusion (SSVG-style), V2A-only Cross-Attention (Liu et al. 2024a / MMAudio-style ControlNet), and A2V-only Cross-Attention (MTV-style) on both AV-Align and VA-IB Score throughout training; FullAttn is second [§4.3.2, Fig. 4].

Method

The architecture has two parts. First is HVGC, a captioning pipeline run once over each training video to produce disentangled (c_V, c_A) caption pairs: Qwen2.5-VL-72B with an in-context prompt produces a detailed visual description c_V, an LLM extracts auditory-event tags as a chain-of-thought filter against hallucination, and Qwen2.5-72B writes c_A conditioned on c_V + tags in non-visual auditory language. Second is BridgeDiT itself. The video tower is Wan 2.1 (1.3B, UMT5-XXL text encoder, 81 frames × 15 fps × 480p, flow-matching loss) and the audio tower is Stable Audio Open 1.0 (T5-base, 44.1 kHz, v-prediction diffusion loss); both are kept largely frozen. Four BridgeDiT Blocks are uniformly inserted between the corresponding layers of the two towers. Each block is a symmetric Dual Cross-Attention: in the A→V stream, LN(video) projects to Q and LN(audio) projects to K/V, attention runs, output residually updates video, then LN+MLP+residual; the V→A stream is the mirror. AdaLN conditions each stream on the per-modality diffusion timestep (the two towers can sit at independent t_V, t_A because the video tower uses flow matching and the audio tower uses diffusion). The training loss is the sum of the v-prediction audio loss and the flow-matching video velocity loss [Eqs. 4–8].

Results

On AVSync15 (1350 train / 150 test), BridgeDiT posts FVD 765.74 / KVD 21.33 / FAD 5.34 / KL 1.30 / CLIPSIM 28.52 / CLAP 35.95 / VA-IB 34.59 / AV-Align 0.275, beating the Wan+MMAudio cascade (FAD 7.98, CLAP 34.64, AV-Align 0.243), JointDiT (FAD 6.51, AV-Align 0.156), JavisDiT (FAD 13.48, AV-Align 0.158), SSVG (FAD 14.35, AV-Align 0.126), MTV (FAD 16.46, AV-Align 0.149), and CoDi (FAD 16.56, AV-Align 0.081) [Table 1]. On VGGSound-SS, FVD 615.28 / FAD 6.01 / AV-Align 0.362 (vs MMAudio cascade AV-Align 0.333, JavisDiT 0.179); on Landscape, FVD 628.07 / FAD 4.78 / AV-Align 0.258 (vs MMAudio cascade 0.218, JavisDiT 0.185) [Table 2]. Ablations: HVGC + training yields AV-Align 0.275 vs 0.245 for shared-video-caption + training and 0.242 for AudioLLM-disentangled + training [Table 3]; on the fusion side, DCA dominates FullAttn, V2A-CA, A2V-CA, and Add-Fusion throughout training on both AV-Align and VA-IB [§4.3.2, Fig. 4].

Why it’s interesting

Sits directly inside the Joint audio-video generation cluster but at a different scale point: 1.3B video + ~1B audio with backbones frozen and only 4 bridge blocks trained, against the 14B+5B LTX-2 (LTX-2: Efficient Joint Audio-Visual Foundation Model) and 32B/18B MOVA (MOVA: Towards Scalable and Synchronized Video-Audio Generation) settings the wiki has been tracking. It adds two specific design-space data points that the existing cluster has not had: (a) an explicit, controlled fusion-mechanism shootout (DCA vs FullAttn vs V2A-only vs A2V-only vs Additive), where the wiki’s prior data points all picked one bidirectional variant without head-to-head comparison; (b) a captioning-pipeline contribution that turns “shared caption vs not” into an architectural variable — relevant because LTX-2 / MOVA / SkyReels-V4 all use a shared MLLM text encoder feeding both branches and have not characterized the modal-interference cost of that choice. Complements OmniForcing: Unleashing Real-time Joint Audio-Visual Generation (which keeps a single text condition but distills for streaming) by showing that even at offline scale the conditioning axis matters independently of fusion design.

See also

• Joint audio-video generation — sets the dual-tower-with-bidirectional-fusion baseline this paper picks DCA as the specific fusion choice within
• Dual-stream diffusion transformer — DCA is a per-block bidirectional cross-attention, the canonical primitive in this concept; this paper supplies the first explicit ablation against FullAttn / unidirectional alternatives
• LTX-2: Efficient Joint Audio-Visual Foundation Model — also asymmetric dense dual-tower at 14B+5B vs this paper’s 1.3B+1B frozen-backbone setup; LTX-2 uses temporal-only-RoPE cross-attention, this paper uses standard DCA
• MOVA: Towards Scalable and Synchronized Video-Audio Generation — MoE bridge between a 14B video and 1.3B audio backbone; same “frozen pretrained backbones connected by a trainable bridge” pattern at larger scale
• SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model — also splits “shared MLLM, both branches read it” — this paper argues that move is the wrong default and provides HVGC as the alternative
• The Design Space of Tri-Modal Masked Diffusion Models — the discrete-MDM alternative to dual-tower DiT; orthogonal recipe, also has to solve modality-pure conditioning (does so via separate per-modality tokenizers + boundary tokens rather than caption disentanglement)