3MDiT: Unified Tri-Modal Diffusion Transformer for Text-Driven Synchronized Audio-Video Generation

TL;DR

3MDiT proposes a unified text+video+audio diffusion transformer for joint T2AV that can either be trained from scratch or bolted on top of a frozen T2V backbone (SD3 11B or Wan2.1-T2V-1.3B). The recipe has two pieces: (1) tri-modal omni-blocks that run joint attention over concatenated video+text+audio tokens — with modality-specific RoPE (3D for video, 1D for audio, none for text) — inserted before the audio/video heads, and (2) an optional dynamic text conditioning mechanism that lets the text representation evolve through depth via alternating SD3-style dual-stream blocks. The orthogonal plug-in design preserves the frozen T2V’s video quality while adding an isomorphic 1D audio branch and explicit cross-modal coupling.

Key claims

3MDiT can be trained either from scratch on audio-video data or as a plug-in adaptation of a pretrained T2V backbone with the video weights frozen; when omni-block gates are zero the video pathway is mathematically identical to the base model [§3.2 “Orthogonality and Plug-in Adaptation”].
The audio branch is isomorphic to the video branch: same DiT micro-architecture (self-attention + text interaction + AdaLN-modulated gated MLP), with 1D RoPE replacing 3D spatiotemporal RoPE [§3.2 “Isomorphic Audio Branch”].
Omni-blocks perform joint attention over the concatenated [video; text; audio] sequence with 3D RoPE on video tokens, 1D RoPE on audio tokens, and no RoPE on text, returning modality-specific updates scaled by learnable per-branch gates [§3.2 “Omni-Blocks”, Eqs. 9–13].
Dynamic text conditioning treats the text representation as a mutable state that alternating SD3-style dual-stream video and audio blocks both read from and write back to, so evidence discovered in one modality influences how the other interprets the prompt; for backbones with a read-only text stream (e.g., Wan) the path is disabled and only omni-blocks couple A↔V [§3.2 “Dynamic Text Conditioning”, Eqs. 5–8].
On Landscape, 3MDiT (Wan-Adapted) matches the frozen Wan T2V backbone’s FVD (283.1 vs 282.8) and CLIP score (0.230 vs 0.229) while adding audio (FAD 2.45) and AV-Align 0.518; SD3-Adapted reaches FAD 2.03 [Table 1].
The Wan-Adapted variant uses 8 audio blocks + 8 omni-blocks while keeping all 30 video blocks frozen; audio quality and AVH/Javis sync scores improve steadily through training while video quality stays stable [§4.2, Fig. 2].
Ablations on AVSync15 (SD3 backbone) show monotone improvement: SD3+A vs SD3+A+D vs SD3+A+O+D drops FVD 648→539→573 and FAD 8.29→5.79→5.97 while AV-Align climbs 0.462→0.521→0.500 [Table 2].
Dynamic text conditioning (from-scratch SD3 on Landscape) trades a small video-quality regression (FVD 288.6 → 383.1) for a meaningful FAD gain (2.45 → 1.86) and better synchronization (AV-Align 0.590 → 0.615, Javis 0.017 → 0.050), suggesting a synchrony↔stability trade-off [Table 3].
Capacity scales smoothly: within a reasonable range, more audio blocks improve audio quality (deeper audio DiT capacity) and more omni-blocks improve audio-video alignment without affecting visual fidelity [§4.3, Fig. 3].

Method

Video is patchified into spatiotemporal tokens; audio is encoded as 1D tokens via a pretrained DAC neural audio codec at 16 kHz; text is encoded with a transformer text encoder (T5-style for Wan, the SD3-native encoder for the SD3 variant). The model is trained with rectified flow matching: noise and data are linearly interpolated, and the model predicts the velocity field $u_t = x_1 - x_0$ minimized in MSE.

The architecture has three components that compose:

Isomorphic 1D audio branch. A DiT mirroring the video tower’s micro-architecture but with 1D RoPE and 1D layout, denoising audio latents at the same flow time $t$ as the video branch.
Omni-blocks. Inserted before the audio/video heads. Each takes the concatenation [video_tokens; text_tokens; audio_tokens], computes joint multi-head attention with modality-specific RoPE applied independently to the video and audio spans (3D and 1D respectively), projects out modality-specific updates, and writes them back through learnable per-branch gates.
Dynamic text conditioning. When the backbone exposes a writable text stream (SD3 dual-stream), the text state $z^{(\ell)}$ is maintained as a shared variable; alternating video and audio dual-stream DiT blocks consume [modality; text], run joint attention with RoPE only on the modality span, and write gated residuals back to both streams. When the backbone has a read-only text stream (Wan-style cross-attention from modality to text), this is disabled and cross-modal coupling falls entirely to the omni-blocks.

This makes the design a plug-in adaptation recipe: given any pretrained T2V backbone, attach an isomorphic 1D audio branch, insert N omni-blocks before the head, optionally enable dynamic text conditioning if the backbone supports it. Video weights stay frozen.

Results

Headline Landscape numbers (Table 1):

Wan-Adapted (8 audio + 8 omni-blocks, 30 frozen video blocks): FVD 283.1, FAD 2.45, CLIP 0.230, AV-Align 0.518. FVD essentially matches the audio-free Wan baseline (282.8); CLIP slightly improves over the baseline (0.229).
SD3-Adapted (frozen 11B SD3 video tower): FVD 424.1, FAD 2.03, CLIP 0.225, AV-Align 0.627. Best FAD in the family.
SD3-From-Scratch: FVD 388.4, FAD 3.67, AV-Align 0.626. Trained on Landscape only (900 train / 100 test clips).

For reference, JavisDiT [JavisBench] reports FVD 408.6 / FAD 5.55 / AV-Align 0.098 on a related setup, and BridgeDiT reports FVD 628.1 / FAD 4.78 / AV-Align 0.258 — though cross-paper comparison is acknowledged as unreliable (different data, different durations, different backbones). The paper is careful that 9.7 s clips are evaluated without truncation, vs JavisDiT’s 4 s.

AVSync15 ablation (Table 2, SD3 backbone):

SD3 base (T2V only): FVD 1157.5, no audio.
- audio branch: FVD 648.0, FAD 8.29, CLAP 0.245, AV-Align 0.462.
- dynamic text conditioning: FVD 538.9, FAD 5.79, CLAP 0.306, IB-AV 0.190, AV-Align 0.521.
- omni-blocks + dynamic: FVD 573.1, FAD 5.97, CLAP 0.312, Javis 0.103, AV-Align 0.500.

Dynamic text conditioning study on Landscape (Table 3): swapping static for dynamic conditioning trades FVD 288.6→383.1 for FAD 2.45→1.86, AV-Align 0.590→0.615, Javis 0.017→0.050 — i.e. better sync and audio at a small video-quality cost.

Why it’s interesting

3MDiT is the first paper in the wiki’s joint-A+V cluster to argue that the dual-stream + bidirectional cross-attention recipe is not the only way to bolt audio onto a strong T2V backbone — instead of training a from-scratch dual-stream model end-to-end (LTX-2, MOVA, SkyReels-V4 in Joint audio-video generation), it offers a plug-in recipe where omni-blocks attach orthogonally to a frozen T2V and the omni-block gates can be initialized at zero for exact backbone equivalence. This is closer in spirit to BridgeDiT’s “bridge two pretrained towers” framing than to LTX-2’s monolithic train-everything-jointly approach, but with a unified tri-modal joint-attention operator instead of bidirectional cross-attention bridges. It also extends Dual-stream diffusion transformer with a third stream-as-mutable-state position (text), parallel to MoS’s argument that the static-final-layer text read is a bottleneck (Mixture of States: Routing Token-Level Dynamics for Multimodal Generation). The trade-offs the paper surfaces — dynamic text conditioning helps sync at a small video-quality cost; omni-blocks scale capacity smoothly — feed directly into the cluster’s open question of whether more-than-two-modality joint generation needs a different architectural recipe than two-modality.