Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation

TL;DR

Ovi is an 11B unified text-to-audio-video (T2AV) diffusion transformer built from two architecturally identical 5B Wan2.2-derived DiT towers — one video, one audio — coupled by blockwise bidirectional cross-attention at every block, with audio RoPE frequencies scaled by 21/157 to map both modalities onto a shared physical-time index. The audio tower is pretrained from scratch on hundreds of thousands of hours of audio (speech + sound effects), then fused with the pretrained video tower by training only self-attention and cross-attention while freezing the FFNs. The model generates 5 s × 720×720 × 24 FPS clips with synchronized speech and sound effects from a single combined text prompt, using one frozen T5 encoder shared by both branches. In a blind pairwise human-preference study (n=50) on Verse-Bench, Ovi is substantially preferred over JavisDiT and UniVerse-1 on audio quality, video quality, and A/V sync.

Key claims

The two backbones are architecturally symmetric: 30 transformer blocks, model dim 3072, FFN dim 14336, 24 heads × 128 head-dim, with paired self-attention + text cross-attention + AV cross-attention + FFN per block [§4.1, Table 1].
Architectural symmetry between modalities eliminates the need for projection layers when bridging streams, preserves the attention structure established during unimodal pretraining, and the authors claim this improves training stability and efficiency [§4.1].
Audio latents have 157 tokens for 5 s and video latents have 31 frame-tokens for the same span; the audio branch’s RoPE frequencies are rescaled by 21/157 so cross-modal attention runs on a shared temporal index, sharply aligning the cross-modal RoPE affinity diagonal [§4.1, Fig. 2].
A single frozen T5 encoder consumes a combined prompt that interleaves visual description with audible-speech transcripts wrapped in <S>...<E> tags and an audio description in <AUDCAP>...<ENDAUDCAP> tags; the same T5 embedding feeds cross-attention in both branches [§4.1, §3.2].
Training is two-stage: (i) train the audio tower from scratch with a flow-matching objective on a 16 kHz MMAudio 1D-VAE latent over hundreds of thousands of hours of audio (12 s pretraining → 5.04 s finetuning with sound-effects mixed in); (ii) fuse with the pretrained Wan2.2-5B video tower by training only self-attention and cross-attention while freezing the FFNs (5.7B of 11B trainable) [§4.2.1, §4.2.2].
Paired noise sampling with a shared timestep t across modalities, plus the bidirectional cross-attention, learns A/V correspondences (lip-sync, action-sound alignment) without any explicit sync loss [§4.2.2].
Audio-video data is filtered with SyncNet — even small amounts of out-of-sync data are reported to impede lip-sync ability, so strict confidence/offset/dB thresholds are applied [§3.2].
The unified-T5 prompt is empirically better than separate CLAP+T5 encoders the authors initially tried: ablation shows FD_PaSST 7.13 → 5.02, IS 8.34 → 11.20, CLAP 0.190 → 0.224 with WER essentially flat at 0.033/0.035 [§5.5, Table 3].
Ovi-Aud (the audio-only tower) is competitive with task-specialized T2A and TTS models (FD 18.03, IS 11.20, CLAP 0.224, WER 0.035) despite being a unified T2A+TTS model rather than a specialist [§5.4, Table 2].
In a 50-participant blind pairwise study on Verse-Bench, Ovi wins by a substantial margin over JavisDiT and UniVerse-1 on audio quality, video quality, and A/V sync; the joint training causes only a slight degradation vs the Wan2.2 base on video-only quality [§5.4, Fig. 4].
Limitations explicitly noted: 5 s clips only, 16 kHz audio path through a fixed 1D-VAE limits bandwidth, symmetric 5B+5B dense fusion plus CFG forward makes sampling expensive — distillation à la DMD2 and stitched-chunk causal extensions are flagged as future work [§6].

Method

Ovi keeps each modality on its own DiT tower (initialized identically to Wan2.2-5B) and bridges them at every block with bidirectional cross-attention — video tokens attend to audio tokens and vice versa — preserving each branch’s pretrained self-attention statistics. The audio tower is pretrained from scratch on a flow-matching objective in the MMAudio 16 kHz 1D-VAE latent space, conditioned on T5 embeddings of the combined prompt; pretraining uses up to-12 s waveforms emphasizing speech diversity, then fine-tunes on 5.04 s clips with sound-effects mixed in. In the fusion stage, both towers run in parallel under a shared diffusion timestep and independent noise samples, with newly-initialized cross-modal attention layers and original self/text-attention modules unfrozen but all FFNs frozen. Aligned RoPE rescaling (audio frequencies × 21/157) puts the differing token-rate modalities on the same temporal axis. The single combined T5 prompt drives both branches via text cross-attention, so the same semantic context conditions both visual content and audio content; this is the recipe the authors found materially better than the original two-encoder (CLAP + T5) design. Inference uses a single UniPC ODE solver stepping both branches jointly with per-modality flow-matching velocity predictions.

Results

T2A on AudioCaps-style metrics: FD_PaSST 5.02, FD_PANNs 18.03, IS 11.20, CLAP 0.224 — competitive with MMAudio-L (FD_PANNs 15.04, IS 12.08, CLAP 0.348) and substantially better than AudioLDM-2-L, while also being a TTS model [Table 2].
TTS WER 0.035 — comparable to CosyVoice (0.034) and FireRedTTS (0.038), slightly worse than F5-TTS (0.018) and Fish Speech (0.008), but the only model in the comparison that is also a strong T2A generator [Table 2].
Joint A/V human preference (blind pairwise, n=50, Verse-Bench): Ovi wins by large margins over both JavisDiT and UniVerse-1 on every axis — audio quality, video quality, audio-video sync [Fig. 4].
Video quality vs Wan2.2 base: only a slight degradation, attributed to the narrower paired A/V corpus relative to Wan2.2’s pretraining set [§5.4].
Combined-T5 vs separated CLAP+T5 ablation: FD_PANNs 20.78 → 18.03, FD_PaSST 7.13 → 5.02, IS 8.34 → 11.20, CLAP 0.190 → 0.224 [Table 3].

Why it’s interesting

Ovi pre-dates the three dual-stream A/V models already on the wiki (LTX-2: Efficient Joint Audio-Visual Foundation Model, MOVA: Towards Scalable and Synchronized Video-Audio Generation, SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model) by ~3 months and is in fact the architectural template they converge on: symmetric twin DiT towers, bidirectional cross-attention at every block, scaled RoPE for temporal alignment, and a pretrained video tower with a from-scratch audio tower. SkyReels-V4 in particular is essentially the Ovi recipe scaled up (architecturally symmetric, pretrained video + from-scratch audio, audio RoPE rescaling — Ovi uses ×21/157, SkyReels-V4 uses ×218/21), confirming that Ovi’s blueprint generalizes. The single-T5 combined-prompt design is also notable as the simplest text-conditioning recipe in this cluster — both LTX-2: Efficient Joint Audio-Visual Foundation Model (frozen Gemma3-12B + per-stream connectors + learnable thinking tokens) and SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model (shared MLLM with an extra video text-cross-attention layer) chose more elaborate text-conditioning stacks; the FD/IS/CLAP gains in Ovi’s combined-vs-separated ablation argue that fewer encoders can be enough when the prompt syntax encodes the modality structure.