UniVerse-1: Unified Audio-Video Generation via Stitching of Experts

TL;DR

UniVerse-1 is an open-source, Veo-3-like joint text-to-audio-video model that avoids training from scratch by stitching two pretrained experts — Wan2.1 (1.3B text-to-video) and Ace-step (3.5B music) — into a unified dual-stream architecture via lightweight cross-modal MLP connectors inserted into every transformer block. Two further contributions: an online annotation pipeline that captions sampled clips dynamically during training (rather than caching captions for whole videos offline) to eliminate semantic drift between clip and label, and an independent-noise-sampling fix for a previously unnoticed PRNG-induced cross-modal noise correlation that degrades audio at inference when video resolution/duration changes. The paper also introduces Verse-Bench, the benchmark several later open joint-A+V models (MOVA, OmniForcing) evaluate on.

Key claims

Stitching of Experts (SoE) fuses Wan2.1 (video) and Ace-step (music) at the transformer-block level via two-layer linear adapters that project each stream’s hidden states into the other’s K/V space, then injects them through both a frame-by-frame cross-attention path and the existing text cross-attention (with cross-modal K/V concatenated to text K/V) [§3.4.1, Fig. 1, Fig. 2].
A layer-interpolation step bridges the depth mismatch between Wan2.1 and Ace-step: new blocks are inserted at uniform intervals into the shallower model and initialized by linearly interpolating the weights of bracketing layers, claimed to mitigate training instability when starting the joint run [§3.4.2].
An online annotation pipeline runs as a server process concurrent with training, dynamically captioning each randomly-sampled fixed-length clip via Whisper (ASR) + QWen2.5-Omni (joint video caption + ambient-audio caption + verified speech transcript), eliminating clip↔caption misalignment that offline whole-video captioning produces [§3.3].
The standard single-seed PRNG produces a deterministic cross-modal noise correlation E[ε_v ε_a^T] ≠ 0 when v-noise and a-noise are sampled sequentially from one stream; the model learns this as a shortcut, so changing inference resolution or duration silently degrades audio. Fix: independently seeded PRNG instances per modality [§3.5].
Pretrained-input adjustments: Wan2.1 is modified to accept a reference image by replacing the first frame of the noisy video latent with the clean reference latent; Ace-step is moved from 44.1 kHz to 25.6 kHz mel input to make the audio frame rate commensurate with the 25 fps video, and the speaker-encoder input is removed to enable open-set generation [§3.4].
A low-quality-data loss schedule prevents AudioSet/VGGSound’s low visual fidelity from corrupting video quality: the flow-matching loss on the video stream is only computed at high noise timesteps (t > 800/1000) for samples from these datasets, where the model is learning coarse motion/structure rather than high-frequency detail [§3.4.3].
Training: ~7400 h of audio-video data (5400 h general AV + 1100 h verified speech-centric + ~900 h VGGSound/AudioSet), AdamW lr=1e-4, FSDP across nodes, gradient accumulation = 4, semantic-similarity loss against MERT (music) + mHuBERT (speech) intermediate features at layer 16 of the fused block, λ = 0.1 [§3.4.3, §4.1].
Verse-Bench introduced as a new joint-T2AV evaluation benchmark with image-text prompt pairs covering diverse sound categories, plus a Verse-Ted subset for audio-to-video synthesis specifically [§1, §4.1].

Method

The architecture stitches two pretrained models at the transformer-block level rather than training a joint DiT from scratch. Each fused block contains a video sub-block (Wan2.1 self-attention + text cross-attention) and an audio sub-block (Ace-step linear-attention + text cross-attention), with bidirectional cross-modal coupling: the video-stream hidden states post-self-attention are injected into the audio-stream cross-attention as additional K/V (concatenated to the text-derived K/V), and the audio-stream hidden states post-linear-attention are reciprocally injected into the video stream’s self-attention and cross-attention via dedicated K/V projection layers. A shared LayerNorm before injection keeps feature scaling consistent. A frame-by-frame cross-attention between video queries and audio K/V — with a temporal alignment derived from the 25-fps-matched audio sampling rate — provides the per-frame A/V coupling. Layer interpolation handles the depth mismatch by inserting weight-interpolated bracketing-layer copies at uniform intervals in the shallower model. The model is trained with a Conditional Flow Matching loss on both streams plus the MERT/mHuBERT semantic-similarity loss on the audio stream’s intermediate features, with the conditional video-loss schedule for low-quality AV data.

Results

The paper compares UniVerse-1 against (a) text-to-video systems including Wan2.2 (14B), HunyuanVideo, CogVideoX-1.5, Kling 2.1, SeeDance 1.0; (b) text-to-audio Stable Audio Open and AudioLDM2; (c) TTS systems CosyVoice/CosyVoice2/VibeVoice; (d) audio-to-video FantasyTalking and Wan-S2V; (e) video-to-audio HunyuanVideo-Foley; and (f) joint A+V baseline SVG (MM-Diffusion and R-FLAV are class-conditional and excluded). Specific headline numbers are truncated from the fetch, but the framing is: a 1.3B + 3.5B stitched model finetuned on ~7400 h reaches Veo-3-like coordinated A/V on Verse-Bench despite no joint pretraining from scratch, the independent-noise-sampling ablation recovers audio quality lost when changing inference resolution, and the online annotation pipeline is reported to mitigate the static-misalignment degradation [§1, §4].

Why it’s interesting

UniVerse-1 is the earliest entry (Sept 2025) in the wiki’s open joint-A+V cluster (Joint audio-video generation) and chronologically precedes LTX-2: Efficient Joint Audio-Visual Foundation Model, MOVA: Towards Scalable and Synchronized Video-Audio Generation, and SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model — all three of which then converged on architecturally similar dual-stream-with-bidirectional-cross-attention recipes (Dual-stream diffusion transformer). Three things distinguish it from the later cluster: (1) Stitching of Experts is explicitly a fusion of pretrained unimodal experts recipe rather than a unified dual-stream pretrain — closer in spirit to a Mixture-of-Experts than to LTX-2/MOVA/SkyReels-V4 joint training (the paper itself draws the MoE analogy); (2) the independent-noise-sampling observation is a previously unnoticed PRNG-level bug that none of the three later papers discuss — open whether they hit and silently worked around the same issue, or whether their architectures sidestep it; (3) it introduced Verse-Bench, which MOVA later evaluates on (Verse-Bench Set 3 lip-sync, LSE-D 7.094 / LSE-C 7.452 — see the Joint audio-video generation page), and OmniForcing also references — i.e. this paper is the source of one of the field’s emerging evaluation conventions.