Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model (daVinci-MagiHuman)

TL;DR

daVinci-MagiHuman (SII-GAIR + Sand.ai) is a 15B-parameter, 40-layer single-stream transformer that jointly denoises text, video, and audio tokens through self-attention only — no cross-attention, no per-modality towers, no multi-stream branches. This is the first credible open joint-audio-video foundation model that explicitly rejects the dual-stream-with-bidirectional-cross-attention recipe that LTX-2 / MOVA / SkyReels-V4 had converged on. The architecture is a “sandwich”: 4 input layers + 32 fully-shared middle layers + 4 output layers, with text tokens, an optional reference image latent, and noisy video+audio latents flattened into one sequence. Reported wins: 80.0% pairwise vs Ovi 1.1, 60.9% vs LTX 2.3 (2000 human comparisons), 5 s × 256p in 2 s and 5 s × 1080p in 38 s on a single H100, full stack Apache-2.0.

Key claims

The architecture is a unified 15B-parameter, 40-layer transformer that processes text, video, and audio tokens via self-attention only — no cross-attention, no multi-stream branches [README §Highlights].
A “sandwich” depth design: the first 4 layers and last 4 layers use modality-specific projections/normalization; the middle 32 layers share parameters across all modalities [project page §Architecture].
Text tokens, an optional reference image latent, and noisy video and audio latents are concatenated into a single token sequence and jointly denoised — lip-sync alignment is learned during joint denoising rather than as a post-hoc fusion step [README §Architecture].
Generates a 5-second 256p clip in ~2 s and a 5-second 1080p clip in ~38 s on a single H100 [README §Highlights].
80.0% pairwise win-rate vs Ovi 1.1 and 60.9% vs LTX 2.3 over 2,000 human comparisons [README §Highlights].
Word-error rate on generated speech is 14.60% vs 40.45% for Ovi 1.1 — large lip-sync intelligibility margin attributed to joint denoising [reported in coverage; see project page].
Multilingual: Chinese (Mandarin & Cantonese), English, Japanese, Korean, German, French — natively supported by a single model [README §Highlights].
Latent-space super-resolution: base model generates at 256p, then a dedicated SR module refines in latent space (no extra VAE encode/decode round-trip) using audio as an auxiliary signal to preserve lip-sync when upscaling [project page §SR].
Turbo VAE Decoder is a lightweight re-trained decoder that substantially reduces decoding overhead [README §Optimizations].
MagiCompiler does full-graph operator fusion across transformer layers for ~1.2× end-to-end speedup on H100 [README §Optimizations].
DMD-2 distillation lets the model generate in 8 denoising steps with no classifier-free guidance, without sacrificing quality [README §Optimizations].
Released under Apache 2.0: base model, distilled model, super-resolution module, and inference code — full stack open [README §Highlights].
External dependencies used at inference: T5Gemma-9B-9B-UL2 (text encoder) and Stable Audio Open 1.0 (audio model) [project page §Setup].

Method

The model collapses the joint-T2AV problem onto a single transformer trunk. Text tokens (from a frozen T5Gemma-9B text encoder), an optional reference image latent, and noisy video + audio latents are concatenated into one sequence and passed through a 40-layer transformer. Only the first 4 and last 4 layers carry modality-specific parameters (input projections + RMSNorm; output projections); the middle 32 layers are fully shared. Attention is plain self-attention over the unified sequence — there are no cross-attention blocks bridging separate per-modality towers, and no separate modality fusion modules. Reported architectural details from coverage include learned scalar gates with sigmoid activation on each attention head for training stability at 15B scale, and a timestep-free denoising scheme where the model infers its denoising state directly from the noisy input rather than from an explicit timestep embedding.

Inference uses a two-stage pipeline. Stage 1 (base) denoises at low resolution (256p) under the joint sequence. Stage 2 (SR) refines the latent in 5 additional denoising steps directly in latent space; the audio latent participates as an auxiliary signal so lip-sync is preserved during upscaling, avoiding a full VAE encode→decode pass. The distilled (DMD-2) variant runs in 8 denoising steps without classifier-free guidance. MagiCompiler fuses operators across transformer layers for ~1.2× speedup on H100, and a Turbo VAE Decoder cuts the per-frame decoding overhead.

Results

5-second 256p video in ~2 s and 5-second 1080p video in ~38 s on a single H100; ~1 minute on consumer Nvidia GPUs per third-party reports.
80.0% pairwise win-rate vs Ovi 1.1 and 60.9% vs LTX 2.3 in 2,000 human comparisons; visual quality 4.80/5 vs Ovi 1.1 4.73 and LTX 2.3 4.76 in the authors’ rubric.
Word-error rate on generated speech: 14.60% (daVinci-MagiHuman) vs 40.45% (Ovi 1.1) — large lip-sync intelligibility margin.
DMD-2 distilled student achieves the headline 38 s @ 1080p in 8 denoising steps without CFG, with no reported quality regression vs the multi-step base model.
Apache-2.0 full stack: base 15B model, distilled model, SR module, inference code on GitHub (GAIR-NLP/daVinci-MagiHuman) and Hugging Face (GAIR/daVinci-MagiHuman).
Native multilingual coverage across 6 language families (zh-Mandarin, zh-Cantonese, en, ja, ko, de, fr) from a single model.

Why it’s interesting

This is the first wiki entry that explicitly rejects the dual-stream + bidirectional-cross-attention recipe that the open joint-A+V cluster — LTX-2: Efficient Joint Audio-Visual Foundation Model, MOVA: Towards Scalable and Synchronized Video-Audio Generation, SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model — had converged on, and posts SOTA pairwise human-eval against the strongest of those baselines (LTX 2.3, Ovi 1.1). Together with The Design Space of Tri-Modal Masked Diffusion Models (Apple’s discrete-token unified MDM, also single-trunk but discrete and text+image+audio rather than continuous video+audio), this is the second filed paper that argues continuous dual-stream isn’t structurally necessary — but unlike the MDM line, this one is continuous and includes video, hitting exactly the modality combination where dual-stream was supposed to be required. The dual-stream concept’s TL;DR (“the key architectural primitive is bidirectional cross-attention at every block”) is now empirically contested by a 15B open release that beats two of its three exemplars. Also a clean datapoint on the Diffusion Distillation thread: DMD-2 distillation works on a single-stream T2AV backbone without classifier-free guidance, in 8 NFE.