Joint audio-video generation

TL;DR

Models that jointly synthesize synchronized video and audio from a single prompt within one architecture, rather than chaining T2V→V2A or A2V→video pipelines. The emerging open-source consensus is a dual-stream diffusion transformer with bidirectional cross-attention at every block coupling a video tower and an audio tower; capacity allocation, joint vs separate text encoders, and cross-modal CFG knobs are the main axes of disagreement. Proprietary reference points are Veo 3, Sora 2, Kling 2.6/3.0 (Kling Video 2.6 — first Kling AI model with native audio (announcement)); the open-source frontier is now defined by LTX-2 (LTX-2: Efficient Joint Audio-Visual Foundation Model), MOVA (MOVA: Towards Scalable and Synchronized Video-Audio Generation), and SkyReels-V4 (SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model).

Key claims

• Decoupled T2V→V2A or A2V→V pipelines cannot model the joint distribution because acoustic environment depends on visual context and lip-sync depends on audio, so a unified model is needed for true T2AV (LTX-2: Efficient Joint Audio-Visual Foundation Model §1).
• Cascaded T2V→V2A pipelines accumulate error and drift out of sync, leaving a quality gap vs proprietary joint systems like Veo 3 and Sora 2 (MOVA: Towards Scalable and Synchronized Video-Audio Generation §1).
• The current state-of-the-art open architectural recipe is a dual-stream DiT with two modality-specific transformer towers exchanging information via bidirectional cross-attention at every block — used by LTX-2 (asymmetric dense 14B+5B; LTX-2: Efficient Joint Audio-Visual Foundation Model §3.1), MOVA (asymmetric MoE 32B/18B, with a 14B video + 1.3B audio tower bridge; MOVA: Towards Scalable and Synchronized Video-Audio Generation §3), and SkyReels-V4 (symmetric architecture but asymmetric initialization — pretrained video tower + from-scratch audio tower; SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model §3.1).
  • Alternative architectural family: The Design Space of Tri-Modal Masked Diffusion Models proposes a single-trunk, unified-vocabulary discrete masked diffusion recipe over text + image + audio — no dual-stream towers, no cross-attention bridge, no continuous flow-matching loss. Modalities are tokenized separately (SBER-MoVQGAN, Higgs Audio v2, Tiktoken) and interleaved in one sequence with modality-specific boundary and mask tokens. Currently scoped to text+image+audio rather than text+image+audio+video, but is the strongest argument that joint-multimodal generation does not need a dual-stream DiT (The Design Space of Tri-Modal Masked Diffusion Models §3, §3.1).
• Capacity allocation across modalities is not yet settled: LTX-2 is dense 14B+5B asymmetric (LTX-2: Efficient Joint Audio-Visual Foundation Model §3.1); MOVA is MoE 32B-total / 18B-active, with a 14B video and 1.3B audio backbone (MOVA: Towards Scalable and Synchronized Video-Audio Generation §3); SkyReels-V4 is symmetric in width and depth across both branches (SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model §3.1).
• Sampling-rate mismatch between modalities (video at tens of FPS in latent space vs audio at hundreds of frames/s) is handled by scaling per-modality temporal RoPE frequencies so the two streams attend on a common physical-time index — Aligned RoPE in MOVA (MOVA: Towards Scalable and Synchronized Video-Audio Generation project page) and a 218/21 ≈ 10.4× audio RoPE rescaling in SkyReels-V4 (SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model §3.1); LTX-2 instead retains 3D RoPE on video and 1D RoPE on audio but uses only the temporal RoPE component during cross-modal attention (LTX-2: Efficient Joint Audio-Visual Foundation Model §3.1).
• Multi-modal CFG with independent guidance scales per stream is becoming standard — LTX-2 uses per-stream (text, cross-modal) scales of (4.0, 0.5) for video and (5.0, 1.0) for audio (LTX-2: Efficient Joint Audio-Visual Foundation Model §4.1); MOVA exposes Dual CFG as a per-scene visual-quality vs lip-sync knob (MOVA: Towards Scalable and Synchronized Video-Audio Generation project page).
• Joint audio-visual training does not appear to cost video quality: LTX-2 ranks 3rd on I2V and 4th on T2V on Artificial Analysis (Nov 2025), beating Sora 2 Pro and Wan 2.2-14B despite also generating audio (LTX-2: Efficient Joint Audio-Visual Foundation Model §6.2); SkyReels-V4 reports #1 on the Artificial Analysis Arena (SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model §1).
• Joint-T2AV-capable systems are starting to support up to 20 s continuous synchronized output (LTX-2: Efficient Joint Audio-Visual Foundation Model §6.3) and 15 s at 1080p × 32 FPS multi-shot (SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model §1) — meaningfully beyond Ovi (10 s) and Wan 2.5 (10 s).
• Joint-A+V models can also subsume image-to-video, video extension, video editing, and reference-conditioned generation as different mask configurations of a single inpainting interface (SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model §3.2).
• A frozen MLLM as the shared text encoder lets joint models accept compositional multi-modal prompts that interleave text with image/video reference tokens in one input — used by SkyReels-V4 with a shared MLLM consumed by both branches (SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model §3.1, §3.3); LTX-2 uses a similar frozen-LLM approach with Gemma3-12B plus per-stream text connectors (LTX-2: Efficient Joint Audio-Visual Foundation Model §3.2).
• Open benchmarks are starting to consolidate around Verse-Bench and Arena-style human-preference Elo, with MOVA leading on Verse-Bench Set 3 lip-sync (LSE-D 7.094 / LSE-C 7.452) and on Arena (Elo 1113.8, >70% win rate vs Ovi) (MOVA: Towards Scalable and Synchronized Video-Audio Generation §6).
• Discrete-MDM tri-modal models prefer a multiplicative-form scaling law and need ~4× more tokens per parameter than AR LLMs ($D^* \propto N^{0.476}$, fitted on 262 isoFLOP runs at $R^2 = 99.3\%$). This sets a concrete data-budget floor for the alternative architectural family relative to AR-LLM precedents: a 3B unified-MDM wants ~250B tokens, vs ~60B for AR-Chinchilla (The Design Space of Tri-Modal Masked Diffusion Models §5.3, Eq. 5.2, Fig. 3).
• Single-GPU 1080p TI2AV is reachable purely via system-level optimization of a dual-stream T2AV base: NVFP4-quantized linears + SM100/SM103-tuned attention + end-to-end graph fusion drop LTX-2.3 to ~4.55 s for a 5 s × 1088×1920 × 24 FPS clip with audio on a single B200, no algorithmic step reduction (FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU).
• Real-time streaming joint T2AV is reachable via distillation of the bidirectional dual-stream teacher into a causal AR student — OmniForcing distills LTX-2 (14B video + 5B audio) into a causal streaming generator at ~25 FPS on a single GPU with ~0.7 s time-to-first-chunk, a ~35× speedup over the ~197 s offline teacher, while matching teacher fidelity on JavisBench (OmniForcing: Unleashing Real-time Joint Audio-Visual Generation Abstract, Community).
• Naive causal distillation of dual-stream T2AV fails for two specific reasons that don’t appear in video-only AR distillation: (a) extreme temporal asymmetry between video and audio token rates causes multi-modal synchronization drift, fixed by Asymmetric Block-Causal Alignment + zero-truncation Global Prefix; (b) extreme audio-token sparsity during the causal shift causes gradient explosion, fixed by an Audio Sink Token mechanism with Identity-RoPE constraint (OmniForcing: Unleashing Real-time Joint Audio-Visual Generation Abstract).
• Kuaishou’s Kling video line has shipped native joint audio at the 2.6 release with no technical disclosure — no parameter count, architecture, audio sample rate, or benchmark numbers are public, making 2.6 a fourth closed-source datapoint (alongside Veo 3, Sora 2, Seedance 2.0) whose recipe is opaque (Kling Video 2.6 — first Kling AI model with native audio (announcement)).

Recent contributions

• [2026-07-20] Happy Oyster — Real-Time World Model for Interactive Creation: Happy Oyster (Alibaba Token Hub) claims joint audio-video generation inside a real-time interactive-rollout WFM — a corner of the concept mostly populated by offline dual-stream DiTs (SkyReels-V4, Ovi, 3MDiT, HunyuanImage/Video variants). Product framing only, no architecture; but pins joint AV as a base-product-surface requirement rather than a post-hoc TTS/V2A pipeline, alongside Runway GWM-1’s audio channel and Cosmos 3’s omnimodal action+AV output.
• [2026-06-01] Cosmos 3: Omnimodal World Models for Physical AI: Cosmos 3 supports synchronous audio-video generation as one of seven generation modes inside a single MoT backbone (stereo 48 kHz audio VAE at 25 tokens/sec aligned via absolute temporal modulation), reaching audio score 7.31/7.34 vs Veo-3.1’s 7.45 while sharing parameters with all other modalities.
• [2026-05-24] Kling Video 2.6 — first Kling AI model with native audio (announcement): Kuaishou announces Kling Video 2.6 as the first Kling model with native audio — “See the Sound, Hear the Visual”, framed as a single-model joint T2AV rather than a T2V→V2A chain. No architecture, parameter count, sample rate, max duration, or benchmark numbers are disclosed; the launch is a 43-second co-produced short film and a credits promo. Useful only as a closed-source reference point — confirms that the third frontier video lab (after Google Veo and OpenAI Sora) has now shipped native joint audio, leaving Vidu and Pika as the remaining major closed labs whose joint-audio status is unclear.
• [2026-05-23] OmniForcing: Unleashing Real-time Joint Audio-Visual Generation: OmniForcing — first real-time streaming joint T2AV. Distills the LTX-2 bidirectional dual-stream teacher (14B video + 5B audio) into a causal AR streaming student via a three-stage pipeline: Stage I Bidirectional DMD → causal-conversion stages → Joint Self-Forcing Distillation. Two dual-stream-specific contributions over the video-only Self-Forcing / Causal Forcing / Causal Forcing++ lineage: Asymmetric Block-Causal Alignment + zero-truncation Global Prefix to prevent cross-modal sync drift under asymmetric token rates, and an Audio Sink Token + Identity-RoPE to stabilize the gradient explosion that audio-token sparsity causes during causal shifting. Modality-independent rolling KV-cache at inference. Result: ~25 FPS / ~0.7 s TTFC / ~35× speedup over teacher (5 s clip: 197 s → ~0.7 s first-chunk + streaming), JavisBench fidelity on par with teacher. This is the first wiki entry that takes joint A+V out of the offline-minute-scale regime — orthogonal to the systems-level acceleration FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU and stackable in principle.
• [2026-05-23] FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU: Hao AI Lab FastVideo team accelerates LTX-2.3 (open T2AV) to ~4.55 s end-to-end for 5 s × 1080p × 24 FPS with audio on a single B200, claimed as 3.9× over the next-fastest published 1080p option. Recipe is system-only: NVFP4 linears, SM100/SM103 attention kernels, full-pipeline graph/kernel fusion, optimized ffmpeg I/O — no distillation, no architectural change. The first wiki entry that pushes joint A+V into sub-5-second single-GPU 1080p, on the systems lever rather than the algorithmic one.
• [2026-05-23] The Design Space of Tri-Modal Masked Diffusion Models: Apple’s tri-modal Masked Diffusion Model — discrete-token unified vocabulary over text + image + audio, single transformer trunk, no dual-stream cross-attention. Pretrained from scratch (3B, 6.4T tokens) on text + image-text + audio-text. Provides the design-space study (SDE batch-size invariance, $\gamma^* \approx 0.50$ drift-horizon allocation, multiplicative Kaplan-form scaling laws with $D^* \propto N^{0.476}$) that the dual-stream DiT cluster has not yet produced. The first credible alternative architectural family to dual-stream DiT for unified multimodal generation — though currently text+image+audio, not text+image+audio+video.
• [2026-05-23] SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model: SkyReels’ third entry — symmetric dual-stream MMDiT with one pretrained video branch and one from-scratch audio branch, a shared frozen MLLM text encoder, audio RoPE rescaled by ~10.4× to match video time, and a unified channel-concatenation inpainting interface that handles T2V / I2V / extension / interpolation / editing as different mask configurations. Targets 1080p × 32 FPS × 15 s multi-shot via a two-pass low-res-full + high-res-keyframes pipeline; reports #1 on the Artificial Analysis Arena. The third independent confirmation of the dual-stream + bidirectional cross-attention recipe.
• [2026-05-22] MOVA: Towards Scalable and Synchronized Video-Audio Generation: OpenMOSS MOVA — 32B / 18B-active MoE with a 14B pretrained video tower + 1.3B pretrained audio tower bridged by bidirectional cross-attention at every block, an Aligned-RoPE mechanism that maps both modalities onto unified physical time, and Dual CFG as a per-scene quality/lip-sync knob. Adds the second open architecture data point and ships a separate 732-prompt MOVA Arena benchmark for reproducible human-preference Elo evaluation of joint A+V models.
• [2026-05-22] LTX-2: Efficient Joint Audio-Visual Foundation Model: Lightricks LTX-2 — the first credible open T2AV foundation model. Asymmetric dense 14B video + 5B audio dual-stream DiT with bidirectional temporal cross-attention and cross-modality AdaLN; frozen Gemma3-12B + per-stream text connectors + learnable thinking tokens; modality-aware CFG; 18× faster than Wan 2.2-14B at 720p × 121 frames despite generating audio. Sets the open T2AV baseline.

Open questions

• Capacity allocation across modalities: dense asymmetric (LTX-2 14B+5B) vs MoE asymmetric (MOVA 32B/18B with 14B+1.3B backbone) vs symmetric architecture with asymmetric init (SkyReels-V4 — symmetric, pretrained video, from-scratch audio). No head-to-head comparison yet on matched compute.
• Temporal alignment mechanism: per-modality temporal-RoPE rescaling (MOVA Aligned RoPE; SkyReels-V4 218/21× audio rescaling) vs keeping native 3D-vs-1D RoPE and only using the temporal component during cross-modal attention (LTX-2). Are these meaningfully different or two equivalent framings? OmniForcing adds a third option for the streaming setting — an Asymmetric Block-Causal Alignment with a zero-truncation Global Prefix that handles the rate mismatch at chunk boundaries rather than via per-modality RoPE rescaling (OmniForcing: Unleashing Real-time Joint Audio-Visual Generation).
• Whether the audio branch should be pretrained alongside video or trained from scratch — MOVA pretrains both, LTX-2 trains both jointly, SkyReels-V4 initializes the video branch from a pretrained T2V and trains audio from scratch. Open whether the from-scratch audio branch underperforms in absolute quality or merely in convergence speed.
• Whether unified-inpainting-as-interface (SkyReels-V4’s channel-concat formulation that subsumes T2V, I2V, extension, interpolation, and editing under one model) scales to longer-than-15-s videos without per-task degradation, or whether dedicated heads are still needed at longer rollouts.
• Whether the open dual-stream consensus generalizes to more than two synchronized modalities (e.g. + IMU, + depth, + control signals) without breaking the bidirectional-cross-attention-at-every-block recipe. None of the three dual-stream filed papers tests this, but The Design Space of Tri-Modal Masked Diffusion Models supplies a partial answer from the discrete-MDM side: a single trunk + unified vocabulary scales to three modalities at 3B / 6.4T tokens without per-modality towers.
• Discrete unified MDM vs continuous dual-stream DiT is now an explicit fork. The Design Space of Tri-Modal Masked Diffusion Models is currently scoped to text+image+audio (no video); LTX-2 / MOVA / SkyReels-V4 are video+audio (no joint text generation). Whether the two recipes converge — i.e. whether discrete MDM extends to video tokens, or whether continuous dual-stream DiT extends to interleaved text generation — is open and would determine which recipe survives.
• The MOVA Arena benchmark (732 prompts, human-preference Elo) is currently the only public evaluation set designed specifically for joint A+V — open question whether the field consolidates around it or fragments. JavisBench (used by OmniForcing) is now a second candidate.
• Whether closed-source joint A+V systems (Veo 3, Sora 2, Kling 2.6 Kling Video 2.6 — first Kling AI model with native audio (announcement), Seedance 2.0, Vidu Q3) use the same dual-stream-with-bidirectional-cross-attention recipe or something materially different. The open-source convergence is striking, but evidence about the closed-source recipe is indirect — Kling 2.6 in particular ships with zero technical disclosure.
• How systems-level optimization (NVFP4 + Blackwell-tuned attention + graph fusion, per FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU) composes with algorithmic step reduction / distillation for joint A+V — neither the LTX-2 paper nor FastVideo combines both. OmniForcing now opens the streaming leg of this comparison, and is the natural target to stack FastVideo’s kernels on top of. The stackability and quality interaction at 1080p × 24 FPS is the obvious next datapoint.
• Whether the Audio Sink Token + Identity-RoPE recipe is specific to LTX-2’s 5B audio branch, or a general fix for any dual-stream causal-distillation pipeline. OmniForcing: Unleashing Real-time Joint Audio-Visual Generation introduces it but doesn’t ablate against alternative audio-sparsity fixes (e.g. simple loss reweighting, or per-modality gradient clipping). The architectural cleanliness suggests generalizability but it’s untested on MOVA or SkyReels-V4.

Papers

2026-04

• Seedance 2.0: Advancing Video Generation for World Complexity — Seedance 2.0: Advancing Video Generation for World Complexity (published 2026-04-15)
• Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing — Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing (published 2026-04-12)
• LPM 1.0: Video-based Character Performance Model — LPM 1.0: Video-based Character Performance Model (published 2026-04-09)
• HappyHorse 1.0: #1 State-of-the-Art Joint Audio-Video Generator (Alibaba Taotian Future Life Lab) — HappyHorse 1.0: #1 State-of-the-Art Joint Audio-Video Generator (Alibaba Taotian Future Life Lab) (published 2026-04-07)

2026-03

• Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model (daVinci-MagiHuman) — Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model (daVinci-MagiHuman) (published 2026-03-23)
• OmniForcing: Unleashing Real-time Joint Audio-Visual Generation — OmniForcing: Unleashing Real-time Joint Audio-Visual Generation (published 2026-03-12)
• FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU — FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU (published 2026-03-11)

2026-02

• The Design Space of Tri-Modal Masked Diffusion Models — The Design Space of Tri-Modal Masked Diffusion Models (published 2026-02-25)
• SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model — SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model (published 2026-02-25)
• DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation — DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation (published 2026-02-12)
• MOVA: Towards Scalable and Synchronized Video-Audio Generation — MOVA: Towards Scalable and Synchronized Video-Audio Generation (published 2026-02-09)

2026-01

• LTX-2: Efficient Joint Audio-Visual Foundation Model — LTX-2: Efficient Joint Audio-Visual Foundation Model (published 2026-01-06)

2025-12

• Wan 2.6 — native multimodal video model with audio, multi-shot storytelling, starring references (Alibaba Wan announcement) — Wan 2.6 — native multimodal video model with audio, multi-shot storytelling, starring references (Alibaba Wan announcement) (published 2025-12-16)
• Seedance 1.5 pro: A Native Audio-Visual Joint Generation Foundation Model — Seedance 1.5 pro: A Native Audio-Visual Joint Generation Foundation Model (published 2025-12-15)
• JoVA: Unified Multimodal Learning for Joint Video-Audio Generation — JoVA: Unified Multimodal Learning for Joint Video-Audio Generation (published 2025-12-15)
• Runway GWM-1 launch broadcast — world model, avatars, audio, multi-shot editing — Runway GWM-1 launch broadcast — world model, avatars, audio, multi-shot editing (published 2025-12-11)
• Kling Video 2.6 — first Kling AI model with native audio (announcement) — Kling Video 2.6 — first Kling AI model with native audio (announcement) (published 2025-12-03)

2025-11

• 3MDiT: Unified Tri-Modal Diffusion Transformer for Text-Driven Synchronized Audio-Video Generation — 3MDiT: Unified Tri-Modal Diffusion Transformer for Text-Driven Synchronized Audio-Video Generation (published 2025-11-26)
• PrismAudio: Decomposed Chain-of-Thoughts and Multi-dimensional Rewards for Video-to-Audio Generation — PrismAudio: Decomposed Chain-of-Thoughts and Multi-dimensional Rewards for Video-to-Audio Generation (published 2025-11-24)

2025-10

• LongCat-Flash-Omni Technical Report — LongCat-Flash-Omni Technical Report (published 2025-10-31)
• Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation — Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation (published 2025-10-28)
• GAGA-1 release — 'Holistic AI Actor' with co-generated voice, lipsync, and performance (Sand.ai / Gaga AI) — GAGA-1 release — ‘Holistic AI Actor’ with co-generated voice, lipsync, and performance (Sand.ai / Gaga AI) (published 2025-10-10)
• Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction — Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction (published 2025-10-03)

2025-09

• Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation — Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation (published 2025-09-30)
• Sora 2 is here — Sora 2 is here (published 2025-09-30)
• X-Streamer: Unified Human World Modeling with Audiovisual Interaction — X-Streamer: Unified Human World Modeling with Audiovisual Interaction (published 2025-09-25)
• Wan 2.5 live on WaveSpeed — 5s/10s clips up to 1080p (AI Pulse correction tweet) — Wan 2.5 live on WaveSpeed — 5s/10s clips up to 1080p (AI Pulse correction tweet) (published 2025-09-23)
• UniVerse-1: Unified Audio-Video Generation via Stitching of Experts — UniVerse-1: Unified Audio-Video Generation via Stitching of Experts (published 2025-09-07)

2025-06

• Ming-Omni: A Unified Multimodal Model for Perception and Generation — Ming-Omni: A Unified Multimodal Model for Perception and Generation (published 2025-06-11)

2025-05

• Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (MultiTalk) — Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (MultiTalk) (published 2025-05-28)

2025-02

• UniForm: A Unified Multi-Task Diffusion Transformer for Audio-Video Generation — UniForm: A Unified Multi-Task Diffusion Transformer for Audio-Video Generation (published 2025-02-06)

Unknown date

• Veo 3 Tech Report — Veo 3 Tech Report (published unknown)