LTX-2: Efficient Joint Audio-Visual Foundation Model

TL;DR

LTX-2 is Lightricks’ open-weights text-to-audio+video (T2AV) foundation model: a single 19B-parameter asymmetric dual-stream DiT (14B video + 5B audio) that jointly generates synchronized video and stereo audio from a single text prompt. The two streams use modality-specific causal VAEs and 3D-vs-1D RoPE, but exchange information at every block via bidirectional temporal cross-attention and cross-modality AdaLN gating. On an H100, LTX-2 runs at 1.22 s/step at 720p×121 frames vs 22.30 s/step for Wan 2.2-14B (video-only) — roughly 18× faster — while reportedly matching Veo 3 / Sora 2 in human preference and supporting up to 20 s clips with synchronized stereo audio. Weights, code, distilled and quantized variants (fp8/nvfp4), and spatial/temporal upscalers are all released.

Key claims

A unified model is required for true T2AV: decoupled T2V→V2A or A2V→V pipelines cannot model the joint distribution because acoustic environment depends on visual context and lip-sync depends on audio, simultaneously [§1].
Asymmetric dual-stream design — 14B video, 5B audio — allocates capacity by modality information density rather than forcing a shared latent, and lets each modality keep modality-appropriate positional embeddings (3D vs 1D RoPE) [§1, §3.1].
Cross-modal coupling uses bidirectional cross-attention with only the temporal RoPE component on queries/keys during inter-stream attention, plus cross-modality AdaLN where one stream’s scale/shift parameters are conditioned on the other stream’s hidden states [§3.1, §3.1.2].
Text conditioning is built on a frozen Gemma3-12B backbone with two additions: a learned multi-layer feature extractor that aggregates intermediate decoder layers (mean-centered, flattened, projected), and per-stream “text connector” transformers with learnable “thinking tokens” that replace padding and act as global information carriers [§3.2.1, §3.2.2].
Modality-aware CFG: each stream gets two independent guidance scales, one for text and one for cross-modal conditioning; reported values are (w_T, w_M) = (4.0, 0.5) for video and (5.0, 1.0) for audio. Stronger cross-modal guidance improves temporal sync and semantic coherence [§4.1].
Audio VAE: stereo input as two-channel mel-spectrogram at 16 kHz; each latent token ≈1/25 s of audio with 128-D features; HiFi-GAN-V1 vocoder doubled in width to jointly upsample and synthesize stereo at 24 kHz [§3.3, §3.3.1].
Multi-scale, multi-tile inference reaches 1080p: base generation at ~0.5 MP, latent upscaling, then overlapping spatial/temporal tile refinement blended in latent space before VAE decoding [§4.2].
Speed: 18× faster than Wan 2.2-14B at 720p×121 frames on H100 (1.22 s/step vs 22.30 s/step), despite LTX-2 being a larger model (19B) and also generating audio [§6.3, Table 1].
Temporal scope: up to 20 s of continuous video with synchronized stereo audio, vs Veo 3 (12 s), Sora 2 (16 s), Ovi (10 s), Wan 2.5 (10 s) [§6.3].
On Artificial Analysis public rankings (Nov 6, 2025) the visual stream alone ranked 3rd in I2V and 4th in T2V, surpassing Sora 2 Pro and Wan 2.2-14B — i.e. joint audiovisual training does not cost video quality [§6.2].

Method

The architecture has three components: modality-specific VAEs, a refined text-conditioning pipeline, and an asymmetric dual-stream DiT. Video latents come from a spatiotemporal causal VAE (the same family as LTX-Video) and use 3D RoPE; audio latents come from a separate causal mel-spectrogram VAE and use 1D temporal RoPE. Each dual-stream block performs four operations in sequence per modality: self-attention, text cross-attention, audio-visual cross-attention, FFN — interleaved with RMS norms. During the audio-visual cross-attention, only the temporal RoPE component is applied to Q/K, forcing inter-modal attention to align in time rather than space. Cross-modality AdaLN gates further allow one modality’s hidden state to scale/shift the other’s representation, useful when the two streams sit at different diffusion timesteps.

Text conditioning is unusual. Rather than read only Gemma3-12B’s final hidden state, LTX-2 extracts all decoder-layer outputs (B, L, D, N_layers), applies mean-centered scaling per layer, flattens to (B, L, D · N_layers), and projects through a learned matrix W to the conditioning dim. W is briefly jointly trained with the diffusion model, then frozen for the rest of training. On top of that, a per-stream “text connector” — a few bidirectional transformer blocks — refines those embeddings and prepends learnable thinking tokens (replacing padded positions) that act as global aggregators before the embeddings ever reach the diffusion cross-attention layers. The authors credit this stack for the model’s phonetic accuracy in synthesized speech.

For inference, LTX-2 uses a bimodal CFG that decouples text guidance and cross-modal guidance with independent scales per stream (§4.1), and a multi-scale / multi-tile pipeline for 1080p: generate at ~0.5 MP, upscale latents, then spatially-and-temporally tile-refine with the same backbone and blend in latent space (§4.2).

Results

Speed (H100, 720p, 121 frames, Euler, CFG=1): LTX-2 19B at 1.22 s/step; Wan 2.2-14B at 22.30 s/step → ~18× speedup despite +5B params and audio generation [§6.3, Table 1].
Audiovisual quality: human preference studies report LTX-2 significantly outperforms Ovi (open-source SOTA peer) and is comparable to Veo 3 and Sora 2 [§6.1].
Visual-only: Artificial Analysis Nov 6 2025 rankings — 3rd I2V, 4th T2V; beats Sora 2 Pro and Wan 2.2-14B on visual benchmarks [§6.2].
Continuous synchronized A+V duration: up to 20 s, exceeding Veo 3 (12 s), Sora 2 (16 s), Ovi (10 s), Wan 2.5 (10 s) [§6.3].
Checkpoints released: full bf16 dev model, fp8 and nvfp4 quantized variants, 8-step CFG=1 distilled variant, a distilled LoRA, plus ×2 spatial and ×2 temporal latent upscalers (per HF model card).

Why it’s interesting

This is the first credible open-weights joint T2AV foundation model and lands directly in Luma’s lane. Three things stand out:

(1) The asymmetric capacity split (14B/5B) is a concrete data point against the symmetric “duplicate-the-backbone-per-modality” approach that Ovi/BridgeDiT take, and it doesn’t appear to cost visual quality — useful evidence for any internal joint-modality work.

(2) The text-conditioning stack — frozen Gemma3 + multi-layer feature extraction + per-stream connector with thinking tokens — is a recipe that’s largely orthogonal to the dual-stream story and could be lifted into video-only or image-only DiTs. The claim that this is what unlocks phonetic-accurate speech is interesting if it generalizes to fine-grained prompt control more broadly.

(3) The 18× speed-up at 720p over Wan 2.2-14B (a smaller, video-only model) is the kind of efficiency story that, if it holds outside Lightricks’ benchmark, changes what “real-time” means for video diffusion. Worth probing whether this is mostly the LTX-Video latent compression carrying over or whether the dual-stream design itself contributes.