OmniForcing: Unleashing Real-time Joint Audio-Visual Generation

TL;DR

OmniForcing distills the offline bidirectional dual-stream T2AV teacher LTX-2 (14B video + 5B audio) into a causal streaming autoregressive generator that runs at ~25 FPS on a single GPU with ~0.7 s time-to-first-chunk — a ~35× speedup over the teacher, which takes ~197 s for a 5-second clip. The technical core is solving two problems that block naive causal distillation of dual-stream architectures: severe temporal asymmetry between modalities (video ≪ audio token rate) and the gradient explosion caused by extreme audio-token sparsity under causal shifting. The fix is an Asymmetric Block-Causal Alignment with zero-truncation Global Prefix, an Audio Sink Token mechanism with Identity-RoPE constraint, and a Joint Self-Forcing Distillation paradigm that self-corrects cumulative cross-modal errors during long rollouts. This is the first wiki paper that pushes joint A+V generation into the real-time streaming regime — the prior open frontier (LTX-2, MOVA, SkyReels-V4) was minute-scale offline.

Key claims

Joint audio-visual diffusion models like LTX-2 and Veo 3 take ~197 s to generate a 5-second clip due to bidirectional full-sequence attention, making real-time or interactive use unreachable [Abstract, Community].
OmniForcing is the first framework to distill an offline, dual-stream bidirectional audio-visual diffusion model into a streaming autoregressive generator [Abstract].
Naive causal distillation of dual-stream architectures triggers severe training instability due to the extreme temporal asymmetry between video and audio token streams and the resulting token sparsity [Abstract].
An Asymmetric Block-Causal Alignment with a zero-truncation Global Prefix prevents multi-modal synchronization drift across the asymmetric token rates [Abstract].
The gradient explosion from extreme audio-token sparsity during the causal shift is resolved by an Audio Sink Token mechanism equipped with an Identity-RoPE constraint [Abstract].
A Joint Self-Forcing Distillation paradigm lets the model dynamically self-correct cumulative cross-modal errors arising from exposure bias during long rollouts [Abstract].
A modality-independent rolling KV-cache inference scheme enables ~25 FPS streaming T2AV generation on a single GPU [Abstract].
End-to-end: ~25 FPS streaming, ~0.7 s time-to-first-chunk, ~35× speedup over LTX-2 teacher, with visual and acoustic fidelity on par with the teacher across nearly all dimensions on JavisBench [Abstract, Community].
Distillation pipeline is three-stage: Stage I — Bidirectional DMD (Distribution Matching Distillation compresses multi-step diffusion sampling into few-step denoising while preserving original global attention); subsequent stages introduce causality and self-forcing on top [GitHub README].
The student is built on LTX-2 specifically (14B video + 5B audio asymmetric dense dual-stream DiT) [GitHub README].

Method

OmniForcing starts from a pretrained dual-stream bidirectional T2AV teacher — LTX-2’s asymmetric 14B video + 5B audio MMDiT with bidirectional cross-attention at every block — and converts it into a streaming causal generator in three distillation stages. Stage I (Bidirectional DMD) reduces the teacher’s multi-step diffusion sampling to a few-step student while keeping global bidirectional attention, getting most of the quality at a fraction of the NFE. Stages II–III introduce causality: the model is taught to denoise in chunks while attending only to past chunks via a rolling per-modality KV cache, and Joint Self-Forcing Distillation closes the train-vs-inference gap by feeding the student’s own previous-chunk outputs into the conditioning during training.

Two specific architectural moves handle the dual-stream asymmetry that breaks naive causal distillation. (1) Asymmetric Block-Causal Alignment + zero-truncation Global Prefix: rather than aligning video and audio chunks one-to-one (which fails because audio token rate is ~10× video in latent space), each video chunk is paired with a block of audio tokens at the matching physical-time interval, and a zero-truncation Global Prefix anchors the start of both streams so that cumulative drift between the two modalities is bounded. (2) Audio Sink Token with Identity-RoPE: when the audio stream becomes sparse under chunkwise causal shifting, gradients explode because a small number of audio tokens carry the full cross-modal-attention signal; allocating dedicated sink tokens (cf. attention-sink lineage) with a constant Identity-RoPE position widens the softmax denominator without inflating compute, stabilizing training. At inference a modality-independent rolling KV cache lets video and audio each keep their own bounded window, decoupling the two streams’ memory budgets.

Results

Headline numbers: ~25 FPS streaming joint T2AV generation on a single GPU, ~0.7 s time-to-first-chunk, ~35× speedup over the LTX-2 teacher’s ~197 s / 5 s clip [Abstract, Community, GitHub README]. The student maintains visual and acoustic fidelity on par with the bidirectional teacher across nearly all dimensions on JavisBench [Community]. As of the v2 release the paper is offline-released; code, weights, and full eval tables are scheduled within two weeks per the project page [GitHub README].

The comparison set positions OmniForcing against (a) the dual-stream bidirectional T2AV baseline LTX-2 (its own teacher) and (b) prior streaming-AR video distillations like Self-Forcing, CausVid, and DMD — the paper explicitly thanks LTX-2, Self-Forcing, CausVid, and DMD as the works it builds on [GitHub README]. Unlike those streaming predecessors, OmniForcing jointly streams synchronized audio and video rather than video alone [Community].

Why it’s interesting

This is the first paper on the wiki to push joint audio-video generation into the real-time streaming regime. The prior open T2AV frontier — LTX-2: Efficient Joint Audio-Visual Foundation Model, MOVA: Towards Scalable and Synchronized Video-Audio Generation, SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model — is all minute-scale offline; the only “fast” alternative (FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU) attacks the same problem from the systems-level (NVFP4 + Blackwell kernels + graph fusion on a still-bidirectional LTX-2), getting ~4.55 s end-to-end for 5 s at 1080p but not actually streaming. OmniForcing takes the orthogonal algorithmic lever — distill the dual-stream bidirectional teacher into a causal AR streaming student — and demonstrates the two are composable in principle.

It also lands cleanly into the autoregressive-video-distillation lineage already on the wiki: Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation showed that distilling AR students from bidirectional teachers requires fixing a frame-level injectivity violation, and Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generation replaced offline PF-ODE trajectories with causal consistency distillation. OmniForcing extends the same playbook into the dual-modality regime and exposes two new failure modes (cross-modal sync drift, audio-token-sparsity-driven gradient explosion) plus their fixes (Block-Causal Alignment + Global Prefix; Audio Sink Token + Identity-RoPE). The Audio Sink Token in particular is a direct cross-pollination from the attention-sink lineage tracked on the wiki (The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks, A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training) — sink tokens are now load-bearing in joint multimodal streaming, not just LLM stability.