MOVA: Towards Scalable and Synchronized Video-Audio Generation

TL;DR

MOVA (MOSS Video and Audio) is OpenMOSS / SII’s open-source joint video+audio foundation model: a 32B-parameter Mixture-of-Experts (18B active at inference) with an asymmetric dual-tower architecture — a pretrained 14B video diffusion tower and a pretrained 1.3B audio diffusion tower — coupled by a bidirectional cross-attention “bridge” so the two modalities exchange information at every layer. The model supports image+text → video+audio (IT2VA) and text → video+audio (T2VA) at up to 720p, 8 s clips, with emphasis on lip-synced multilingual speech, environment-aware sound effects, and content-aligned music. The release ships full weights (360p and 720p), inference code, LoRA training scripts, ComfyUI integration, an SGLang Day-0 backend, and a 732-sample Arena benchmark; on Verse-Bench it leads open-source baselines on lip-sync and ASR-accuracy metrics, with reported Elo ~1113.8 in pairwise human preference.

Key claims

Open-source video generators have largely treated audio as an afterthought, leaving a quality gap vs proprietary joint systems like Veo 3 and Sora 2; cascaded T2V→V2A pipelines accumulate error and drift out of sync [§1].
MOVA’s architecture is an asymmetric dual-tower: a pretrained 14B video tower + a pretrained 1.3B audio tower, fused by a bidirectional cross-attention bridge that runs at every block, so each modality continuously perceives the other’s hidden state during denoising [§3].
The total model is 32B parameters with 18B active at inference under a Mixture-of-Experts design — capacity scaled well past the LTX-2 / Ovi class of open joint models [§Abstract, §3].
An Aligned RoPE mechanism reconciles the large sampling-density gap between video and audio by mapping both modalities’ tokens onto a unified physical-time coordinate via per-modality scaling, addressing audio-visual desynchronization at the position-encoding level [external project page].
Dual CFG inference exposes two independent guidance weights — one for text instructions, one for cross-modal coupling — so the user can trade visual quality off against lip-sync fidelity per-scene [external project page; §4-5].
A fine-grained bimodal data annotation pipeline (lip-sync labels, environmental sound tags, multi-speaker / speaker-switching annotations) is described as the data side of the scaling story [§4].
On Verse-Bench, MOVA-720p with Dual CFG enabled reports an LSE-D of 7.094 and LSE-C of 7.452, leading open-source baselines on the lip-sync subset (Set 3) [§6, Tables].
MOVA also leads on cpCER — a metric that combines speech-recognition accuracy with speaker-switching accuracy on a multi-speaker subset proposed by the authors [§6].
Arena-based human evaluation collected >5,000 valid votes across the latest open-source A+V models; MOVA achieves Elo 1113.8 (from a 1000 baseline), with win rates >50% across the field and >70% against Ovi [§6, Fig. 8].
The 732-sample MOVA Arena benchmark and per-variant generated-video sets are released on Hugging Face for reproducible Elo-style evaluation, separately from the weights [§6 / repo].

Method

MOVA is built as a bimodal DiT with two pretrained backbones rather than a single shared trunk. The video tower is a 14B diffusion transformer; the audio tower is a 1.3B diffusion transformer operating on a mel-spectrogram latent space. The two towers are not interleaved into one stream — they run in parallel — but at every block a bidirectional cross-attention bridge lets queries from each modality attend to keys/values from the other, so visual generation can condition on audio rhythm (and vice versa) without forcing the two modalities into a shared latent. The full system is an MoE with 32B total / 18B active, distinguishing it from LTX-2’s dense 19B asymmetric (14B + 5B) design.

Two pieces address the joint-generation pathology that text-only conditioning is too weak to keep video and audio synchronized. First, Aligned RoPE rescales the temporal RoPE on each tower so video and audio tokens, which have very different sampling rates (e.g. tens of fps vs hundreds of audio frames/s), share a single physical-time coordinate — making “what’s happening at second 1.7” the same index across modalities. Second, Dual CFG at inference uses two independent guidance scales, one for the text condition and one for the cross-modal bridge condition; this is the same family of multi-scale CFG that LTX-2 uses, but applied as a quality vs lip-sync knob per scene rather than per stream.

Training is supported by a bimodal annotation pipeline that produces fine-grained labels for speech, environmental sounds, music, and speaker-switching events. The release stack is uncommonly complete: bf16 weights for 360p and 720p, training pipelines, LoRA fine-tuning configs (with documented memory/step-time profiles), an SGLang inference path with day-0 support, ComfyUI nodes via the community comfyui-mova plugin, and the MOVA Arena evaluation set (732 prompts with first-frame images, plus generated videos for each MOVA variant).

Results

Lip-sync (Verse-Bench Set 3, MOVA-720p + Dual CFG): LSE-D 7.094, LSE-C 7.452 — best vs the listed open-source baselines [§6, Tables].
Speech accuracy: best cpCER on a multi-speaker / speaker-switching subset proposed by the authors [§6].
Human preference (Arena, >5,000 votes, all latest open-source A+V models): Elo 1113.8 starting from 1000, win-rate >50% against the field and >70% against Ovi [§6, Fig. 8].
Audio quality and AV-Align metrics are reported on all Verse-Bench subsets; lip-sync and speech metrics are restricted to Set 3 because they require speaking-face content [§6].
Released checkpoints: MOVA-360p (≈15 GB on disk, faster prototyping) and MOVA-720p (full quality) — both support IT2VA and T2VA. SGLang has day-0 high-throughput inference; LoRA fine-tuning is documented across multiple modes with VRAM / step-time tables.
Apache 2.0 license — commercial-use permissible.

Why it’s interesting

MOVA is the second open joint video+audio foundation model (after LTX-2: Efficient Joint Audio-Visual Foundation Model) the team has filed this cycle, and the two converge on the same high-level architecture — asymmetric dual-tower diffusion transformer with bidirectional cross-attention coupling — from independent code bases. That’s a strong piece of evidence that for joint A+V, the right inductive bias is “two pretrained backbones bridged at every block,” not “one trunk with a few extra audio tokens.” The interesting axis of disagreement is MoE vs dense: MOVA uses a 32B/18B MoE on top of the dual-tower, while LTX-2 is dense 14B+5B; the two will be directly comparable on Verse-Bench / Arena once external numbers land.

The Aligned-RoPE story is the most transferable bit of the recipe. Sampling-rate mismatch between modalities is the unstated reason most cascaded systems drift — getting both modalities onto the same physical-time index at the position-encoding level should generalize cleanly to any bimodal DiT, including ones the team might build internally. Dual CFG as a per-scene quality vs lip-sync knob is in the same family as LTX-2’s bimodal CFG, but it’s exposed as a user-tunable inference parameter rather than a fixed-recipe choice — useful if downstream applications care about dialogue scenes vs ambient-only scenes differently.

The release completeness (weights at two resolutions, training pipeline, LoRA configs, SGLang day-0, ComfyUI, Apache 2.0, and a separately-published 732-sample human-preference Arena benchmark) sets a new bar for open A+V releases. The Arena benchmark in particular is worth grabbing — it’s the only public evaluation set designed for human-preference Elo on jointly-generated video+audio, and applying it to LTX-2 / Ovi / future internal models would yield a single comparable yardstick.