JoVA: Unified Multimodal Learning for Joint Video-Audio Generation

TL;DR

JoVA (HKU + ByteDance) is a unified text-to-video+audio framework targeting human speech with lip-sync, not just ambient sound. Unlike the dual-stream-with-bidirectional-cross-attention recipe converging in LTX-2 / MOVA / SkyReels-V4, JoVA concatenates video, audio, and text tokens into a single sequence and applies joint self-attention in each transformer layer — no per-modality towers, no cross-attention bridge, no alignment modules. A simple mouth-area loss (BCE-style supervision on a facial-keypoint-derived VAE-latent mask) sharpens lip-speech sync without adding architecture. The paper reports competitive or better results vs unified and audio-driven baselines on lip-sync, speech quality, and overall A+V fidelity.

Key claims

Existing joint-A+V methods address ambient sound but lack the ability to generate human speech that lip-syncs to the generated video [§1].
Existing unified-human-A+V approaches use explicit fusion or modality-specific alignment modules (extra cross-attention stacks, modality bridges), complicating the transformer architecture [Abstract, §1].
JoVA replaces those add-ons with joint self-attention over concatenated video + audio + text tokens within each transformer layer, enabling direct cross-modal interaction with no additional alignment modules [Abstract, §1].
A mouth-area loss based on facial-keypoint detection — masking the mouth region in the VAE latent space and emphasizing its supervision during training — improves lip-speech sync without architectural changes [Abstract, Fig. 6].
The mouth-area mask is validated by zero-masking the detected mouth region in the VAE latent and decoding, showing selective visual degradation localized to the mouth — confirming the localization is correct in latent space [Fig. 6, Fig. 7].
JoVA outperforms or is competitive with both unified and audio-driven SOTA methods on lip-sync accuracy, speech quality, and overall video-audio fidelity [Abstract].
ByteDance affiliation; datasets and models are research-use only and will not ship in ByteDance products [§ Acknowledgements].

Method

JoVA is a single transformer trunk operating on a concatenated sequence of video tokens, audio tokens, and text tokens. Every layer applies joint self-attention across all three modalities, so cross-modal interaction is mediated by attention scores within the unified sequence rather than by a dedicated cross-attention path between separate video and audio towers. This is the same design philosophy as MMDiT-style single-stream models in image generation, ported to joint video-audio synthesis.

The lip-sync mechanism is purely a loss contribution. A face-keypoint detector produces a mouth bounding box per frame; that box is projected into the VAE latent and used as a soft mask that upweights reconstruction (or score-matching) loss on the mouth region. The authors validate the latent-space localization by zero-masking the detected region and decoding the VAE: only the mouth area degrades, confirming the mask is on the right latents.

No extra modules (no audio-tower, no cross-attention bridge, no alignment block) are added beyond what a vanilla joint-sequence transformer needs.

Results

The Abstract states JoVA “outperforms or is competitive with both unified and audio-driven state-of-the-art methods in lip-sync accuracy, speech quality, and overall video-audio generation fidelity.” Specific benchmark numbers and head-to-head tables are in the paper body (not retrievable at filing time due to arxiv rate-limiting).
Mouth-area mask validation (Fig. 6, Fig. 7): zero-masking the predicted mouth region in VAE latent space and decoding produces selective visual degradation localized to the mouth — across four validation samples — confirming the mask is correctly aligned in latent space.

Why it’s interesting

JoVA opens a third architectural fork in the Joint audio-video generation cluster. The open-source consensus the wiki has been tracking — LTX-2 (LTX-2: Efficient Joint Audio-Visual Foundation Model), MOVA (MOVA: Towards Scalable and Synchronized Video-Audio Generation), and SkyReels-V4 (SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model) — converges on dual-stream DiT with bidirectional cross-attention at every block. The discrete-token alternative comes from The Design Space of Tri-Modal Masked Diffusion Models (Apple’s tri-modal MDM). JoVA is neither: it’s a continuous joint A+V model with a single-stream transformer trunk, where cross-modal coupling is just self-attention over concatenated tokens — the simplest possible recipe.

It’s also the first wiki entry that explicitly targets lip-sync human speech as the joint-A+V test case, rather than ambient sound effects. That puts JoVA next to the Audio-Driven Character Animation cluster (MultiTalk Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (MultiTalk), HuMo HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning, Wan-S2V Wan-S2V: Audio-Driven Cinematic Video Generation) but with the audio generated jointly rather than provided as conditioning — a regime no other filed paper occupies. The mouth-area loss is the lip-sync analog of HuMo’s “focus-by-predicting” soft mask on audio cross-attention, applied to the reconstruction side rather than the attention side.