Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (MultiTalk)

TL;DR

MultiTalk adapts Wan2.1-I2V-14B into the first audio-driven multi-person conversational video generator, accepting one audio stream per speaker plus a multi-person reference image and text prompt. The core contribution is Label Rotary Position Embedding (L-RoPE): a scheme that assigns scalar labels to per-person video-latent regions (located adaptively via the reference-to-video self-attention map) and to each audio stream, then rotates audio-cross-attention queries/keys by the label-derived angle so the dot product binds each audio stream to the right person, even when subjects move around the frame. Training is partial-parameter (only audio cross-attention + audio adapter unfrozen) and multi-task (AI2V + plain I2V with multi-event interaction data), which the authors show is required to preserve the base model’s instruction-following ability under limited compute.

Key claims

The new task — audio-driven multi-person conversational video — is not solved by existing single-human animators: naively concatenating dual-stream audio leads to all subjects lip-syncing the same stream, and naively splitting video latents into left/right halves only works for static framings [§1, §3.3].
Among four multi-stream injection schemes (concatenate-then-attend / per-stream attend-then-add / left-right latent split / L-RoPE), only L-RoPE reliably binds the right audio to the right person when subjects move [§3.3, Fig. 3].
L-RoPE assigns the first person a video-latent label range [0,4], the second person [20,24], background a static −1, and the two audio streams static 2 and 22 respectively; the audio-cross-attention dot product is then biased by the angular difference of these labels and binds each audio stream to its matched person [§3.3].
Adaptive person localization is done from the reference-to-video self-attention map of the I2V backbone: per-person masks on the reference image are scored against each video latent token’s similarity to that reference region, yielding per-token person assignment without an external detector [§3.3, Fig. 4].
L-RoPE is insensitive to the specific label range; [0,2]/[2,4] and [0,4]/[20,24] give comparable Sync-C/Sync-D/E-FID/FID/FVD on MTHM [Table 3].
Partial parameter training (only audio cross-attention + audio adapter unfrozen, all other Wan2.1 layers frozen) preserves the base model’s instruction-following and prevents hand/object distortion that full-parameter training causes under limited data [§3.4].
Multi-task hybrid training is required: AI2V (audio + image → video) on talking-head/body data plus plain I2V on a 200K-clip multi-event human-object-interaction set; AI2V alone degrades prompt adherence [§3.4, Fig. 5].
Multi-stream multi-person training does not regress the model on single-person evaluation: the “multiple” variant matches or beats the “single” variant on HDTF and CelebV-HQ (Sync-C 8.53/7.33 vs 8.54/7.07; FVD 124.06/184.86 vs 95.99/219.19) [Table 1].
On HDTF talking head: MultiTalk-single Sync-C 8.54 / Sync-D 6.69 / FID 24.01 beats Sonic (8.35 / 6.43 / 29.53), Hallo3 (6.55 / 8.49 / 33.98) and Fantasy Talking (3.61 / 10.78 / 32.64) [Table 1].
On EMTD talking body: MultiTalk-multiple Sync-C 8.34 / Sync-D 7.30 / FID 31.93 beats Echomimic v2 (6.31 / 8.41 / 35.99) and Fantasy Talking (3.32 / 11.41 / 37.68) [Table 2].
Audio embeddings come from Wav2Vec with a custom audio adapter: split the first frame from the remaining frames along the temporal axis, downsample the tail, MLP-encode each, concatenate, and MLP again to match the 3D-VAE-compressed video latent frame rate [§3.2, Eq. 2].
Long video is generated autoregressively by reusing the last few latent frames of the previous clip as additional conditioning (zero-padded for the rest, concatenated with noise and a video mask) at the next inference step [§3.5].
Training: 2K hours single-person talking video + 200K multi-event interaction clips (~10s avg) in stage 1, 100 hours of two-person conversation in stage 2; AdamW with constant LR 1e-5 + warmup on NVIDIA H800-80G [§4.1].
Limitation: the model is noticeably worse with TTS-synthesized audio than real audio because training audio is all real — facial expression degrades on the synthetic side [§5].

Method

The backbone is the open-source Wan2.1-I2V-14B DiT, conditioned on a CLIP-encoded reference image, T5 text embeddings, and a 3D-VAE-encoded video latent. Audio is injected by inserting a LayerNorm + cross-attention block after the text cross-attention in every DiT block; queries come from video latents, keys/values from the Wav2Vec-derived audio embedding pipeline described above. For multi-person conversation the same audio cross-attention is reused, but two streams are concatenated along the key/value axis.

The binding problem — which audio stream attends to which person — is solved by L-RoPE. For each token in the video-latent query, the model first identifies which person (or background) that token belongs to by computing similarity of the token to each masked-out subject in the reference image via the first-frame-to-video self-attention map. Each person gets a range of labels [αi, βi] (e.g. person 1 → [0,4], person 2 → [20,24]); the actual scalar label for a given token is interpolated within its person’s range by the similarity score. The two audio streams are tagged with static scalar labels picked to sit inside the corresponding range (2 for person 1, 22 for person 2); background is −1. RoPE-style rotation by the label-derived angle then biases the cross-attention dot product so a video token “near” label 2 attends to audio stream 1 and a video token “near” label 22 attends to audio stream 2. The label scheme is the key innovation: it generalizes the left/right latent-split heuristic to arbitrary moving subjects via a single learnable position-encoding mechanism.

Training is two-stage and partial-parameter: stage 1 trains single-person AI2V on 2K hours; stage 2 adds 100 hours of two-person conversation. Only the audio cross-attention and audio adapter weights are unfrozen — everything else in Wan2.1 stays at its pretrained weights. A hybrid AI2V + I2V multi-task schedule shares parameters across the two heads; the I2V leg trains on 200K multi-event human-object interaction clips (zeroed audio embedding) and is what preserves the base model’s prompt-following on body motion / object interaction.

Results

Talking head (HDTF): MultiTalk-single — Sync-C 8.54, Sync-D 6.69, E-FID 1.00, FID 24.01, FVD 95.99 (best Sync-C, best E-FID, best FID; second-best Sync-D and FVD). Beats Sonic, Hallo3, AniPortrait, VExpress, Echomimic, Fantasy Talking on most axes [Table 1].
Talking head (CelebV-HQ): MultiTalk-multiple — Sync-C 7.33, Sync-D 7.18, E-FID 1.48, FID 34.08, FVD 184.86 (best Sync-C, best FVD) [Table 1].
Talking body (EMTD): MultiTalk-multiple — Sync-C 8.34, Sync-D 7.30, E-FID 1.51, FID 31.93, FVD 238.77 vs Echomimic v2 (6.31 / 8.41 / 1.91 / 35.99 / 163.60) and Fantasy Talking (3.32 / 11.41 / 1.98 / 37.68 / 284.29). Best on Sync-C, E-FID, FID [Table 2].
Multi-person conversation (MTHM, authors’ new internal dataset): no prior baseline — only ablation against video-concatenation; MultiTalk produces interactively consistent left-right scenes whereas concatenation produces seam artifacts and incorrect lip motion [Fig. 7].
L-RoPE label-range ablation: [0,2]/[2,4] and [0,4]/[20,24] give Sync-C 7.47 vs 7.56, FVD 506.49 vs 508.01 on MTHM — virtually identical, confirming insensitivity to range choice [Table 3].
Qualitative: under explicit text prompts (e.g. “Nick picks up the mug and gently touches Judy’s head”), only MultiTalk among the compared methods executes both the lip-sync and the described action; baselines either follow the audio but not the prompt or vice versa [Fig. 6, project page].

Why it’s interesting

MultiTalk is the first wiki entry to seriously attack multi-stream audio-to-person binding as a stand-alone research problem — adjacent to but distinct from the Joint audio-video generation concept’s joint generation lineage (LTX-2: Efficient Joint Audio-Visual Foundation Model, MOVA: Towards Scalable and Synchronized Video-Audio Generation, SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model) where both modalities are outputs. Here audio is an input condition, but the architectural challenge — how to bind one audio stream to one spatial region of the video latent without an external detector — is what’s novel. L-RoPE is a clean answer that piggybacks on RoPE’s existing position-encoding role and works adaptively as subjects move, unlike the static left/right latent split it explicitly improves on.

The closest sibling is TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models, which also adapts Wan2.1-I2V-14B into an audio-driven avatar via new cross-attention layers and partial-parameter training, but for a single-character FaceTime application with face-region attention masking. MultiTalk’s L-RoPE is the natural generalization to N speakers; TalkingMachines’s Score-VAE disaggregation and CausVid-style sparse-causal distillation are the natural complement for real-time deployment that MultiTalk doesn’t yet do. Stacking the two — L-RoPE binding inside a TalkingMachines-style 2-NFE causal AR student — is the obvious next paper.

The partial-parameter + multi-task training recipe is also a strong empirical data point for Parameter-Efficient Finetuning: adapting a 14B foundation video model to a new conditioning modality with only the new cross-attention layers unfrozen plus a parallel I2V leg on diverse interaction data is enough to add audio without losing prompt adherence, and explicitly cheaper than full-parameter training.