InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing

TL;DR

InfiniteTalk is an audio-driven, unlimited-length talking-video model built on the same Wan2.1-I2V-14B backbone as the authors’ earlier MultiTalk. It generalizes audio-driven character animation from image-to-video (I2V) into a new task the paper calls sparse-frame video dubbing: given an input video plus a new audio track, the model regenerates the video — not just lips, but also head pose, body, and facial expressions — while preserving identity via sparse keyframes from the original video. The same model also runs as a single-image-conditioned talker. Long videos are produced autoregressively by reusing the last few latent frames of the previous clip as conditioning.

Key claims

Sparse-frame video dubbing is a new paradigm: rather than editing the mouth region only (traditional dubbing), the model is given freedom to regenerate audio-aligned mouth, facial, and body movements while sparse keyframes preserve identity, emotional cadence, and iconic gestures [Fig. 1, §1].
Unlike single-talking-head methods, InfiniteTalk synchronizes head pose, body posture, and facial expression with audio in addition to lips [§1].
Infinite-length generation is supported via autoregressive clip extension, using the last few latent frames of the prior clip as conditioning at the next inference step [§1, §3].
The model is unified — the same checkpoint handles both image-audio-to-video and video-audio-to-video (dubbing) modes via different conditioning inputs [§1].
Compared to MultiTalk, the same authors report reduced hand/body distortions and superior lip synchronization accuracy [§1, Key Features].
The training stack reuses Wan2.1-I2V-14B as the diffusion backbone with chinese-wav2vec2-base as the audio encoder [Model Preparation].
Inference recommendations: audio CFG ≈ 3–5 for lip-sync quality; FusionX LoRA enables 8-step inference and lightx2v enables 4-step inference, at the cost of color drift past ~1 min and weaker ID preservation in long generations [Quick Inference tips].
V2V dubbing mode can be combined with SDEdit to better follow the source video’s camera trajectory, at the cost of color shift and short-clip-only viability [Quick Inference tips].
The default --mode streaming configuration generates up to 1000 frames (~40 s at 25 fps) per call, extensible indefinitely by re-feeding the tail [CLI args].
Compatible with both 480P and 720P inference and ships FP8 quantized weights for single-GPU low-VRAM use [Quick Inference §3].

Method

The architecture inherits MultiTalk’s recipe wholesale. Wan2.1-I2V-14B serves as the frozen DiT backbone; audio embeddings come from chinese-wav2vec2-base passed through a custom audio adapter (frame-rate-aligned with the 3D-VAE-compressed video latent), then injected via a LayerNorm + cross-attention block inserted after the text cross-attention in every DiT block. Only the audio cross-attention layers and the audio adapter are trained — the same partial-parameter training pattern MultiTalk used to preserve the base model’s prompt-following.

The key conceptual addition over MultiTalk is the sparse-frame conditioning interface. In V2V dubbing mode the source video provides a small set of keyframes (“sparse frames”) that anchor identity and high-level motion; the model is free to regenerate every other frame from audio. This gives the model latitude to align body and head motion to the new audio rather than being locked to the source video’s pixel-level motion, while still preserving identity and gesture style. In I2V mode the same machinery degenerates to a single keyframe (the conditioning image).

Long-video generation is autoregressive: each clip’s last several latent frames (default --motion_frame 9) are passed as conditioning latents into the next clip’s denoising, padded with noise and a video mask — the same recipe MultiTalk uses for >5-second outputs and structurally identical to the Autoregressive Video Generation last-k-latent-frame trick.

The release ships TeaCache and FP8 quantization for inference acceleration; LCM distillation and sparse attention are listed as TODO. Code, weights, Gradio demo, and a ComfyUI branch are released under Apache 2.0 (model weights only — the base Wan2.1 and wav2vec2 weights remain under their original licenses).

Results

The paper does not publish quantitative head-to-head numbers in the README; the README’s quantitative claims are limited to qualitative “superior lip synchronization to MultiTalk” and “reduced hand/body distortions”. A follow-up release (LongCat-Video-Avatar / LongCat-Video-Avatar-1.5, Dec 2025 / May 2026) from the same authors replaces Wav2Vec2 with Whisper-Large and adds 8-step step-distillation and stylized-domain generalization (anime, animals), suggesting the InfiniteTalk recipe has been iterated on rather than retired [README, Latest News].

Why it’s interesting

InfiniteTalk is the direct sequel to Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (MultiTalk) (MultiTalk) by the same author group (Kong, Yang, Gao, Zhang et al. at Meituan/MeiGen-AI) and reuses the same Wan2.1-I2V-14B partial-parameter recipe. Where MultiTalk’s novelty was multi-stream binding (L-RoPE for multi-person), InfiniteTalk’s novelty is sparse-frame video-to-video dubbing: turning the audio-driven I2V model into a V2V dubber by treating only a handful of source frames as identity anchors and letting audio drive the rest. This is structurally the same move Wan-Animate: Unified Character Animation and Replacement with Holistic Replication (Wan-Animate) makes for character-replacement V2V, but the conditioning signal there is a driving video rather than audio.

It also sits beside TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models (TalkingMachines — same Wan2.1-I2V-14B base, single-character FaceTime-style real-time via causal distillation) and EchoMimicV3: 1.3B Parameters are All You Need for Unified Multi-Modal and Multi-Task Human Animation (EchoMimicV3 — 1.3B parameter unified multi-modal/multi-task human animation). The cluster of audio-driven human-animation systems built on a frozen Wan-family DiT + partial-parameter audio cross-attention is now several papers deep and is a plausible candidate concept — see _candidates.md.

The pattern of “freeze the big video DiT, add new cross-attention layers for the new conditioning modality, train only those” makes this a textbook Parameter-Efficient Finetuning success story for adapting a video foundation model to a new task without forgetting the base capabilities.