Audio-Driven Character Animation

TL;DR

A cluster of video-generation systems that take audio (speech, singing, or arbitrary sound) plus a reference image as primary conditioning and emit a video of the referenced character moving in time with the audio — lips, head, body, gesture. The 2025 consensus recipe freezes a pretrained video DiT (overwhelmingly the Wan family — Wan2.1-I2V-14B in MultiTalk / InfiniteTalk / TalkingMachines, Wan2.2-14B in Wan-S2V) and adds a new audio cross-attention path with partial-parameter training; subsystems differ in how they bind audio to spatial regions (L-RoPE), to time (local audio windows), and to identity (sparse keyframes vs single reference).

Key claims

• The cluster’s canonical recipe is freeze the big video DiT, add an audio adapter + new cross-attention layers in every DiT block, train only those new parameters — used by MultiTalk on Wan2.1-I2V-14B (Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (MultiTalk) §3.4), inherited by InfiniteTalk on the same backbone (InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing Method), and used by TalkingMachines also on Wan2.1-I2V-14B (TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models §4.1). HuMo extends this to a T2V backbone (Wan-2.1) with audio cross-attention + reference-image latent concatenation, updating only self-attention layers and audio modules (HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning §3.3).
• Partial-parameter training is empirically required to preserve the base model’s instruction-following: full-parameter fine-tuning under limited audio-pairs data causes hand/object distortion and degrades prompt adherence (Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (MultiTalk) §3.4). HuMo’s ablation independently confirms this: full fine-tune drops AES 0.589 → 0.529 and TVA 6.508 → 6.157 vs the self-attention-only update (HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning Table 3).
• Multi-task hybrid training — AI2V (audio-image-to-video) alongside plain I2V on diverse human-object interaction clips — is required for the same reason: AI2V alone degrades prompt adherence (Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (MultiTalk) §3.4). HuMo arrives at the same conclusion via progressive curriculum: start at 80% subject-preservation / 20% audio-sync and shift to 50/50, since single-stage joint training degrades identity similarity (ID-Cur 0.747 → 0.724) (HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning §3.3, Table 3).
• Speech-driven motion is predominantly temporally local, so audio cross-attention works better with a local ±5-latent-frame window per video latent than with global audio cross-attention (TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models §4.1). HuMo independently adopts the same recipe: each video frame attends to a centered temporal window of Whisper audio embeddings (HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning §3.3, Eq. 2).
• Multi-person audio binding cannot be solved by naive dual-stream concatenation (all subjects lip-sync the same stream) or static left-right latent split (fails when subjects move). L-RoPE solves this by assigning label ranges to per-person video regions (located adaptively via the reference-to-video self-attention map) plus static labels to audio streams, biasing the dot product so each stream binds to the right person (Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (MultiTalk) §3.3).
• The same model can be repurposed from image-conditioned (I2V) to sparse-frame-video-conditioned (V2V dubbing) by treating only a handful of source frames as identity anchors and letting audio drive every other frame (InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing §1, Fig. 1).
• Real-time deployment requires asymmetric distillation of the bidirectional teacher into a 2-NFE sparse causal autoregressive student — TalkingMachines distills MultiTalk-style audio-conditioned Wan into a 3-frame-chunk / 2-NFE student with sparse causal attention restricted to (current chunk, previous chunk, starting reference chunk) (TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models §4.2). The Wan2.2 analogue is deployed in RealVideo: Real-time Streaming Conversational Video System (Z.ai), which composes Wan2.2-S2V-14B + Self-Forcing distillation + an LLM/TTS frontend and reports ~306 ms per DiT block at sp=4 / 2 steps on H100s.
• Multi-character listening/speaking turn-taking is handled by a Speaking/Silence-mode embedding (zero-embedding swap for silent faces, SyncNet-conf-based mode assignment at training), letting one model handle both roles (TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models §4.1).
• The cluster scales down: EchoMimicV3 reports a 1.3B unified multi-modal/multi-task human animator, showing the recipe works below the 14B Wan-family scale (EchoMimicV3: 1.3B Parameters are All You Need for Unified Multi-Modal and Multi-Task Human Animation). HuMo’s 1.7B variant on Wan-2.1-1.3B independently demonstrates the same: it already beats most open-source specialists on Sync-C 6.005 vs FantasyTalking 3.202 / Hallo3 5.189 (HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning Table 2).
• Cinematic output (full-body, multi-character interaction, dynamic camera work) — not just talking-head — is the 2025-Q3 frontier. Wan-S2V partitions conditioning channels by scope: text for global dynamics (camera, trajectories, interactions), audio for local fine-grained motion (lip sync, facial micro-expression, gesture cadence) (Wan-S2V: Audio-Driven Cinematic Video Generation §1). HuMo adopts the same global-text / local-audio partitioning but extends to a true triplet (text + reference image + audio) rather than start-frame + audio (HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning §1).
• On head-to-head benchmark (project page Table 1), Wan2.2-S2V-14B leads on identity preservation (CSIM 0.677), hand keypoint confidence (HKC 0.435), FID (15.66), SSIM (0.734), PSNR (20.49) vs Hunyuan-Avatar, EMO2, EchoMimicV2, MimicMotion, FantasyTalking at comparable Sync-C (Wan-S2V: Audio-Driven Cinematic Video Generation Table 1).
• Long-form video is generated autoregressively by reusing the last several latent frames of the previous clip as conditioning at the next inference step — the same last-k-latent-frames recipe as Self-Forcing and Causal Forcing (Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (MultiTalk) §3.5, InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing Method). Alternative: InfinityHuman replaces the recycled RGB-latent buffer with a coarse audio-synchronized pose sequence as the temporal intermediate, plus the initial frame as a fixed visual anchor, on the argument that pose is appearance-decoupled and resists temporal degradation (InfinityHuman: Towards Long-Term Audio-Driven Human Animation).
• Audio encoders vary but are typically small frozen pretrained models with a custom adapter to match the 3D-VAE-compressed video latent frame rate: Wav2Vec2 / chinese-wav2vec2-base in the MultiTalk/InfiniteTalk line, with later iterations (LongCat-Video-Avatar 1.5, HuMo) moving to Whisper (InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing Method, HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning §3.3).
• The cluster’s identity-preservation lead over Hunyuan-Avatar comes specifically under highly dynamic motion, where Hunyuan-Avatar exhibits facial distortion and identity shifts — the Wan-family models maintain identity integrity in those conditions (Wan-S2V: Audio-Driven Cinematic Video Generation §4 qualitative).
• I2V vs T2V backbone is a fork in the cluster. MultiTalk / InfiniteTalk / TalkingMachines / Wan-S2V all start from an I2V backbone and require a subject-complete start frame, which constrains text editability (cannot re-edit the rendered subject via text prompts). HuMo starts from a T2V backbone and treats reference images as retrievable identity priors via VAE-latent concatenation at the end of the noisy video latent sequence (not start, to prevent start-frame misinterpretation), restoring text editability — e.g. generating a “clearly visible face” from a dimly-lit reference, or compositing reference images into a “temple” scene (HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning §1, §3.3, Fig. 6).
• Audio-region binding via mask-prediction supervision (soft) vs hard gating. Prior I2V-based methods restrict audio cross-attention to detected face regions in the start frame (hard gating). This breaks when the input is a full-frame face image (no localization is available before denoising). HuMo’s focus-by-predicting trains a small mask predictor M(·) attached after audio cross-attention in the last four DiT blocks, supervised by ground-truth face mask via size-aware-weighted BCE — a soft regularizer that biases audio attention toward facial regions without crippling full-body kinematics. Removing it drops Sync-C 6.252 → 5.946 (the largest single ablation drop) (HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning §3.3, Eq. 3, Table 3).

Recent contributions

• [2026-05-25] RealVideo: Real-time Streaming Conversational Video System (Z.ai): RealVideo (Z.ai) — system release, not a new model. Composes Wan2.2-S2V-14B + Self-Forcing distillation + GLM-4.5-AirX (LLM) + GLM-TTS into a WebSocket-streamed text-to-real-time-avatar pipeline. Reports ~306 ms per DiT block at sp=4 / 2 denoising steps on H100s (within the 500 ms/block real-time budget); minimum 2× 80 GB GPUs, one dedicated to the VAE. Confirms TalkingMachines’s recipe (single-character real-time on a Wan-family backbone via causal AR distillation) ports cleanly to the Wan2.2-S2V branch, and is the first deployable productization of the audio-driven-character-animation stack filed on the wiki.
• [2026-05-25] HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning: HuMo — first cluster entry to use a T2V (not I2V) backbone (Wan-2.1, 1.7B and 17B). Treats reference images as retrievable identity priors concatenated at the end of the video-latent sequence, not as a start frame, restoring text editability. Introduces (a) cross-paired retrieval of reference images (same identity, different attributes) to break the copy-paste failure mode; (b) progressive task-weighting curriculum (80/20 → 50/50 subject-preservation/audio-sync) instead of MultiTalk’s hybrid-batch training; (c) “focus-by-predicting” soft mask supervision in place of hard face-region gating; (d) time-adaptive CFG with separate (w_T, w_I, w_A) and a per-timestep configuration switch (text/image-dominant at t ∈ [1.0, 0.98], audio/image-dominant thereafter). Beats specialized SOTA on both subject preservation (vs Phantom, MAGREF, HunyuanCustom, Kling 1.6) and audio-visual sync (vs Hallo3, FantasyTalking, HunyuanCustom; Sync-C 6.252 vs OmniHuman-1’s 6.526) with a single face reference, not a subject-complete start frame.
• [2026-05-24] InfinityHuman: Towards Long-Term Audio-Driven Human Animation: InfinityHuman — coarse-to-fine framework that breaks the long-horizon drift problem differently from the rest of the cluster: instead of recycling RGB-latent overlap frames, it synthesizes a stable audio-driven pose sequence and refines it into high-resolution video with a pose-guided refiner anchored on the initial frame. The pose intermediate is decoupled from appearance and is claimed to suppress identity drift, color shift, and scene instability over extended durations.
• [2026-05-24] Wan-S2V: Audio-Driven Cinematic Video Generation: Wan-S2V — first audio-driven branch from the official Wan team, built on Wan2.2-14B (vs prior community recipes on Wan2.1). Explicitly frames the design around global-text / local-audio conditioning partitioning, targets cinematic output (camera, multi-character, full-body) not just talking-head, leads on identity (CSIM) and hand-keypoint (HKC) metrics, extends to long-form generation and lip-sync editing. Positioned as the first entry in HumanAIGC’s “Vida” research series.
• [2026-05-24] InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing: InfiniteTalk — extends MultiTalk to sparse-frame V2V dubbing. Same Wan2.1-I2V-14B + partial-parameter audio cross-attention, but reframes I2V → V2V by treating a handful of source frames as identity anchors. Supports infinite-length via autoregressive clip extension (last 9 latent frames). The same checkpoint handles I2V and V2V via conditioning differences.
• [2026-05-24] EchoMimicV3: 1.3B Parameters are All You Need for Unified Multi-Modal and Multi-Task Human Animation: EchoMimicV3 — 1.3B unified multi-modal/multi-task human animation. Demonstrates the recipe works below the Wan 14B scale, providing a smaller-footprint alternative.
• [2026-05-24] TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models: TalkingMachines — real-time single-character FaceTime-style avatar via 2-NFE sparse causal AR distillation of Wan2.1-I2V-14B. Introduces face-region attention masking, local ±5-latent-frame audio windowing, Speaking/Silence-mode embedding for turn-taking, Score-VAE disaggregation for interactive latency.
• [2026-05-24] Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (MultiTalk): MultiTalk — first multi-stream audio-to-person binding via L-RoPE. Sets the partial-parameter + multi-task training recipe (audio cross-attention + audio adapter unfrozen, I2V leg preserves prompt adherence) the cluster has converged on.

Open questions

• Closed-source comparison: how do Veo 3, Sora 2, and Kling 3.0 implement audio-driven character animation? The open-source convergence on partial-parameter Wan adaptation is striking, but closed-source recipes are unknown — they may use joint A+V training (closer to the Joint audio-video generation cluster) instead.
• Wan2.2 vs Wan2.1 migration: Wan-S2V is the first cluster entry to use Wan2.2-14B; MultiTalk / InfiniteTalk / TalkingMachines all use Wan2.1-I2V-14B. HuMo is on Wan-2.1 T2V. Open whether the Wan2.2 backbone changes the audio cross-attention recipe or whether the community will port to Wan2.2 without architectural change.
• I2V-fork vs T2V-fork: HuMo is the first T2V-based cluster entry. Head-to-head with the I2V-based MultiTalk / Wan-S2V at matched scale on a unified benchmark would clarify whether the editability gain from T2V is worth the loss of visual-quality priors I2V backbones inherit from start-frame conditioning. The MoCha numbers suggest yes on every metric except Sync-C, but the comparison was not at matched compute.
• Cinematic vs talking-head benchmark fragmentation: MultiTalk uses HDTF / CelebV-HQ / EMTD for talking-head; Wan-S2V introduces a custom benchmark; HuMo introduces an in-house 100-case S2V benchmark plus MoCha. No unified cinematic benchmark exists yet.
• L-RoPE × real-time: nobody has stacked MultiTalk’s L-RoPE multi-person binding with TalkingMachines’s 2-NFE causal AR distillation. Multi-person streaming is the obvious next paper. RealVideo’s stack (RealVideo: Real-time Streaming Conversational Video System (Z.ai)) is single-character only.
• Audio encoder choice: Wav2Vec2 vs Whisper-Large (LongCat-Video-Avatar 1.5, HuMo) vs custom — no head-to-head comparison at matched conditioning architecture.
• TTS-driven audio degradation: MultiTalk explicitly notes that synthetic (TTS) audio gives noticeably worse facial expression than real audio because all training audio is real (Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (MultiTalk) §5). Open whether this transfers to Wan-S2V / HuMo and how to close the real-vs-synthetic gap. RealVideo drives the video stage exclusively from GLM-TTS output, so this is the operational regime for at least one deployed product.
• Camera control disentanglement: Wan-S2V claims text handles camera moves and audio handles fine motion, but the paper doesn’t ablate camera-control quality vs a dedicated camera-controlled video diffusion model. HuMo adopts the same global-text / local-audio split. Whether the global-text channel is sufficient for cinematic camera work or whether explicit camera trajectory injection (cf. Camera-Controlled Video Diffusion) is still needed is open.
• Pose-intermediate vs latent-overlap for long-form: InfinityHuman bets on a pose-sequence intermediate, MultiTalk/InfiniteTalk on recycled RGB-latent windows. No head-to-head exists yet — open whether pose decoupling actually wins over latent overlap once both pipelines are tuned at matched compute.
• Soft mask supervision vs hard gating for audio-region binding: HuMo’s focus-by-predicting is the first soft alternative to hard face-region attention restriction (FantasyTalking, prior I2V-based methods). Whether soft mask supervision generalizes to non-face audio binding (e.g. hand-driven gesture, full-body music sync) is untested.

Papers

2025-12

• Wan 2.6 — native multimodal video model with audio, multi-shot storytelling, starring references (Alibaba Wan announcement) — Wan 2.6 — native multimodal video model with audio, multi-shot storytelling, starring references (Alibaba Wan announcement) (published 2025-12-16)
• JoVA: Unified Multimodal Learning for Joint Video-Audio Generation — JoVA: Unified Multimodal Learning for Joint Video-Audio Generation (published 2025-12-15)

2025-10

• GAGA-1 release — 'Holistic AI Actor' with co-generated voice, lipsync, and performance (Sand.ai / Gaga AI) — GAGA-1 release — ‘Holistic AI Actor’ with co-generated voice, lipsync, and performance (Sand.ai / Gaga AI) (published 2025-10-10)
• TalkCuts: A Large-Scale Dataset for Multi-Shot Human Speech Video Generation — TalkCuts: A Large-Scale Dataset for Multi-Shot Human Speech Video Generation (published 2025-10-08)

2025-09

• X-Streamer: Unified Human World Modeling with Audiovisual Interaction — X-Streamer: Unified Human World Modeling with Audiovisual Interaction (published 2025-09-25)
• Kling-Avatar: Grounding Multimodal Instructions for Cascaded Long-Duration Avatar Animation Synthesis — Kling-Avatar: Grounding Multimodal Instructions for Cascaded Long-Duration Avatar Animation Synthesis (published 2025-09-11)
• HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning — HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning (published 2025-09-10)
• UniVerse-1: Unified Audio-Video Generation via Stitching of Experts — UniVerse-1: Unified Audio-Video Generation via Stitching of Experts (published 2025-09-07)

2025-08

• InfinityHuman: Towards Long-Term Audio-Driven Human Animation — InfinityHuman: Towards Long-Term Audio-Driven Human Animation (published 2025-08-27)
• Wan-S2V: Audio-Driven Cinematic Video Generation — Wan-S2V: Audio-Driven Cinematic Video Generation (published 2025-08-26)
• InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing — InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing (published 2025-08-19)

2025-07

• EchoMimicV3: 1.3B Parameters are All You Need for Unified Multi-Modal and Multi-Task Human Animation — EchoMimicV3: 1.3B Parameters are All You Need for Unified Multi-Modal and Multi-Task Human Animation (published 2025-07-05)

2025-06

• TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models — TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models (published 2025-06-03)

2025-05

• Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (MultiTalk) — Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (MultiTalk) (published 2025-05-28)

Unknown date

• RealVideo: Real-time Streaming Conversational Video System (Z.ai) — RealVideo: Real-time Streaming Conversational Video System (Z.ai) (published unknown)