EchoMimicV3: 1.3B Parameters are All You Need for Unified Multi-Modal and Multi-Task Human Animation

TL;DR

EchoMimicV3 (Ant Group, AAAI 2026) is a 1.3B-parameter unified human-animation model — text/image/audio conditioned, multi-task (T2V / I2V / FLF2V / lip-sync), built by adapting Wan2.1-Fun-1.3B with three primary additions: (1) a “Soup-of-Tasks” reformulation that treats every animation task as a spatiotemporal masked-reconstruction problem with task-specific 0-1 mask patterns, trained hard-to-easy with EMA-based implicit task mixing; (2) a “Soup-of-Modals” Coupled-Decoupled Multi-Modal Cross Attention (CDCA) module — shared query MLP across modalities, modality-specific K/V projections — fused by a Multi-Modal timestep Phase-aware Dynamic Allocation (PhDA) that weights audio/image/text experts by diffusion phase; (3) a Negative-DPO-SFT cycle training strategy plus Phase-aware Negative-enhanced CFG (PNG) and Long Video CFG at inference. Reports competitive results vs 14B baselines (FantasyTalk, HunyuanAvatar) at ~10× smaller parameter count and 4-minute end-to-end generation for 5 s talking-human video.

Key claims

A single 1.3B-parameter video diffusion model can be made to handle T2V, I2V, First-Last-Frame-to-Video, and lip-sync within one architecture by recasting each task as a spatiotemporal masked reconstruction with task-specific 0-1 mask patterns on the latent input [§3.2].
A counter-intuitive hard-to-easy task schedule — start with the highest-mask-ratio task (I2V/FLF2V) closest to the pretraining distribution, then fold in simpler tasks (lip-sync) — beats the easy-to-hard curriculum and uniform random sampling on Sync-C, HA, and identity preservation [§3.2, Table 2 rows “Task Schedule A/B”].
An inter-task Exponential Moving Average (EMA) implicit task-mixing recipe avoids catastrophic forgetting across tasks; ablating EMA degrades performance, particularly on motion (Sync-C 5.32 vs 5.49, HA 0.90 vs 0.95) [Table 2 row “w/o EMA”].
A Coupled-Decoupled Multi-Modal Cross Attention module that shares one query MLP across audio/text/image experts (Coupled) while keeping modality-specific K/V projections (Decoupled) is the core multi-modal injection block; modal expert outputs are summed with timestep-dependent weights $\beta_m(t)$ [Eq. 1, §3.3 “CDCA module”].
Modal Importance varies by diffusion phase: text is roughly uniformly important across phases, image is most important in early/middle phases, audio is most important in the initial phase — formalized as the Multi-Modal PhDA piecewise-linear timestep schedule [Eq. 2, §3.3].
Deviating from the modality-phase correspondence causes specific failures: late-phase audio allocation breaks lip-sync, missing text in early/middle phases causes motion collapse, missing image in middle phases impairs identity preservation [Table 2 rows “Modals Allocation A/B/C”].
Audio injection uses a segment-wise center-extended alignment of Wav2Vec features to latent frames, with the audio modal-expert’s output modulated by a binary facial-region attention mask to focus lip-sync on the mouth region [§3.3 “Segments-wise audio-frame alignment”].
Negative DPO drops the paired-sample requirement of standard DPO — it only minimizes the generation probability of negative samples collected from intermediate SFT checkpoints, with user-annotated issue labels and no positive pair [Eq. 3, §3.4].
Interleaving Negative DPO with SFT in a DPO-SFT cycle dynamically mitigates staged issues (identity-preservation, color-shift) sequentially; SFT-only and SFT-paired-DPO both underperform the NDPO-SFT cycle, with SFT+DPO showing identity-consistency issues from pair scarcity [Table 2 rows “SFT only”, “SFT+DPO”].
Phase-aware Negative CFG (PNG) is a novel inference-time CFG variant: weight motion-related negative prompts in the early diffusion phase and detail-related negative prompts in the later phase, suppressing artifacts characteristic of each phase. Removing PNG drops HA from 0.95 → 0.89 [Table 2 row “w/o PNG”].
Long Video CFG addresses the sliding-window overlap problem in long-video inference by carefully smoothing the CFG noise prediction within overlap frames — computing a per-frame weighted average across consecutive sliding windows rather than naive frame stitching [Eq. 4-5, §3.5]. Removing Long Video CFG raises FVD 530 → 497 (worse) and degrades IQA / ASE [Table 2 row “w/o Long Video CFG”].
On the authors’ 300-video evaluation, EchoMimicV3 (1.3B) ties or beats EchoMimicV2 (0.8B), Hallo3, FantasyTalk-14B, HunyuanAvatar-14B, and OmniAvatar-1.3B across Sync-C, FID, FVD, IQA, ASE, ID, HA, HC, BC [Table 1].
Inference efficiency: talking-head generation in 5 diffusion steps; talking-body in 25 steps with TeaCache, ~4 min for 5 s of video — vs HunyuanAvatar-14B (50 steps, 17 min) and FantasyTalk-14B (25 steps, 18 min) [Table 1 columns “Head Steps/Decay”, “Human Steps/Decay”].
The released EchoMimicV3-Flash variant (post-paper, January 2026 update) runs at 8 steps, 12 GB VRAM, up to 768×768, no face-mask required [README §Updates].

Method

The backbone is Wan2.1-FUN-inp-480p-1.3B — a video DiT with a 0-1 masked-sequence-as-input interface. EchoMimicV3 reuses this interface for the Soup-of-Tasks trick: each task gets a distinct 0-1 mask pattern (T2V = all-mask, I2V = first-frame-clean, FLF2V = first+last-frame-clean, lip-sync = mouth-region-mask), so all tasks share the same architecture and the same mask-input slot. Training proceeds hard→easy by mask ratio, with an EMA-based implicit mixture so that adding a new (lower-mask-ratio) task doesn’t overwrite the previous task’s capability.

Soup-of-Modals injects audio/text/image conditioning via the CDCA module, inserted at every DiT block. The three modalities are encoded by umT5 (text), an audio extractor (audio), and CLIP (image). CDCA computes one shared query from the video latents and three modality-specific K/V pairs; the three expert outputs are summed weighted by $\beta_{\text{text}}(t), \beta_{\text{audio}}(t), \beta_{\text{image}}(t)$ from the Multi-Modal PhDA. PhDA is a hand-designed piecewise-linear schedule encoding the Phase-specific Modal Importance Law: audio peaks early, image peaks early/middle, text is roughly constant.

Audio is segmented and aligned frame-wise — each video latent frame receives ±N audio embeddings centered on its corresponding audio segment, with the audio modal-expert output multiplied by a binary face-region attention mask so audio influence is confined to mouth region.

The training strategy is a Negative-DPO-SFT cycle. Sampled intermediate SFT checkpoints generate videos; humans annotate which generations exhibit the current staged issue (identity drift, color shift, etc.); these become unpaired negative samples. Stage-wise Negative DPO minimizes only the probability of these negative samples (no positive pair), interleaved with SFT (flow matching) to recover positive quality.

Inference combines PNG and Long Video CFG. PNG weights different negative prompts at different diffusion phases — motion-related negatives early, detail-related negatives late — based on the empirical observation that PhD-Loss-style phase analysis predicts which artifact types arise at which timestep. Long Video CFG modifies the CFG noise prediction within the overlap region between consecutive sliding windows in long-video extension: instead of a naive frame stitch, the overlap-frame CFG is a per-frame weighted average across the two windows’ predictions (Eqs. 4-5), smoothing cross-window color shifts and identity drift.

Results

Headline (Table 1, 300-video internal eval): EchoMimicV3-1.3B — Sync-C 5.49, Sync-D 9.67, FID 42.45, FVD 496.76, IQA 4.91, ASE 3.77, ID 1.0, HA 0.95, HC 0.99, BC 0.97; talking-head at 5 steps / 1 min, talking-human at 25 steps / 4 min.
vs FantasyTalk-14B: EchoMimicV3 beats on Sync-C (5.49 vs 4.05), Sync-D (9.67 vs 11.01), FVD (497 vs 604), ID (1.0 vs 0.96), HA (0.95 vs 0.90), at ~10× fewer parameters and ~4× faster inference.
vs HunyuanAvatar-14B: Closer comparison — HunyuanAvatar leads on Sync-C (6.12 vs 5.49) and Sync-D (9.11 vs 9.67); EchoMimicV3 ties on FID (42.54 vs 42.45) and HC (1.0 vs 0.99); EchoMimicV3 beats on FVD (497 vs 676), HA (0.95 vs 0.85), at 14× fewer parameters and ~4× faster.
vs EchoMimicV2-0.8B (same family): Improvements across the board — FVD 497 vs 544, IQA 4.91 vs 4.81, ASE 3.77 vs 3.34, HA 0.95 vs 0.90 — at marginally larger parameter count.
vs OmniAvatar-1.3B (matched size): EchoMimicV3 beats on FID (42 vs 53), FVD (497 vs 705), all consistency metrics — same parameter count, ~half the steps.
Ablations (Table 2): All five components contribute. Removing EMA → Sync-C 5.32 (vs 5.49). Easy-to-hard schedule → Sync-C 5.40. Random schedule → 4.98. Modal-allocation deviations → Sync-C drops to 4.76 (Modal A omits audio early) or HA drops to 0.81 (Modal B omits text early). SFT-only training → Sync-C 4.94, FVD 540 — substantially worse than the NDPO-SFT cycle. Removing PNG → HA 0.89 (vs 0.95). Removing Long Video CFG → FVD 530, IQA 4.77 (vs 497, 4.91).
Multi-task qualitative: Single model handles LC (lip-sync), IA2V (Image-Audio-to-Video), FLFA2V (First-Last-Frame-Audio-to-Video) — none of which can be done by single-task SOTA baselines at this parameter scale [Fig. 5].
Post-paper Flash variant (Jan 2026): 8 inference steps, 12 GB VRAM, up to 768×768.

Why it’s interesting

Three contributions stand out from a CFG-research perspective and from the audio-driven-avatar lineage on the wiki.

First, Phase-aware Negative CFG (PNG) is a novel axis in the Classifier-Free Guidance variants cluster: instead of redesigning the unconditional branch (SparkVSR’s SparkVSR: Interactive Video Super-Resolution via Sparse Keyframe Propagation), the noise schedule (Ostris Random noise augmentation during diffusion training eliminates the need for CFG), the conditioning content (ALG Improving Motion in Image-to-Video Models via Adaptive Low-Pass Guidance), or the extrapolation space (NAG Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models), PNG schedules the negative prompt’s weight by diffusion phase — motion-negatives early, detail-negatives late — based on a phase-specific artifact taxonomy. The PhD-Loss timestep-phase analysis (originally from EchoMimicV2) is the same machinery that drives Multi-Modal PhDA, suggesting a broader pattern: diffusion-timestep phases predict both which modality matters and which negative artifacts arise. Long Video CFG is a second variant — CFG-over-overlap-frames in autoregressive long-video extension.

Second, this is the third audio-driven human-animation paper on the wiki alongside TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models and Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (MultiTalk). The three converge on a shared recipe — Wan2.1-family backbone + new cross-attention block for audio injection + face-region attention masking — but differ on the orthogonal axis they optimize: TalkingMachines optimizes real-time streaming via sparse-causal distillation + Score-VAE disaggregation; MultiTalk optimizes multi-stream binding via L-RoPE; EchoMimicV3 optimizes parameter efficiency + multi-task unification at fixed quality (1.3B handles four task types vs the others’ single specialization). Stacking these — EchoMimicV3’s CDCA + PhDA inside MultiTalk’s L-RoPE binding, distilled à la TalkingMachines for real-time — would be the obvious open follow-up.

Third, the Negative-DPO recipe is a notable simplification of DPO for the generative-video preference regime where high-quality positive pairs are expensive. The ablation (SFT+DPO underperforms NDPO-SFT due to pair scarcity) is the cleanest evidence on the wiki for the unpaired-DPO direction.