Wan-Animate: Unified Character Animation and Replacement with Holistic Replication

TL;DR

Wan-Animate is a unified character animation + character replacement framework built on the Wan-I2V backbone (Wan 2.2 in the productized release). Given a character image and a driving reference video, it can either animate the character to replicate the driver’s body motion and facial expressions, or splice the animated character into the driver’s video while matching its lighting and color tone via an auxiliary Relighting LoRA. The control signal is decoupled: spatially-aligned 2D skeleton maps for body motion (added to noise latents) and implicit facial features extracted from source frames for expression. Reported as SOTA against open-source baselines (Animate Anyone, UniAnimate, VACE) on SSIM/LPIPS/FVD and competitive with closed-source DreamActor-M1 and Runway Act-Two via user study.

Key claims

A single Wan-I2V-based model unifies “animation” (drive a static character image with a reference video) and “replacement” (swap the character in a reference video while preserving environment) by varying an input paradigm that distinguishes reference-condition regions from generation regions [§1, §3].
The input paradigm preserves Wan-I2V’s original latent-input structure, which the authors argue minimizes distributional shift during post-training [§3].
Control signals are decoupled into body motion and facial expression: a 2D skeleton image (spatially aligned) is added to the initial noise latents, while expression is conditioned via implicit features extracted from source frames [§3, §3.2].
A Relighting LoRA is the only mode-specific module (active in replacement mode) and preserves the character’s appearance while reapplying the reference video’s lighting and color tone [Abstract, §3].
On a self-reconstruction quantitative benchmark (SSIM / LPIPS / FVD) Wan-Animate outperforms mainstream open-source character-animation frameworks (Animate Anyone, UniAnimate, VACE) and a portrait-only subset is reported separately against face-specialized baselines [Table 6, §4].
A 20-participant user study against the closed-source SOTA (DreamActor-M1, Runway Act-Two) using cross-ID animation pairs reports preference for Wan-Animate on a composite of generation quality, identity consistency, motion accuracy, and expression accuracy [§4].
The authors commit to open-sourcing weights and code [§1]; Wan2.2-Animate-14B is on Hugging Face.

Method

Wan-Animate post-trains the Wan-I2V latent video DiT to take a character image plus a driving reference video and produce a video in either animation or replacement mode. The single architectural change is the input paradigm: tokens carry an additional region flag that distinguishes (a) reference-condition latents (the character image, plus — in replacement mode — the masked driving video frames) from (b) latents that the model must generate. This region flag is what lets the same forward pass cover reference-image injection, temporal-frame guidance (so the generated clip extends naturally from a reference), and the animation-vs-replacement mode switch.

Holistic control is decomposed into two channels. Body motion uses a skeleton-based representation: a per-frame 2D skeleton image rendered from the driving video, balancing accuracy and cross-identity generality. Because the skeleton is spatially aligned with the target frame, it’s injected by simply being added to the initial noise latents (no extra adapter is needed for body motion). Expression is handled separately, via implicit facial features extracted from source frames of the driver — these are injected through the model’s existing conditioning pathways rather than as a spatial map. In replacement mode, a Relighting LoRA on top of the same backbone re-tones the synthesized character to match the reference video’s lighting and color, without otherwise altering the character’s appearance.

For replacement mode specifically, preprocessing pipeline (per the released code) extracts skeleton, character mask, and a background-inpainted reference video before invoking the DiT; optional Flux-2 Edit identity enhancement is layered on top in the fal hosted version.

Results

Quantitative (self-reconstruction benchmark, SSIM / LPIPS / FVD): Wan-Animate is reported as outperforming Animate Anyone, UniAnimate, and VACE on the authors’ curated test set [Table 6, §4].
Portrait-only subset: separate comparison against specialized facial-animation methods is reported in the same table [§4].
User study (20 participants, anonymized side-by-side, cross-ID animation pairs): preferred over DreamActor-M1 and Runway Act-Two on a composite of video quality, identity consistency, motion accuracy, expression accuracy [§4].
Public secondary reporting (third-party Medium write-up and product pages) characterizes Wan2.2-Animate as the first open-source release that simultaneously handles body, face, and environmental integration at quality close to proprietary systems; FVD-leadership on Real East Dance Val has subsequently been contested by Steady Dancer (Nov 2025).
Productized in May 2026 as “Wan Motion” on fal.ai with optimization for inference latency, built-in motion retargeting (handles different body proportions between driver and character), and optional Flux-2 Edit identity sharpening.

Why it’s interesting

The “frozen Wan + lightweight adapter” recipe is now the de-facto pattern across this team’s filed Camera-Controlled Video Diffusion cluster (SpaceTimePilot, NeoVerse, VerseCrafter, OmniTransfer — all on top of Wan 2.1). Wan-Animate is the same backbone family taken in a different direction: instead of geometric trajectory control, it post-trains for character-level control (skeleton + implicit face features) and unifies a generation-task switch (animation vs replacement) via region-flagged input tokens — the same conceptual move OmniTransfer makes with RoPE offsets to switch between appearance- and temporal-transfer tasks (OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer §4.2). It is also concrete evidence that Wan-family DiTs absorb skeleton conditioning by additive injection into noise latents without architectural surgery — relevant for any internal pipeline that would want to graft a 2D control map onto a Wan-style backbone. The May 2026 productization on fal (Wan Motion + Flux-2 Edit identity sharpening) is the first instance in this cluster of an open-source Wan-adapter recipe shipping as a hosted commercial endpoint with a non-trivial inference-side optimization layer.