Wan-S2V: Audio-Driven Cinematic Video Generation

TL;DR

Wan-S2V is an audio-driven, single-image-to-video model from HumanAIGC / Tongyi Lab (Alibaba) built on the Wan2.2 14B DiT backbone. It targets cinematic — not just talking-head — generation: nuanced character interaction, full-body motion, and dynamic camera work driven by an audio track plus a text prompt and a single reference image. The released artifact is the Wan2.2-S2V-14B model (weights, inference code, and tech report); the paper reports best-or-near-best scores against Hunyuan-Avatar, EMO2, EchoMimicV2, MimicMotion, and FantasyTalking on FID, FVD, SSIM, PSNR, EFID, HKC, HKV, and CSIM at comparable Sync-C. Extends to long-form video generation and precise lip-sync editing.

Key claims

Existing SOTA audio-driven character animators (Hunyuan-Avatar, Omnihuman, EMO2, EchoMimicV2, MimicMotion, FantasyTalking) perform well on speech/singing portraits but fall short on film-level requirements: multi-character interaction, realistic body movement, dynamic camera work [Abstract, §1].
Text is best used for global dynamics (camera moves, character trajectories, entity interactions); audio is best used for local fine-grained motion (lip sync, facial micro-expression, gesture cadence). Wan-S2V is explicitly designed around this division of labour [§1].
The model is built on Wan as the video foundation (the project page lists Wan2.2-S2V-14B as the released checkpoint), inheriting Wan’s DiT backbone and 16×16×4-compression VAE [Abstract, project README].
Wan-S2V achieves significantly enhanced expressiveness and fidelity in cinematic contexts vs the audio-driven character-animation prior art [Abstract, §1].
On the project-page benchmark, Wan2.2-S2V-14B reports FID 15.66, FVD 129.57, SSIM 0.734, PSNR 20.49, Sync-C 4.51, EFID 0.283, HKC 0.435, HKV 0.142, CSIM 0.677 — best on FID, SSIM, PSNR, CSIM (identity), and HKC; competitive on FVD and Sync-C [Table 1].
The method explicitly extends to long-form video generation and precise video lip-sync editing as two downstream applications of the same model [Abstract].
Wan-S2V is positioned as the first entry in HumanAIGC’s “Vida” research series, with planned follow-ups on advanced character control and dynamic dancing [§7 / acknowledgements].
Released under the Wan family: model weights, inference code, and the technical report dropped on 2025-08-26 [HF README “Aug 26, 2025”].

Method

The model takes three inputs: a single reference image (full-body or half-body), an audio track, and a text prompt describing scene-level dynamics. The video backbone is Wan2.2-14B; audio is encoded and injected as a conditioning signal into the DiT, with text retaining its role as the high-level director (camera, trajectories, interactions). The paper’s framing — text for global motion, audio for local fine-grained motion — explicitly carves up the conditioning channels rather than letting one signal dominate.

Data acquisition combines automated screening of large-scale public datasets (OpenHumanVid, Koala36M) with manual curation of high-quality samples focused on human characters speaking, singing, and dancing, producing millions of human-centric video samples. The paper explores model training and inference strategies tuned to cinematic settings — long-form rollouts and lip-sync editing in particular — but the details of the audio-injection module and the partial-parameter training recipe are in the PDF beyond what was retrievable at filing time.

The release also positions Wan-S2V as the foundation for downstream “Vida” series work on character control and dancing, suggesting the audio-injection mechanism is meant to compose with additional control modalities rather than stand alone.

Results

Headline benchmark (Table 1, project page): Wan2.2-S2V-14B vs Hunyuan-Avatar / EMO2 / FantasyTalking / EchoMimicV2 / MimicMotion across 9 metrics. Wan2.2-S2V leads on FID (15.66 vs Hunyuan-Avatar 18.07 next-best), FVD second (129.57 vs EMO2 129.41), SSIM 0.734 (best), PSNR 20.49 (best), CSIM 0.677 (best — identity preservation), HKC 0.435 (best — hand keypoint confidence). Sync-C is mid-pack at 4.51 (Hunyuan-Avatar 4.71 leads) and EFID 0.283 (EMO2 0.218 leads).
Identity preservation under dynamic motion: qualitatively, Hunyuan-Avatar exhibits facial distortion and identity shifts during highly dynamic movement; Wan-S2V maintains identity integrity, which the paper highlights as critical for cinematic applications [§4 qualitative].
Motion diversity: Wan-S2V generates a wider range of motion than Omnihuman, which tends to produce constrained outputs [§4 qualitative].
Long-form and lip-sync editing: the paper demonstrates extension to long-form video generation and precise lip-sync editing as additional applications, though no separate quantitative benchmark numbers for these modes are surfaced in the abstract or project page.

Why it’s interesting

Wan-S2V slots directly into the Audio-Driven Character Animation cluster: Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (MultiTalk) (MultiTalk, Wan2.1-I2V-14B + L-RoPE for multi-person), InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing (sparse-frame V2V dubbing on the same MultiTalk recipe), TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models (single-character real-time on Wan2.1-I2V-14B + causal distillation), and EchoMimicV3: 1.3B Parameters are All You Need for Unified Multi-Modal and Multi-Task Human Animation (1.3B unified multi-modal/multi-task human animator). Wan-S2V is the first of these to come from the Wan/Tongyi team itself rather than a community fine-tune, and the first to scale to the Wan2.2 14B backbone rather than Wan2.1, with a deliberate emphasis on cinematic rather than talking-head outputs (full-body + camera + interactions, not just lips). It’s also the first to explicitly partition the conditioning channels along the global-text / local-audio axis as a design principle.

Two things make this notable in context. First, the Wan2.2-S2V-14B release is the open weights drop the cluster needed: prior audio-driven Wan derivatives (MultiTalk, InfiniteTalk, TalkingMachines) were community recipes layered on top of Wan2.1; Wan-S2V is the official team’s own audio-driven branch on Wan2.2 and ships under the Wan license. Second, the benchmark table’s CSIM (identity) and HKC (hand keypoint) leads — at comparable Sync-C — are the dimensions the cluster previously struggled with (Hunyuan-Avatar’s identity drift, hand distortion in non-Wan-based methods), suggesting the cinematic framing translates to measurable quality gains beyond mouth-region sync.

The bigger structural read is that the Audio-Driven Character Animation cluster has now matured to the point where the Wan team itself is shipping in it — strong signal that the recipe (frozen-or-mostly-frozen Wan DiT + new audio cross-attention path + curated human-centric data) is the converged design for this task category.