ID-V2V: Identity-Preserving Video Restylization

TL;DR

ID-V2V (Netflix + Eyeline Labs, SIGGRAPH Asia 2026) formalizes identity-preserving video restylization: given a source performance video plus an edited first keyframe (any image-editor output — e.g. NanoBanana), propagate the keyframe’s scene, lighting, and style across the whole video while strictly preserving each subject’s face and performance (expressions, gaze, lip sync). The core trick is to sidestep the missing paired-data problem by decoupling edit-driven synthesis (keyframe + depth control) from identity preservation (relit-face pixels + face-normal maps), where a trained face relighting model synthesizes on-the-fly training pairs from any single video. The system is built on VACE / Wan 2.1 with a multi-ControlNet architecture summing per-control features into shared DiT blocks; weights (idv2v.pth) and an alternate depth-augmented variant are released on Hugging Face.

Key claims

Facial appearance and expression should remain invariant under video restylization; illumination is the only permissible variation — so identity preservation can be cast as a video relighting problem [§1, §3.2].
Paired restylization data is unobtainable at scale; a dedicated image-based face relighting model (SVD-based, trained on ~330K OLAT-relighting pairs from LuxPostFacto data) generates relit-face conditioning at training time, letting supervised training pairs be constructed from a single video [§3.2, §3.3].
Four complementary control signals: relit face regions + face normal maps (identity/geometry preservation, from DAViD) and edited first frame + depth sequence (edit propagation, from DepthAnything-V2) [§3.2, Fig. 2].
Multi-ControlNet extension of VACE reuses one VACE context block across all control signals and sums their features before injection into the main DiT blocks, keeping compute low relative to per-control blocks [§3.3, Fig. 3].
Training-time facial region uses relit faces to simulate arbitrary lighting; inference uses raw source-video face pixels directly, so no identity information is lost [§3.3].
On single-subject restylization ID-V2V improves AdaFace identity from 0.529 (WanAnimate best baseline) to 0.701 and EMOCA expression similarity from 0.841 to 0.899; user study prefers ID-V2V ~70–75% on facial likeness, performance preservation, and video quality [Table 1].
On two-subject restylization the gap widens: 85.6% / 86.1% / 81.1% user-study preference vs. VACE’s 9.8% / 10.0% / 12.4%, on facial likeness / performance / quality [Table 1].
Ablations show face-video pixels contribute more identity signal than face-normals but both are needed: dropping normals drops AdaFace 0.701 → 0.676, dropping the face video drops it to 0.574 [Table 1].
The trained condition is facial-region-restricted, but at inference the guided region can be expanded to full body (preserves body appearance) or full frame (enables full-scene relighting) without retraining [§3.3].

Method

The task inputs are a source video (any length) plus an edited first frame (typically produced by an image editor like Qwen-Image-Edit or NanoBanana); the output is a restylized video that inherits the keyframe’s scene / lighting / style but faithfully preserves each subject’s identity and performance. The model is a VACE-on-Wan2.1 backbone extended to multi-ControlNet: relit face pixels, face surface normals, edited keyframe, and depth sequence are all processed through a shared VACE context block, feature-summed, and injected into the main DiT blocks. For each face detected by SCRFD (up to 5 per frame), an image-based relighting model — an SVD-derived, HDR-map-conditioned architecture retrained on 330K OLAT-based (input, target) pairs where the target uses a half-white/half-black HDRI producing regular top-lighting — normalizes cinematic training-video illumination to the “regular lighting” that source videos exhibit at inference. Face normals come from DAViD; depth from DepthAnything-V2. All models train at 1280×720, 81 frames per clip. At inference the SVD relighting step is skipped and raw source facial pixels are used directly; longer sources are generated as overlapping 81-frame clips seeded by the tail of the previous clip. A separate “relighting” mode drops SAM3 masking and preprocessing entirely, feeding the raw source video as the sole control so only lighting is changed.

Results

Single-subject AdaFace identity: 0.701 vs. WanAnimate 0.529, SteadyDancer 0.479, FantasyPortrait 0.507, VACE 0.465, AnimateAnyone 0.402 [Table 1].
Single-subject expression similarity: Exp-AU 0.668, Exp-EMOCA 0.899 (best baseline 0.600 / 0.841) [Table 1].
Two-subject setting: AdaFace 0.637 vs. WanAnimate 0.432, VACE 0.401, FantasyPortrait 0.433; Exp-EMOCA 0.901 vs. best baseline 0.796 [Table 1].
User study (single-subject): 75.0% / 74.3% / 70.1% preference for facial likeness / performance / video-quality vs. WanAnimate 10.5% / 11.4% / 12.6% (next-best baseline) [Table 1].
User study (two-subject): 85.6% / 86.1% / 81.1% vs. VACE’s 9.8% / 10.0% / 12.4% [Table 1].
VBench aggregate quality is on par with strongest baselines (subject / background consistency, temporal flickering, human anatomy all within ~0.01 of the best row) — the win is in identity and performance, not general video fidelity [Table 1].
Two model variants ship: idv2v (1 control: foreground-on-gray) as default; idv2v_with_normal_depth (3 controls) as an alternate when the source depth already matches the target scene [README].

Why it’s interesting

ID-V2V is a mirror-image bet to Wan-Animate-2: Pushing the Application Boundaries of Character Animation and Wan-Animate: Unified Character Animation and Replacement with Holistic Replication: those systems keep the source identity (from a reference image) and swap in a new performance (from a driving video); ID-V2V keeps the source performance and swaps in a new look (from an edited keyframe). Both directions run on Wan-family backbones, but the recipes diverge sharply — Wan-Animate-2 argues for end-to-end DiTs with no intermediate representation, whereas ID-V2V argues that carefully-designed intermediate control signals (relit pixels + normals + depth) are exactly what lets you fabricate the paired training data that the real world doesn’t provide. The “cast identity preservation as relighting” reframe is the paper’s most portable idea: the same paired-data-from-a-single-video trick could plausibly extend to other identity-invariant edits (wardrobe, hairstyle) where the invariance can be modeled as a known image-to-image transform. It also complements the instruction-based video editing cluster (Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset (Ditto / Editto), InstructX: Towards Unified Visual Editing with MLLM Guidance) with a keyframe-driven — rather than instruction-driven — control interface that maps directly onto the “shoot first, restyle later” pipeline actual VFX and film production uses.