Kling-MotionControl Technical Report

TL;DR

Kling-MotionControl is Kuaishou’s unified DiT-based holistic character animation system: drive a reference character image with a driving video so that body, face, and hands are all retargeted simultaneously. The system applies a “divide-and-conquer” strategy with separate motion representations tailored to body / face / hand granularities, plus identity-agnostic geometric motion learning and an explicit semantic-motion module, a subject library for multi-reference identity preservation, 3D-aware control enabling free-view cinematic camera moves via text, and a multi-stage CFG-merged distillation pipeline that yields >10× inference speedup. Human GSB evaluations report it preferred over Dreamina, Runway Act-Two, and Wan-Animate across overall quality, visual quality, dynamic quality, ID preservation, motion accuracy, and expression accuracy.

Key claims

A single DiT framework orchestrates heterogeneous, region-tailored motion representations (body / face / hand) trained via a progressive multi-stage strategy, claiming to reconcile large-scale body stability with fine-grained facial/finger detail in one model [§1, Abstract].
Identity-agnostic motion learning decouples motion patterns from the driving subject’s physical attributes at the geometric level, complemented by a semantic motion module that captures action-level intent (e.g. “facepalm”, “clapping”), enabling cross-morphology retargeting (adult→child, human→animal) without manual calibration [§1].
A subject library mechanism accepts multi-view images or extra video clips of the target character as additional reference material, on top of standard single-image reference, to stabilize identity under extreme poses and long-duration generation [§1].
The motion representations are endowed with 3D awareness through large-scale multi-view supervision, enabling text-driven cinematic camera control (pans, zooms, free-view rendering) while maintaining geometric consistency [§1].
A Prompt Enhancer module bridges motion control and textual guidance so users can manipulate scene elements, clothing, background, and camera moves via text alongside the driving-video motion signal [§1].
Inference acceleration combines a dual-branch sampling teacher for multi-conditional CFG, multi-stage distillation that reduces NFE, and CFG-merging that bakes conditional gradients into the student, yielding >10× end-to-end speedup while preserving quality [§1, §2.2].
On a 150-case benchmark with pairwise human GSB evaluation, Kling-MotionControl is preferred over Dreamina (Overall 3.44), Runway Act-Two (Overall 16.25), and Wan-Animate (Overall 4.00) using the (G+S)/(B+S) ratio at 1080p with aligned text prompts [Table 1, §2.2].
Qualitative cases report Wan-Animate “completely fails to reproduce rapid and challenging dynamics” with “severe appearance degradation and significant global color drift” under large-amplitude motion, while Dreamina exhibits limb-proportion drift and Runway Act-Two suffers catastrophic failure on complex hand poses [§2.2].
The data pipeline is multi-dimensional filtered (quality, motion amplitude/fluency, subject consistency) and augmented with high-quality rendering data and high-speed-camera footage for fast/complex motion, plus fine-grained annotation of actions, micro-expressions, human-object interactions, and camera moves [§1].

Method

Kling-MotionControl is a DiT-based latent video diffusion model that takes a reference character image, a driving video, optional subject-library references, and an (optionally enhanced) text prompt, and produces a holistic animation. The core architectural commitment is the divide-and-conquer motion encoder: rather than a single skeleton signal (as in Wan-Animate: Unified Character Animation and Replacement with Holistic Replication‘s 2D skeleton map), distinct motion representations are designed for body, face, and hand regions and harmonized through a progressive multi-stage training strategy. Identity disentanglement is two-pronged: a geometric track strips driving-subject physical attributes from the motion signal, and a semantic track tags the high-level action (e.g. “facepalm”) so retargeting across morphology stays semantically faithful. Identity preservation is handled by a separate identity encoding+fusion pathway, and the subject library mechanism lets the model construct a richer identity prior from extra reference material when available.

3D awareness is injected via large-scale multi-view supervision on the motion representations, which is also what enables the text-controlled free-view cinematic camera moves. Inference acceleration is done in three stacked moves: (1) a dual-branch teacher sampling strategy that handles multi-conditional CFG without paying for multiple inference branches, (2) multi-stage distillation to a few-step student that reduces NFE, and (3) merging conditional gradients into the student to remove CFG overhead — together giving >10× speedup [§1].

Results

GSB pairwise human evaluation against three baselines on a 150-case 1080p benchmark, reported as (G+S)/(B+S) ratios with majority-vote G/S/B labels: vs Dreamina — Overall 3.44 / Visual 3.33 / Dynamic 1.92 / ID 1.56 / Motion 1.05 / Expression 1.20; vs Runway Act-Two — Overall 16.25 / Visual 8.00 / Dynamic 4.64 / ID 2.95 / Motion 3.32 / Expression 4.50; vs Wan-Animate — Overall 4.00 / Visual 6.43 / Dynamic 1.77 / ID 3.07 / Motion 1.34 / Expression 1.16 [Table 1, §2.2].
All five granular dimensions (visual / dynamic / ID / motion / expression) plus Overall favor Kling-MotionControl over every baseline, with the largest absolute margins on Visual Quality and Overall Preference [§2.2].
Qualitative claims (no numbers): Kling-MotionControl handles extreme emotions and intricate hand interactions where Dreamina and Wan-Animate produce artifacts; cross-identity transfer with significant morphological discrepancy (child→adult, realistic→cartoon) maintains reference appearance; large-amplitude / rapid motion is reported as the failure mode of all three baselines but handled by Kling-MotionControl [§2.2].
10× inference acceleration via the dual-branch + multi-stage distillation + CFG-merging stack, without reported quality loss [§1].

Why it’s interesting

This is the direct heir to Wan-Animate: Unified Character Animation and Replacement with Holistic Replication in the open-vs-closed holistic-character-animation race, and the paper explicitly positions Wan-Animate as the strongest open-source comparison while reporting a GSB margin on it across every dimension (Overall 4.00, Expression 1.16, ID 3.07). The architectural contrast with Wan-Animate is the most concrete bit for the team: where Wan-Animate decouples control into just two channels (body skeleton additively injected into noise latents + implicit facial features through the standard conditioning path), Kling-MotionControl runs a three-way divide-and-conquer (body / face / hand) with region-specific representations and adds an explicit semantic-motion channel on top of the geometric one — closer to the multi-channel decoupling pattern in the Camera-Controlled Video Diffusion cluster (VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control background + 3D-Gaussian object trajectories; SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time camera + animation-time embedder). The 3D-aware motion representations + text-driven cinematic camera control is also the first character-animation system filed that explicitly composes character motion with camera trajectory control on the same backbone — directly relevant to the open question on the camera-controlled concept page about whether VerseCrafter’s GeoAdapter, SpaceTimePilot’s E_ani, and Wan-Animate’s skeleton head can be combined.