Kling-Avatar: Grounding Multimodal Instructions for Cascaded Long-Duration Avatar Animation Synthesis

TL;DR

Kling-Avatar (Kuaishou Kling team) is a two-stage cascaded framework for audio-driven portrait animation. Stage one uses an MLLM Director (Qwen2.5-Omni for audio captioning, Qwen2.5-VL for image captioning, plus a third-stage prompt assembler with three-shot in-context learning) to fuse the image, audio, and text prompt into a single coherent storyline that conditions a “blueprint” video. Stage two evenly segments the blueprint, picks expressive anchor keyframes, and generates sub-clips in parallel via first-last-frame conditioning — making long-duration generation embarrassingly parallel. It also introduces a negative-frame CFG mechanism that uses a manually corrupted reference image (texture distortion, blur, exaggerated contrast/saturation, color shifts) as the negative signal to counter identity drift. Reported output: up to 1080p at 48 fps, beats OmniHuman-1 and HeyGen on a 375-sample human-preference benchmark.

Key claims

The framework is a two-stage cascade: an MLLM Director that grounds multimodal instructions into a unified storyline, followed by parallel sub-clip synthesis from blueprint anchor keyframes [§1, §2, Fig. 3].
The MLLM Director consolidates audio captions (from Qwen2.5-Omni, including transcription and emotion) plus image captions (from Qwen2.5-VL) with the user prompt, prioritised by user-knowledge > audio > image, into a storyline covering character features, background layout, actions, visual style, camera planning, and emotional shifts; the storyline template is specified via three-shot in-context learning [§2.1].
Sub-clip parallelisation: blueprint video is evenly segmented; around each segmentation point an anchor frame is selected for identity, motion, occlusion, and expression quality; adjacent sub-clips are generated independently via first-last-frame conditioning, making total runtime nearly independent of the anchor count [§2.2].
Audio-conditioned interpolation between anchors avoids misalignment between selected anchor frames and actual speech timing, enabling seamless transitions [§2.2].
Four expert-model filters define training-data quality: lip-clarity (binary discriminator trained on synthetically perturbed mouth regions), temporal-continuity (manually assembled cross-segment negatives + PySceneDetect), audio-visual sync (SyncNet threshold), and aesthetic scoring [§2.3].
Benchmark: 375 image–audio–prompt triples — 340 human + 35 non-human (cartoon/anime/animal) references; audio in Chinese (150), English (150), Korean (35), Japanese (40); 8s–2min clips; explicit prompt annotations for emotion (calm/excitement/confusion/sadness/surprise/anger × intensity), camera moves, and gestures [§2.3].
Training strategy: sliding-window audio cross-attention (each video token sees only a small temporal window of audio tokens with padding), DWPose-derived mouth-region loss weighting, random empty-pixel padding to shrink face proportion (robustness to small/long shots), and frozen text cross-attention layer to preserve base T2V controllability [§2.4].
Negative-frame Classifier-Free Guidance: the reference image is manually corrupted to match observed identity-drift artefacts (texture distortion, blurring, exaggerated contrast/saturation, color shifts), and the degraded image is used as a negative CFG signal to steer denoising toward identity consistency [§2.4].
Inference-time audio cross-attention values are scaled up (no mouth-region mask available without ground truth) to maintain lip-audio alignment [§2.4].
GSB (Good/Same/Bad) overall win/loss ratio: 2.39 vs OmniHuman-1, 1.37 vs HeyGen across the full 375-sample benchmark [Table 1]. On Chinese speech the gap widens (4.53 vs OmniHuman-1); on bilingual singing 2.69 vs OmniHuman-1 and 2.90 vs HeyGen [Table 1].
HeyGen’s stability comes from looping a 5-second action pattern, which improves identity but harms vividness; Kling-Avatar supports arbitrary input/output resolutions vs HeyGen’s fixed horizontal/vertical crop [§3.2].
Output resolution range: 480p–1080p training resolutions, up to 1080p / 48 fps inference [§3.1].
Architecture: pretrained Video DiT extended with an audio cross-attention layer; Whisper audio encoder + T5 text encoder; AdamW at lr 1e-5 [§3.1].

Method

The pipeline has two stages tied together by an MLLM Director. In stage one, Qwen2.5-Omni produces an audio caption (transcription + emotion) and Qwen2.5-VL produces an image caption; these plus the user prompt are fed to the Director, which uses three-shot in-context learning to emit a storyline covering character traits, background, motion, style, camera, and emotion. The storyline is injected through the text cross-attention layer of a Video DiT to generate a blueprint video. In stage two, the blueprint is uniformly segmented; for each segment boundary an anchor frame is chosen on identity / motion / non-occlusion / expression criteria; adjacent sub-clips are then generated in parallel using first-last-frame conditioning. The MLLM Director also breaks the global storyline into time-localised semantic plans that, together with time-aligned audio, condition each sub-clip. An audio-conditioned interpolation step synthesises transition frames so that anchor selection does not desynchronise lips from speech.

Training adds an audio cross-attention layer atop a pretrained Video DiT. Audio tokens (Whisper) attend to video tokens via a sliding window — each video token attends only to its temporally aligned audio tokens with small padding. DWPose locates the mouth region, whose denoising loss is weighted up. The text cross-attention layer is frozen so the base T2V prior is preserved. Inference uses a negative-frame CFG: the reference image is deliberately corrupted (color/blur/contrast distortion) and used as the negative CFG signal, steering generation away from the visual signature of identity drift.

Results

On a 375-sample human-preference (GSB) benchmark Kling-Avatar beats both OmniHuman-1 and HeyGen overall, with the largest margins on Chinese speech (4.53 vs OmniHuman-1) and bilingual singing (7.69 vs HeyGen on lip sync) [Table 1]. The model produces up to 1080p / 48 fps video with stable identity preservation, coherent visual quality, and rich character dynamics in long-form rollouts (one frame sampled per 10s shows preserved hand gestures, head movements, and background lighting changes) [Fig. 7, §3.2]. Generalisation extends to multi-person, cartoon, anime, and non-human characters [Fig. 6].

Why it’s interesting

Sits squarely in the Audio-Driven Character Animation cluster but breaks from its convergence point: rather than the now-canonical “freeze big I2V Wan + audio cross-attention” recipe used by Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (MultiTalk), InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing, TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models, and Wan-S2V: Audio-Driven Cinematic Video Generation, Kling-Avatar adds a planning stage on top — an MLLM Director that produces a coherent storyline before any video frames are generated. The motivating failure mode (sorrowful song with a smile, because audio and text are aligned only at the diffusion level) is the strongest argument yet that the cluster’s per-modality local-cue alignment is leaving semantic coherence on the table. Long-duration generation is handled orthogonally to the Context Length / Quality Trade-off in Video Generation cluster’s KV-cache and memory-vector recipes: instead of recycled latents (MultiTalk / InfiniteTalk), pose intermediates (InfinityHuman: Towards Long-Term Audio-Driven Human Animation), or partitioned caches (Context Forcing: Consistent Autoregressive Video Generation with Long Context, PackForcing: Short Video Training Suffices for Long Video Sampling and Long Context Inference), it parallelises first-last-frame sub-clip generation off blueprint keyframes — runtime roughly constant in video length, but the cost is that long-range coherence is mediated entirely through the blueprint plus the MLLM-emitted localised plans rather than through learned memory. The negative-frame CFG against deliberately corrupted reference images is also a fresh design point in Classifier-Free Guidance variants — most prior negative CFG signals are text or null embeddings, not artefact-shaped image corruptions.