LPM 1.0: Video-based Character Performance Model

TL;DR

LPM 1.0 is a 17B-parameter Diffusion Transformer for single-person, full-duplex audio-visual conversational character performance — given a character image (with identity references), it generates speaking video from synthesized audio and listening video from user audio, with text prompts steering motion and emotion. The authors frame existing video-generation work as facing a “performance trilemma” (expressiveness vs. real-time inference vs. long-horizon identity stability) and address it system-wide: a curated multimodal dataset with explicit speak/listen/idle frame labels, a bidirectional Base LPM (14B pretrained I2V backbone + 3B interleaved speak/listen audio cross-attention), and a four-stage autoregressive distillation curriculum producing a causal backbone-refiner Online LPM for real-time infinite-length streaming. They also introduce LPM-Bench, the first benchmark targeting interactive conversational performance with multimodal inputs across 1,000 cases.

Key claims

Existing video models face a “performance trilemma” — expressiveness, real-time inference, and long-horizon identity stability cannot currently be jointly satisfied [§1].
Conversational performance requires modeling listening behavior, not just speaking; most prior speech-driven systems neglect attentive gaze, anticipation, and turn-taking that constitute half of natural dialogue [§1].
Single-image character conditioning is insufficient for production — unseen details (teeth, profile geometry, body appearance under novel viewpoints) get inferred frame-by-frame and fail to remain consistent over long horizons; identity-aware multi-reference extraction is required [§1, §2].
Frame-level audio state labeling (speak / listen / idle) via a fine-tuned LR-ASD backbone achieves 89.75% accuracy on Domain 1 and 87.63% on Domain 2 of held-out manually-annotated clips [§2.2, Table 1].
A fine-tuned Qwen3-Omni audio-video understanding model used for semantic verification of conversation clips beats Gemini 2.5 Pro by +7.90 F1 (78.37 vs 70.47) across five labels, with the largest gains on silence (+19.40) and speak (+10.67) [§2.2, Table 2].
Base LPM is a 17B bidirectional DiT — 14B pretrained image-to-video backbone + 3B added parameters as interleaved speak/listen audio cross-attention blocks — trained on tens of millions of clips and over 1.7 trillion multimodal tokens [§1, Contribution 3].
Online LPM is a causal streaming generator distilled from Base LPM via a four-stage autoregressive distillation curriculum into a backbone–refiner architecture: backbone maintains temporally coherent latent trajectories, refiner recovers high-fidelity detail [§1].
LPM-Bench has 1,000 test cases across five scenarios (speaking, listening, conversation, diverse human motion, character generalization), with diversity across 22 expression bases, 78 emotions, >5,000 motion descriptors, and audio durations from 5s to 1 hour (10% long-form) [§1].
In pairwise human evaluation, Base LPM (720P) is preferred over Kling-Avatar-2 by 64.3% and over OmniHuman-1.5 by 42.5% of raters, with the largest margins on identity consistency and motion dynamics [§1].
Online LPM (480P) is preferred over LiveAvatar by 82.5% and over SoulX by 64.1% of raters [§1].
At matched 480P resolution, raters judge Base LPM and Online LPM indistinguishable in 42–88% of cases across dimensions and scenarios — real-time causal generation does not appear to require sacrificing perceived realism [§1].

Method

Base LPM is built on a 14B pretrained image-to-video DiT (Wan-family backbone) extended with 3B parameters of interleaved speak/listen audio cross-attention blocks, yielding 17B total. The model takes a character image plus identity-aware multi-reference images, time-varying text directives (for expression / motion / environment control), and per-frame-separated speaking and listening audio streams. Training data comes from a multi-stage pipeline that combines: (1) shot detection and human-presence filtering on raw long-form video; (2) frame-level speak/listen/idle labeling via a fine-tuned LR-ASD model (trained on 20K manually annotated clips); (3) for multi-person clips, character-audio assignment producing speaker-only and listener-only audio variants; (4) semantic verification by a fine-tuned Qwen3-Omni audio-video understanding model that filters false-listening and not-in-dialogue-listening segments. Overall data retention is <10%.

Online LPM is then produced via a four-stage autoregressive distillation curriculum that progressively converts Base LPM’s bidirectional DiT into a causal backbone–refiner streaming architecture. The backbone carries the temporally coherent latent trajectory; the refiner recovers per-frame detail. This is intended to preserve identity, synchronization, and motion realism over unbounded horizons under causal, low-latency streaming constraints. LPM-Bench, the accompanying evaluation suite, provides 1,000 multimodal test cases with paired speaking/listening audio (5s–1h), structured text directives, high-resolution initial frames, and multi-reference images, designed for systematic diversity across appearance, performance, and audio axes.

Results

Frame-level audio state labeling: Domain 1 accuracy 89.75% / Macro-F1 89.75; Domain 2 accuracy 87.63% / Macro-F1 87.56 [Table 1]. Speak/listen recalls balanced on Domain 1 (91.62 / 87.99); Domain 2 shows asymmetry (94.05 / 81.05) attributed to ambient noise and off-screen speech — the failure mode the downstream semantic verifier is designed to absorb [§2.2].
Semantic verification (Qwen3-Omni fine-tuned vs Gemini 2.5 Pro baseline): overall F1 78.37 vs 70.47 (+7.90). Per-class: silence +19.40, speak +10.67, listening-dialogue +8.90, conversation -0.58, listening-nondialogue -7.99 [Table 2]. Silence-to-speak misclassification rate drops 24.4% → 9.0%; conversation-to-listen-dialogue leakage drops 12.7% → 4.8% [§2.2].
Base LPM (720P) vs Kling-Avatar-2: 64.3% preference; vs OmniHuman-1.5: 42.5% preference, with largest margins on identity consistency and motion dynamics [§1].
Online LPM (480P) vs LiveAvatar: 82.5% preference; vs SoulX: 64.1% preference [§1].
Base LPM vs Online LPM at matched 480P: rated indistinguishable in 42–88% of cases across dimensions and scenarios [§1].

Why it’s interesting

LPM 1.0 sits at the intersection of three active wiki threads. It is the first filed paper that specializes the joint-audio-video-generation dual-stream-DiT recipe (Joint audio-video generation, anchored by LTX-2: Efficient Joint Audio-Visual Foundation Model, MOVA: Towards Scalable and Synchronized Video-Audio Generation, SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model) to the conversational-character setting, adding a listening modality (listener-only audio variant + per-frame state labels) that none of LTX-2, MOVA, or SkyReels-V4 explicitly model. Its distillation pipeline echoes OmniForcing: Unleashing Real-time Joint Audio-Visual Generation‘s bidirectional→causal recipe for streaming joint A+V and the asymmetric-DMD lineage of Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation / Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generation — but at a much larger 17B scale and with a backbone-refiner factorization that is new to the wiki. The paper also doubles as a dataset-engineering recipe: the speak/listen/idle labeling pipeline and the semantic-verification stage are concrete tooling for any team building conversational video data, and the +7.90 F1 over Gemini 2.5 Pro on five-way labeling is a useful datapoint for VLM-as-evaluator pipelines.