InfinityHuman: Towards Long-Term Audio-Driven Human Animation

TL;DR

InfinityHuman is a coarse-to-fine audio-driven human animation framework targeting long-duration, high-resolution video generation without identity drift. It first synthesizes an audio-synchronized coarse representation (audio-driven pose sequences) and then progressively refines it into a high-resolution video using a pose-guided refiner anchored on the initial frame. Because pose is decoupled from appearance and resists temporal degradation, using pose as the intermediate buffer suppresses the color shifts, identity drift, and scene instability that plague existing overlap-frame extension methods.

Key claims

Existing audio-driven human animators extend video by recycling overlapping motion frames, which accumulates error and produces identity drift, color shifts, and scene instability over long durations [Project page §Abstract].
A coarse-to-fine pipeline using audio-synchronized pose sequences as a stable intermediate representation suppresses long-horizon drift because pose is decoupled from appearance [Project page §Method].
The pose-guided refiner uses the initial frame as a visual anchor and reconstructs long-term high-resolution video conditioned on the (stable) pose track, improving lip synchronization and identity preservation [Project page §Method].
The framework generates lifelike hand and body movement at high resolution over extended durations, beyond what overlap-frame extension achieves [Project page §Abstract].

Method

InfinityHuman is structured as two stages. Stage 1 generates an audio-synchronized pose sequence from the driving audio — a low-dimensional, appearance-decoupled intermediate that does not accumulate visual drift as the sequence lengthens. Stage 2 is a pose-guided refiner that consumes the stable pose track plus the initial reference frame as a visual anchor, and progressively reconstructs the long, high-resolution video. Because the visual anchor is fixed (the initial frame) rather than recycled from preceding generated frames, identity and color do not drift; because the pose intermediate is appearance-decoupled, it does not degrade temporally the way RGB-latent overlap frames do.

Results

The project page does not surface quantitative tables, but claims InfinityHuman preserves high-resolution visual consistency over extended durations while generating lifelike hand and body movements — positioned against overlap-frame extension methods that exhibit identity drift, color shifts, and scene instability [Project page §Abstract, §Method].

Why it’s interesting

The pose-as-stable-intermediate trick directly attacks the failure mode the rest of the audio-driven cluster fights with autoregressive last-k-latent recycling — see Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (MultiTalk) and InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing which extend long-form generation by reusing the last 9 latent frames of the previous clip, and which the Audio-Driven Character Animation open-questions section flags as drift-prone. InfinityHuman swaps the recycled RGB-latent buffer for a pose buffer plus a fixed reference-frame anchor, which is the same anchoring idea InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing uses for V2V dubbing but applied as the primary long-horizon mechanism. The two-stage audio→pose→video factorization also echoes the global-text / local-audio conditioning partition in Wan-S2V: Audio-Driven Cinematic Video Generation, where audio drives only fine-grained motion and a separate channel carries global structure.