HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning

TL;DR

HuMo is a unified human-centric video generation framework that conditions on text + reference images + audio simultaneously, rather than going through the standard two-stage T2I-then-I2V pipeline. It frames the imbalanced-controllability problem (turning up one modality damages another) as jointly a data problem and a training-paradigm problem, and proposes: (1) a multimodal data pipeline that retrieves cross-paired reference images with same identity but varied attributes plus lip-sync-filtered audio-video pairs; (2) a two-stage progressive training paradigm that first learns subject preservation via minimal-invasive image injection, then progressively adds audio-visual sync via a focus-by-predicting mask supervision; (3) a time-adaptive CFG that switches guidance configs across the denoising trajectory (text/image-dominant early, audio/image-dominant late). Built on Wan-2.1-1.3B and Wan-2.1-14B backbones, the 17B variant beats specialized SOTA on both subject preservation (vs Phantom, MAGREF, HunyuanCustom, Kling 1.6) and audio-visual sync (vs Hallo3, FantasyTalking, HunyuanCustom, OmniHuman-1) sub-tasks.

Key claims

The triplet-conditioned HCVG problem decomposes into two sub-tasks (subject preservation, audio-visual sync) whose naive joint training causes one ability to degrade the other; the dataset is the missing prerequisite because paired (text, reference image, synced audio, video) data does not exist publicly [§1, §3.2].
Reference images are retrieved by cross-pairing: for each video subject, retrieve images of the same identity/semantic category but with deliberately different visual attributes (makeup, pose, background, clothing, age for people; color, shape, viewpoint for objects). This prevents the “copy-paste” failure mode where the model just reproduces the reference frame [§3.2, Fig. 3].
The Stage-1 dataset has ~M video samples with text + reference images; Stage-2 adds ~K samples with lip-sync-aligned audio (audio data exists predominantly in human-centric content) [§3.2].
Subject preservation is implemented as minimal-invasive image injection: concatenate reference-image VAE latents at the end of the noisy video latent sequence (not the start, to prevent misinterpretation as a start frame), and update only the self-attention layers of the DiT backbone — leaving text-cross-attention frozen to preserve text following [§3.3].
Audio is incorporated via per-block audio cross-attention layers with Whisper features, where each video frame attends to a temporal window of audio embeddings centered at that frame [§3.3, Eq. 2].
The “focus-by-predicting” strategy: a small mask predictor M(·) attached after audio cross-attention in the last four DiT blocks predicts the facial region distribution, supervised by ground-truth face mask via size-aware-weighted BCE. This acts as a soft regularizer that biases audio attention toward facial regions without hard-gating [§3.3, Eq. 3].
The focus-by-predicting strategy is positioned as superior to prior hard-gating designs (FantasyTalking-style face-region attention restriction) because it works when the input is a full-frame face image (where no face localization is available before denoising) and because soft regularization preserves the model’s ability to model body kinematics and full-body interaction [§3.3].
Progressive training: Stage 2 starts at 80% subject-preservation / 20% audio-visual-sync task ratio and gradually shifts to 50/50, with parameter updates confined to self-attention layers and audio modules. Single-stage joint training degrades identity similarity (Tab. 3 row “w/o Progressive Training”) [§3.3, Table 3].
Time-adaptive CFG uses two guidance-scale configurations: text/image-dominant from timestep 1.0 → 0.98 (semantic layout), then audio/image-dominant from 0.98 → 0 (fine details, identity, lip sync). Separate (w_T, w_I, w_A) scales per modality with null-token substitution support flexible omission of any modality [§3.4, Eq. 4].
On the in-house S2V benchmark (100 cases, humans/objects/animals), HuMo-17B leads all baselines on AES (0.657), IQA (0.717), HSP (3.906), TVA (3.939), ID-Cur (0.731), ID-Glink (0.757), CLIP-I (0.687), DINO-I (0.447) — including the closed-source Kling 1.6 [Table 1].
On the MoCha audio-visual sync benchmark, HuMo-17B leads on TVA (6.508), ID-Cur (0.747), ID-Glink (0.740), AES (0.589), IQA (0.718), HSP (4.537); Sync-C 6.252 vs OmniHuman-1’s 6.526 (close second) and HuMo-1.7B already beats most open-source specialists [Table 2].
Ablations: removing progressive training drops ID-Cur 0.747 → 0.724 and Sync-C 6.252 → 6.106; removing focus-by-predicting drops Sync-C 6.252 → 5.946; full fine-tuning (vs the self-attention-only subset) drops AES 0.589 → 0.529 and TVA 6.508 → 6.157 — confirming that what is updated matters as much as how [Table 3].

Method

The backbone is the Wan-2.1 DiT-based T2V model (1.3B and 14B variants). Three additions: (1) reference-image VAE latents concatenated at the end of the noisy-video latent sequence along the temporal dimension, so subject identity propagates via self-attention without architectural changes; (2) per-DiT-block audio cross-attention with Whisper features, where each frame attends to a centered temporal window of audio embeddings; (3) a focus-by-predicting mask predictor M(·) attached in the last four DiT blocks that predicts the facial-region distribution and is supervised by BCE against the ground-truth face mask with size-aware reweighting.

Training is two-stage. Stage 1 trains only the self-attention layers on Stage-1 data (text + reference images) for 40k steps, with audio modules disabled — establishing subject preservation. Stage 2 enables audio modules and trains for another 40k steps on a curriculum that starts at 80/20 subject-preservation/audio-sync task ratio and progresses to 50/50, still updating only self-attention and audio-related modules. Flow matching is the training objective throughout.

At inference, CFG is split into per-modality scales (w_T, w_I, w_A) per Eq. 4, with missing modalities replaced by null tokens. The time-adaptive schedule swaps between a text/image-dominant configuration in the first 2% of the denoising trajectory (timesteps 1.0 → 0.98) and an audio/image-dominant configuration thereafter — motivated by the empirical observation that early steps build semantic structure and late steps refine identity and lip sync.

Results

On the in-house S2V (subject preservation) benchmark of 100 cases, HuMo-17B is top across all 8 metrics: AES 0.657 (next: Kling 1.6 0.645), IQA 0.717 (next: Kling 1.6 0.714), HSP 3.906, TVA 3.939 (next: Phantom 2.877), ID-Cur 0.731, ID-Glink 0.757, CLIP-I 0.687, DINO-I 0.447 [Table 1]. The 1.7B variant is comparable to but below the larger baselines on quality metrics while leading on TVA (3.222) — suggesting the recipe is what unlocks text following at small scale.

On the MoCha audio-visual sync benchmark, HuMo-17B leads on every metric except Sync-C: AES 0.589, IQA 0.718, HSP 4.537, TVA 6.508, ID-Cur 0.747, ID-Glink 0.740. Sync-C 6.252 is slightly below OmniHuman-1 (6.526) — but OmniHuman-1 is a closed-source I2V-based specialist with a subject-complete start frame, and HuMo achieves this with a single face reference image while preserving editability via text [Table 2].

Ablations confirm each design choice contributes: full fine-tuning loses 0.060 AES and 0.351 TVA vs the self-attention-only update; removing progressive training loses 0.023 ID-Cur and 0.146 Sync-C; removing focus-by-predicting loses 0.306 Sync-C — the largest single drop, validating the soft-attention-bias mechanism [Table 3].

Why it’s interesting

HuMo lands directly in the Audio-Driven Character Animation cluster but takes a distinct stance: where MultiTalk / InfiniteTalk / TalkingMachines / Wan-S2V all start from an I2V backbone (Wan2.1-I2V-14B or Wan2.2-14B) and require a subject-complete start frame, HuMo starts from a T2V backbone (Wan-2.1) and treats the reference image as a retrievable identity prior rather than the first frame. This sidesteps the editability ceiling of the I2V approach — Wan-S2V: Audio-Driven Cinematic Video Generation notes the same global-text / local-audio partitioning idea, but HuMo extends it with a true text-image-audio triplet rather than a start-frame-bound recipe. The time-adaptive CFG is the cluster’s first per-timestep guidance-scale schedule, adding a ninth axis to Classifier-Free Guidance variants alongside ALG’s per-timestep content schedule (Improving Motion in Image-to-Video Models via Adaptive Low-Pass Guidance) — both observe that the right knob varies across denoising. The minimal-invasive update strategy (self-attention-only) mirrors the MultiTalk / Wan-S2V partial-parameter recipe but applied to the T2V → triplet-conditioned setting, suggesting “freeze text cross-attention, train self-attention + new audio path” is the converged HCVG fine-tuning recipe regardless of backbone.