Identity-GRPO: Optimizing Multi-Human Identity-preserving Video Generation via Reinforcement Learning

TL;DR

Identity-GRPO (Alibaba Group + Fudan, v3 Jan 2026) targets the failure mode that VACE and Phantom hit on multi-human reference-conditioned video generation: identities swap, blend, or drift across characters in dynamic interactions. The recipe is reward-model-driven GRPO post-training over two base models (VACE-1.3B and Phantom-1.3B) using a Qwen2.5-VL-3B reward model trained on a large pairwise preference dataset of human-annotated plus synthetic-distortion-augmented multi-human videos. To stabilize GRPO on a multi-modal conditioning surface, the authors add two tricks — prompt finetuning (to compensate for VACE-vs-Phantom asymmetry in following prompt vs reference) and initial-noise differentiation — and use larger per-update video batches than single-modal GRPO needs, because cross-modal variance otherwise drives collapse. Reported gains are up to 18.9% on human-consistency metrics over the base; the team observes that off-the-shelf Qwen2.5-VL scores 43% (3B) / 65% (72B) on their identity benchmark, meaning the reward model has to be trained for the task — Yunong’s caveat in the Slack post that “it’s all about data” lands here.

Key claims

The dominant failure mode for VACE and Phantom on multi-human identity preservation is identity confusion / swap during dynamic interactions, distinct from the single-subject “copy-paste” problem that single-subject S2V already solves [Abstract].
Off-the-shelf Qwen2.5-VL is not a usable zero-shot reward signal for multi-human identity preservation: 3B scores 43% and 72B scores 65% on the paper’s multi-human identity-preserving preference benchmark (500 examples) [Abstract; benchmark spec confirmed in the awesome-RL-for-video-generation index entry].
A 3B-parameter video reward model is trained on a pairwise preference dataset built from human annotations plus synthetic distortion data (programmatic perturbations that simulate identity swaps/blends), with all comparisons anchored on human consistency across the full clip [Abstract].
Identity-GRPO is a Flow-GRPO-style variant: rectified-flow SDE rollouts on the base T2V/I2V model, group-relative advantage on the trained reward, applied to both VACE-1.3B and Phantom-1.3B [§3, paper acknowledgements note “based on Flow-GRPO, VideoAlign, Wan2.1, VACE and Phantom”].
VACE and Phantom have asymmetric sensitivity to prompt-vs-reference: VACE tends to follow the prompt even when it disagrees with the reference image, while Phantom tends to preserve the reference even when it disagrees with the prompt; the paper introduces prompt finetuning (per-base rewriting of the prompt so it does not conflict with the reference) to make GRPO trainable on both [§Method, v2 PDF excerpt].
Cross-modal variance forces a larger per-update batch of videos than standard single-modal GRPO — analogous to needing a larger batch size in standard training; without it, GRPO is “prone to instability or collapse” [§Method, v2 PDF excerpt].
Initial noise differentiation — using different initial noise samples (rather than the same noise) across the group when generating rollouts — is the paper’s second stability lever for multi-modal GRPO [§Method, v2 PDF excerpt].
Headline result: up to 18.9% improvement in human-consistency metrics over baseline VACE-1.3B and Phantom-1.3B [Abstract]. Training videos are sampled at 2 fps [§Method, v2 PDF excerpt].
Ablations target the impact of annotation quality and design choices on policy optimization, not just final scores — the paper positions itself as a study of “how to do RL post-training on multi-human S2V” as much as a recipe [Abstract].
Reported reward curves for both VACE-1.3B and Phantom-1.3B show a clear monotone upward trend during GRPO training [§Experiments, project page].

Method

The pipeline has three stages.

Stage 1 — Reward Model. A Qwen2.5-VL-3B backbone is fine-tuned on a large pairwise preference dataset for multi-human identity consistency. The dataset combines (a) human-annotated pairs of videos generated by VACE/Phantom with multi-person reference images, with annotators ranking which clip better preserves all reference identities across the clip’s frames; and (b) synthetic distortion samples constructed by programmatically perturbing identities (likely face-swap / region-swap operations) on otherwise-good generations to provide hard negatives for which “wrong” is unambiguous. The benchmark used to validate the RM itself is a 500-example human-consistency preference set — and the 43% (3B) / 65% (72B) Qwen2.5-VL zero-shot scores Yunong flags are measured on this set.

Stage 2 — Prompt Finetuning and Noise Differentiation. For each base model, the input prompt is rewritten so that it does not conflict with the reference image’s identity content — this is the “prompt finetuning” lever, and it is base-model-specific because VACE and Phantom have opposite default biases (prompt-dominant vs reference-dominant). For each prompt + reference, a group of G videos is generated with different initial noise samples (“initial noise differentiation”), in contrast to the shared-noise convention common in Flow-GRPO setups.

Stage 3 — GRPO Update. The reward model scores each generated video; group-relative advantages are computed against the group mean/std; the rectified-flow policy is updated with the clipped-ratio Flow-GRPO objective. Training operates on 2-fps videos at ~80×… resolution (PDF excerpt cuts off). The acknowledgement that the repo is “based on Flow-GRPO, VideoAlign, Wan2.1, VACE and Phantom” places this firmly in the post-training-of-video-DiTs lineage rather than as a from-scratch RL recipe. Both VACE-1.3B and Phantom-1.3B serve as the policy backbone in separate runs.

The released code lives at github.com/alibaba/identity-grpo; the project page is ali-videoai.github.io/identity_page.

Results

Headline number: up to 18.9% improvement on the paper’s human-consistency metrics over the base VACE-1.3B / Phantom-1.3B [Abstract]. The paper does not report comparable numbers against larger (14B) bases, where the headroom may differ.
Reward-model zero-shot ceiling: Qwen2.5-VL 3B reaches 43% accuracy, Qwen2.5-VL 72B reaches 65% accuracy on the multi-human identity-preserving preference benchmark (500 examples) [reported on the GitHub repo card and in Yunong’s Slack note]. This is the key datapoint behind the “it’s all about data” critique — without the trained 3B reward model, the GRPO signal would be near chance.
Reward training curves: ID-Consistency reward rises monotonically over training for both VACE-1.3B and Phantom-1.3B [§Experiments, project page figures]. No reported plateau / collapse on the published curves.
Per-base comparison: the paper reports both VACE-1.3B and Phantom-1.3B Identity-GRPO variants beat their respective bases on the human-consistency axis; the specific deltas are not reproduced in the project-page excerpt and would need the full PDF.
Qualitative: project page side-by-side videos show baseline outputs with clear identity mismatches (wrong face, attribute blend across characters) being corrected by Identity-GRPO into reference-faithful sequences [project page].
What is not reported in the abstract/project page: motion-amplitude / aesthetic-quality tradeoffs, generalization to non-human multi-subject identity preservation, ablation on the synthetic-distortion fraction of the RM training set, and whether the recipe transfers to 14B-scale VACE/Phantom (where the base already has more identity headroom).

Why it’s interesting

Identity-GRPO is the third filed paper in the Pref-GRPO → RewardDance → Identity-GRPO lineage on the wiki, and it sharpens a specific question that Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learning and RewardDance: Reward Scaling in Visual Generation mostly skirted: what happens when the reward signal itself is the bottleneck, not the GRPO mechanics? Pref-GRPO’s contribution was the pairwise win-rate reformulation given a usable RM; RewardDance’s was scaling the RM to 26B; Identity-GRPO’s is the empirical observation that for multi-human identity, neither pointwise nor pairwise off-the-shelf VLM scoring works (43% / 65% zero-shot Qwen2.5-VL) and a task-specific 3B RM trained on a custom preference dataset is load-bearing. The Slack pointer’s “it’s all about data” critique is correct in a literal sense — the paper’s main artifact is the preference dataset and its augmentation pipeline; the GRPO recipe is a near-Flow-GRPO clone with two stability hacks (prompt finetuning, noise differentiation) added for multi-modal-variance reasons.

Identity-GRPO also complements Phantom-Data: Towards a General Subject-Consistent Video Generation Dataset on the same axis — Phantom-Data attacks subject-consistency at the training-data construction level (cross-pair retrieval, no RL); Identity-GRPO attacks it at the post-training level (RL on a trained RM) on top of a Phantom-style base. The natural ablation neither paper runs is a head-to-head: Phantom-Data-trained 1.3B vs Phantom-Data-trained 1.3B + Identity-GRPO, to isolate how much of the multi-human consistency gap is data and how much is post-training. The fact that both papers report meaningful gains on the same underlying base suggests the gap has both components, but the relative size is unknown. BindWeave: Subject-Consistent Video Generation via Cross-Modal Integration picks the third lever (architecture: MLLM front end on the same Wan 2.1 14B base) — together with Phantom-Data and Identity-GRPO, this triangulates that subject-to-video consistency is not primarily an architecture problem at 1.3B but is moved by data and RL post-training simultaneously.