Phantom-Data: Towards a General Subject-Consistent Video Generation Dataset

TL;DR

Phantom-Data (ICLR 2026, ByteDance Intelligent Creation Lab) is the first open, general-purpose cross-pair dataset for subject-consistent video generation — ~1M identity-consistent (reference, target) pairs where reference and target frames come from different videos, spanning humans, animals, products, and multi-subject scenes. The construction pipeline mines subjects from a 53M-video corpus + 3B-image LAION bank via three stages: (1) S2V detection with Qwen2.5-VL grounding + InternVL2.5 visual-semantic recheck, (2) contextually diverse retrieval using category-specific identity encoders (ArcFace for faces, fine-tuned CLIP for objects, concatenated face+appearance for humans), and (3) prior-guided identity verification via brand-logo gating for products, intra-video temporal sampling for living entities, and a final VLM-based consistency check. A 1.3B Phantom-wan model trained on Phantom-Data beats in-pair baselines on GPT-4o subject score (3.041 vs 2.481), Reward-TA prompt-following (3.827 vs 2.074), and VBench imaging quality, with on-par DINO subject consistency — direct evidence that the “copy-paste” failure mode in subject-to-video is a data problem, not an architecture problem.

Key claims

The dominant failure mode in subject-to-video generation — the “copy-paste problem” where the generator reproduces both subject identity and irrelevant context (background, pose, viewpoint) — is caused by in-pair training where reference and target frames come from the same video clip, entangling identity with context [§1, Fig. 1(a)–2].
Phantom-Data contains ~1M cross-pair identity-consistent samples spanning humans, animals, products, environments, and clothing, with ~30,000 multi-subject scenes; ~720k single-subject samples and ~280k multi-subject samples [§3.1, Fig. 3(d)].
The retrieval bank is constructed from ~53M video clips (Koala-36M + proprietary) plus a 3B-image LAION augmentation that injects scene/pose/appearance variation beyond what video alone provides [§4.1, §4.2.2 Subject Source].
Identity-preserving subject representations are category-specific: ArcFace embeddings for faces, a CLIP-based encoder fine-tuned on a consistency-focused image dataset (ObjectMate-style) for general objects, and concatenated face+appearance embeddings for humans [§4.2.2 Eqns. 1–3].
Retrieval applies both upper- and lower-similarity thresholds: upper threshold drops near-duplicates (overly similar candidates that would re-introduce copy-paste), lower threshold drops mismatched identities [§4.2.2 Query-Based Retrieval].
Identity verification uses prior knowledge differently per category: products are gated on visible brand logos (Nike, Audi) that remain identifiable across scenes; living entities are restricted to different clips of the same long-form video to bound identity drift while preserving pose/scene variation [§4.2.3 Utilization of Prior Knowledge].
A two-stage VLM verifier follows retrieval: per-pair identity consistency (strict color/packaging/textual match for objects; face identity + clothing alignment for humans) plus contextual diversity (substantial background variation) [§4.2.3 VLM-Based Consistency Verification].
On a 100-prompt suite covering humans, animals, products, environments, and clothing, a 1.3B Phantom-wan trained on Phantom-Data beats in-pair, in-pair+augmentation, and face-cross-pair baselines: GPT-4o subject score 3.041 (vs 2.481 / 2.792 / 2.378), Reward-TA prompt-following 3.827 (vs 2.074 / 2.427 / 3.022), VBench imaging quality 0.739 (vs 0.725 / 0.730 / 0.723), VBench background 0.948, VBench subject 0.944 [§5.2, Tab. 2].
DINO subject-consistency is 0.416 — lower than the in-pair baseline 0.478 but higher than face-cross-pair 0.354; the in-pair lead on DINO is consistent with copy-paste behavior (target frames literally reproduce reference appearance) and is not a true generalization win [§5.2, Tab. 2].
Subject diversity contributes monotonically: adding human, IP/animal, product, and multi-subject categories over a face-only baseline improves GPT-4o 2.378 → 2.901 (+0.523) and Reward-TA 3.022 → 3.512 (+0.490) [§5.3, Tab. 8].
Data scale matters: 100k → 1M training samples improves GPT-4o 3.090 → 3.175 and Reward-TA 3.796 → 3.827 with DINO holding steady (0.408 → 0.416) [§5.3, Tab. 8].
Multi-source retrieval (video + LAION 3B images) beats video-only retrieval on recall and candidate diversity, especially for product-centric scenarios with high intra-instance variation [§5.3, Fig. 6(b)].
Without the prior-filtering + VLM-verification stages, the retrieval surface includes visually similar but semantically incorrect matches (false positives) that the cosine-similarity bands alone can’t reject [§5.3, Fig. 6(c)].

Method

The pipeline is three stages over a 53M-clip video corpus paired with 3B LAION images. Stage 1 — S2V Detection. Three frames are sampled at 25%/50%/75% of each clip. Qwen2.5 extracts noun-phrase subject candidates from the clip’s caption; Qwen2.5-VL grounds each phrase to image regions; bounding boxes outside the 10–80% area band or overlapping >0.8 IoU are suppressed; InternVL2.5-7B then re-verifies each retained subject on three axes — completeness (no partial crops), specificity (distinct identifiable object, not a generic “tree” or “rock”), and subject-text matching (region semantically consistent with the phrase). A separate InternVL2.5 instance is invoked for the final consistency check.

Stage 2 — Contextually Diverse Retrieval. All detected subjects from the video corpus are indexed into a retrieval bank, augmented with 3B LAION images. Identity embeddings differ by category: ArcFace for faces, a CLIP-based ObjectMate-style fine-tuned encoder for general objects, and a concatenated face+appearance vector for humans. Querying applies both upper and lower cosine-similarity thresholds — upper to drop near-duplicates, lower to drop unrelated identities.

Stage 3 — Prior-Based Identity Verification. Category-specific priors are applied: non-living subjects must show recognizable brand logos to survive identity verification; living entities are restricted to different clips of the same long-form video (which by construction ensures identity continuity while supplying genuine scene/pose variation). A final VLM-based pairwise check enforces both identity consistency (strict color/packaging/text match for objects, face + clothing alignment for humans) and contextual diversity (substantial background change).

The validation model is a 1.3B Phantom-wan (built on Wan2.1) trained with Rectified Flow on 64 A100s for 30k iterations at 480p; inference uses Euler sampling with 50 steps and classifier-free guidance.

Results

On the 100-prompt evaluation suite, training on Phantom-Data dominates the three baselines on the prompt-following and visual-quality axes: GPT-4o subject score 3.041 vs 2.481 (in-pair) / 2.792 (in-pair+aug) / 2.378 (face cross-pair); Reward-TA 3.827 vs 2.074 / 2.427 / 3.022; VBench IQ 0.739, BG 0.948, Subj 0.944, all best-in-class [§5.2, Tab. 2]. DINO subject consistency at 0.416 sits below in-pair (0.478) but the gap is interpretable: in-pair models literally reproduce the reference frame, which DINO rewards as “consistent,” whereas the paper’s qualitative comparisons in Fig. 5 show the cross-pair model actually following text prompts that the in-pair model ignores. Subject-diversity ablation (Tab. 8) shows monotonic gains adding human, IP/animal, product, and multi-subject buckets onto a face-only baseline (final +0.523 GPT-4o, +0.490 Reward-TA). Scale ablation (Tab. 8) shows 100k → 1M improving across all metrics with DINO essentially flat (0.408 → 0.416) — consistent with the framing that scale buys prompt-following room without sacrificing identity. The retrieval ablations (Fig. 6) verify that (a) minute-level temporal sampling outperforms second-level, (b) video+image retrieval beats video-only, (c) without prior filtering and (d) without the VLM verifier, the retrieved candidate pool contains both near-duplicates and mismatched identities.

Why it’s interesting

Phantom-Data is the cleanest published evidence that the subject-consistency × prompt-following Pareto frontier in subject-to-video generation is moved by training-data construction, not by architecture changes. The wiki already has two adjacent dataset-construction papers worth contrasting: HOIGen-1M: A Large-scale Dataset for Human-Object Interaction Video Generation (HOIGen-1M) and Action100M: A Large-scale Video Action Dataset (Action100M). HOIGen-1M is about labeling an HOI-targeted million-scale corpus via a MoME caption gate; Action100M is about labeling 1.2M HowTo100M videos with a Self-Refine action-caption gate. Phantom-Data is the first filed paper in this lineage where the curation is pair construction — the bottleneck is not what tag to assign each clip, but which (reference, target) pair to train against. The three pipelines share the same “verification gate is the load-bearing piece” thesis (the wiki’s Synthetic Training Data page already tracks this), but Phantom-Data extends it to a fundamentally relational filter — cross-context, same identity — that single-clip captioning gates can’t express. The pipeline is also a real-world stress test of identity-embedding choices: the paper’s per-category recipe (ArcFace / fine-tuned CLIP / concat) implies no single encoder solves identity matching across humans, animals, and products at the scale needed, which is a useful caution for anyone building a unified retrieval bank. Direct beneficiaries on the wiki: FIT: A Large-Scale Dataset for Fit-Aware Virtual Try-On (FIT virtual try-on — Phantom-Data’s product-cross-pair recipe is structurally identical), LPM 1.0: Video-based Character Performance Model (character-performance model, would benefit from cross-pair identity training), and the Phantom-wan-evaluated Wan-Image: Pushing the Boundaries of Generative Visual Intelligence / Wan2.1 backbone.