ConsID-Gen: View-Consistent and Identity-Preserving Image-to-Video Generation

TL;DR

ConsID-Gen tackles identity drift in image-to-video generation by augmenting the first frame with two unposed auxiliary views of the same object and fusing them with a dual-stream visual–geometric encoder (CLIP for semantics, VGGT for geometry) before feeding a Wan2.1-1.3B DiT backbone. The paper also ships ConsIDVid (large-scale object-centric video dataset built from Co3D/OmniObject3D/Objectron + 80h proprietary e-commerce UGC + synthetic interpolated clips) and ConsIDVid-Bench, a multi-view-consistency benchmark adding Chamfer Distance, MEt3R, and DINO-based Object Similarity to the VBench-I2V suite. The combination outperforms Wan2.2-5B and HunyuanVideo on the proprietary subset and beats Wan2.1-14B on Chamfer Distance / Video Similarity on the public subset, with the ablation showing the architecture (not the data) is the load-bearing piece.

Key claims

The dominant I2V failure mode is appearance drift / geometric distortion under viewpoint change, attributed to (i) sparsity of single-view 2D observations and (ii) weak cross-modal alignment between separately-encoded image and text condition streams [§1, Fig. 1].
T2V models consistently beat I2V models on identity preservation because they bypass the sparse-visual + late-fusion alignment problem (e.g. CogVideoX-1.5, Wan2.1 in T2V mode outscore their I2V counterparts on Subject Consistency) [Table 1, §1].
ConsID-Gen’s recipe is: augment the first frame with two unposed auxiliary views of the same object, run a dual visual encoder (CLIP for semantics + VGGT for geometry), then a Multi-Modal Visual–Geometric Module (MVGM) fuses appearance ↔ geometry tokens via dual-stream attention, and a Multi-Modal Text–Visual Module (MTVM) aligns the fused visual stream with text before the DiT backbone [§4.1–4.3, Fig. 5].
The MVGM lifts the MMDiT bidirectional cross-attention idea from text↔image to image↔geometry: a dual-stream attention block between CLIP appearance tokens and VGGT geometry tokens, with cross-attention from two auxiliary-view geometry features injected on top [§4.3].
ConsIDVid is built by a scalable curation pipeline: (i) Co3D + OmniObject3D + Objectron real sources, (ii) ~80h proprietary monocular e-commerce UGC, (iii) MVImgNet2.0-conditioned interpolation-synthesized clips with Qwen2.5-VL captions; filters include shot-boundary stitching, 320p+/81-frame+, LAION-aesthetics ≥3.0, OCR suppression, and DBSCAN-based outlier removal on per-item multi-view image clusters [§3.1–3.4].
Hierarchical two-stage captioning (Qwen2.5-VL): stage 1 produces a strict appearance-only caption from 12 frames (color/material/shape/text/logos; camera + context forbidden); stage 2 conditions on stage 1 + 24 frames to add dynamics (camera, human-object, object motion) [§3.3].
ConsIDVid-Bench reframes I2V evaluation as multi-view consistency, scoring with Chamfer Distance between input/synthesized 3D point sets, MEt3R (DUSt3R-based dense cross-view feature similarity), CLIP Video Similarity, and DINO-based Object Similarity on segmented foreground [§3.5].
On the proprietary 241-video subset, ConsID-Gen beats Wan2.2-5B on Subject Consistency by 3.31 points (95.30 vs 91.99) and lowers MEt3R from 0.1826 → 0.0978 (≈ +46% better multi-view consistency) [Table 2].
On the public 370-video subset, ConsID-Gen achieves the lowest Chamfer Distance (0.1277) and the highest Video Similarity (84.95) but underperforms Wan2.2-5B / Wan2.1-14B on I2V-Subject and I2V-Background — attributed to occasional collapse on distractor-rich inputs (e.g. grid paper) [Table 3, §5.2, Appendix 6.3].
Ablation: adding only the geometry encoder yields no improvement over the Wan2.1 baseline; adding view-assisted multi-view inputs on top helps; the text↔visual MTVM connector is what produces long-range identity stability — i.e. the connector matters more than either auxiliary stream in isolation [Table 4, §5.4, Fig. 9].
Data ablation: LoRA-finetuning Wan2.2-5B (rank 64) on ConsIDVid yields only modest gains, while ConsID-Gen’s architecture trained on the same data wins by a large margin — implying the multi-view + dual-encoder design, not the dataset, is the dominant lever [Table 5, §5.4].
Human preference: against Wan2.1, ConsID-Gen wins on both Identity Consistency and Visual Quality; against proprietary Veo-3.1 it is rated comparable on Identity Consistency [§5.2, Fig. 8].

Method

The model builds on Wan2.1-Fun-1.3B-InP (81-frame clips at the base resolution). Conditioning takes (first frame, two uncalibrated auxiliary images of the same object, text). The first frame is encoded by CLIP into semantic tokens; all three images go through VGGT (with alternating frame-wise and global self-attention) to produce dense geometry tokens. The Multi-Modal Visual–Geometric Module (MVGM) is an MMDiT-style block that runs bidirectional dual-stream attention between CLIP first-frame tokens and first-frame geometry tokens; the two auxiliary views’ geometry features are then cross-attended into the MVGM output, injecting multi-view structural priors. The Multi-Modal Text–Visual Module (MTVM) runs a second dual-stream block between the fused visual representation and the text tokens, producing unified conditioning tokens for the DiT backbone. Training is 33K steps with Adam, per-GPU batch 4 × gradient accumulation 4 (effective 16), on A100-80G. Inference uses 50 sampling steps with CFG=5.

Results

Proprietary subset (241 videos, e-commerce product showcases), [Table 2]: ConsID-Gen achieves Subject Consistency 95.30 vs Wan2.2-5B 91.99 and Wan2.1-14B 90.37; MEt3R 0.0978 vs Wan2.1-14B 0.1572 / HunyuanVideo 0.2270 (lower is better); Object Similarity (DINO) 69.2 vs Wan2.2-5B 68.6. Public subset (370 videos, Co3D/OmniObject3D + synthetic), [Table 3]: best-in-class Chamfer Distance (0.1277) and Video Similarity (84.95), but I2V-Subject/I2V-Background lose to Wan2.1-14B on inputs containing distractor structures. Component ablation [Table 4]: Baseline 90.83 / + Geo Enc. 89.65 (worse) / + View-Assisted 91.87 / Full ConsID-Gen 95.13 on Subject Consistency — the MTVM connector + view-assisted images together are what produces the gain; the geometry encoder alone hurts. Dataset ablation [Table 5]: LoRA-finetuning Wan2.2-5B on the same data gives Subject Consistency 91.22 (down from 91.99 base), while ConsID-Gen on the same data reaches 95.30 — design > data.

Why it’s interesting

The “geometry encoder alone hurts; geometry + auxiliary views + text-aligned connector wins” ablation is the load-bearing finding — it says the wiki’s Dual-stream diffusion transformer pattern (currently filed only as a joint-A+V technique in LTX-2: Efficient Joint Audio-Visual Foundation Model, MOVA: Towards Scalable and Synchronized Video-Audio Generation, SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model) generalizes to the image↔geometry modality pair, partially answering the open question on that concept page about whether the recipe is specific to video/audio sampling-rate mismatch. The data-curation pipeline is also a concrete instance of the verification-gate thesis in Synthetic Training Data — hierarchical Qwen2.5-VL captioning with a strictly-typed stage 1 + dynamics-augmented stage 2 is a structurally similar “stack of generators + curation gate” to Action100M: A Large-scale Video Action Dataset, applied to an object-centric I2V corpus instead of HowTo100M actions. For Luma, the multi-view-as-conditioning recipe is directly relevant when the input is a product or character with one good photograph but other reference views available.