HOIGen-1M: A Large-scale Dataset for Human-Object Interaction Video Generation

TL;DR

HOIGen-1M (CVPR 2025) is the first million-scale T2V training dataset specialized for human-object interaction (HOI), curated from ~80M raw videos sourced from BEHAVE / InterCap / HOI4D / Panda-70M / ViSR / Mixkit / Pixabay / Pexels through a multi-stage filter (metadata → OCR → aesthetics → optical-flow motion score → PLLaVA+Qwen2.5 LLM HOI judgement → 7-annotator human verification). Captions are generated by a Mixture-of-Multimodal-Experts (MoME) pipeline — two VLM caption experts (PLLaVA, Qwen2-VL) cross-verified by a Llama-3.1 decision expert that flags hallucinations and triggers a third expert + regeneration. The release also proposes two new HOI-specific evaluation metrics (CoarseHOIScore from an HOI-triplet detector, FineHOIScore from contact distance between SMPL-X keypoints and object masks) and shows that fine-tuning OpenSora / CogVideoX-{2B,5B} on HOIGen-1M materially improves CoarseHOIScore (e.g. CogVideoX-5B: 32.84% → 44.04%) while leaving VBench appearance metrics roughly unchanged.

Key claims

HOIGen-1M curates over 1M video clips with manually verified HOI from ~80M raw videos, sourced across BEHAVE / InterCap / HOI4D plus five general T2V datasets (Panda-70M, ViSR, Mixkit, Pixabay, Pexels), with ≥720p resolution and ≥20 FPS [§3.1, Tab. 1].
The HOI vocabulary covers 15,000+ objects and 7,000+ interaction action types — substantially larger than prior HOI datasets, which are typically capped at tens of thousands of videos and ~hundred-scale action vocabularies (CAD-120: 10 activities; MPHOI-72: 72 videos; PVSG: 400 videos) [§3.2 Text Analysis, §2].
Average caption length is 153.8 words vs 12.0 (WebVid-10M) and 13.2 (Panda-70M) — captions are explicitly engineered to focus on object attributes, human movements, and temporal interaction structure rather than scene tags [§3.2 Text Analysis].
The data-curation pipeline is staged: metadata (duration/resolution/FPS) → OCR filtering via DBNet++ → Laion Aesthetic Predictor → UniMatch optical-flow motion score (rejects too-static and too-violent flow) → PLLaVA caption + Qwen2.5 HOI judgement → 7 human annotators removing inconspicuous HOI cases. After the LLM stage ~1.5M videos remain; after human verification ~1.1M [§3.1].
MoME (Mixture-of-Multimodal-Experts) is the captioning gate: PLLaVA + Qwen2-VL generate independent captions, Llama-3.1 judges whether they describe the same content; on disagreement a third expert is invoked focused on the disagreement region, and Llama-3.1 regenerates the corrected caption. On agreement Llama-3.1 picks the richer of the two [§3.2, Fig. 4].
Two new HOI-specific evaluation metrics are proposed beyond VBench: CoarseHOIScore aggregates an HOI-triplet detector’s confidence across uniformly sampled frames, and FineHOIScore measures pixel-distance contact between SMPL-X keypoints (humans) and object masks [§4.1 Eqns. 1–2].
Even the strongest commercial model evaluated (Kling 1.5) reaches only 42.72% CoarseHOIScore; Mochi-10B reaches 28.78% despite its 10B parameter scale — current open and closed T2V models do not generate HOI well [Tab. 2].
VBench is a poor proxy for HOI quality: motion-smoothness scores across all 10 evaluated models are >98% while CoarseHOIScore ranges from 8.80% (OpenSoraPlan) to 42.72% (Kling 1.5) [Tab. 2, §4.2].
Fine-tuning open T2V models on HOIGen-1M via LoRA improves CoarseHOIScore substantially: OpenSora 31.86 → 35.38, CogVideoX-2B 31.34 → 39.13, CogVideoX-5B 32.84 → 44.04; appearance/VBench numbers also rise modestly [Tab. 4, §4.3].
The MoME caption recipe itself contributes a measurable share of the gain: under matched fine-tuning protocol, CogVideoX-2B trained with MoME captions reaches CoarseHOIScore 39.13 vs 33.17 for PLLaVA-only captions [Tab. 3].
Fine-tuned CogVideoX-5B is reported to reach CoarseHOIScore competitive with Kling 1.5 (44.04 vs 42.72), suggesting that an open 5B model + HOI-targeted SFT can close the gap to commercial software on this specific axis [Tab. 2, Tab. 4, §1].

Method

The pipeline is bipartite: video curation, then captioning. Curation starts from ~80M videos across 8 source datasets and applies six successive filters — metadata (≥1s, ≥720p, ≥20 FPS), DBNet++ OCR to reject text-heavy frames, Laion Aesthetic Predictor for visual quality, UniMatch optical-flow motion score with mid-range thresholds, then a two-model LLM gate (PLLaVA writes per-video captions; Qwen2.5 judges whether the caption describes an HOI). The output ~1.5M videos are then human-verified by 7 annotators over roughly 8 weeks, leaving ~1.1M clips.

Captioning runs MoME on each remaining clip. Two multimodal experts — PLLaVA and Qwen2-VL — generate independent structured captions under a prompt that biases attention toward objects, human motion, state changes, and interaction temporal structure. A Llama-3.1 decision expert then asks “do these two captions describe the same video (same object, same interaction, same scene)?”. A negative answer triggers a third multimodal expert focused on the regions where the first two disagree; the disagreement reasoning + third caption are fed back to Llama-3.1, which emits a corrected caption. A positive answer triggers Llama-3.1 to score the two existing captions for richness and pick the better one. The output of every video is one caption plus extracted HOI categories.

Evaluation: a 306-prompt suite is built from HICO-DET object/interaction pairs, LLM-extended to descriptive paragraphs. CoarseHOIScore averages an HOI-triplet detector’s confidence-thresholded predictions across uniformly sampled frames; FineHOIScore averages, per frame, whether the pixel distance between any SMPL-X human keypoint and an object mask is below a contact threshold — substituting geometry for the unobservable force signal. Both metrics are averaged across all generated videos per model.

Results

Of ten T2V models evaluated zero-shot, Kling 1.5 leads at CoarseHOIScore 42.72%, then Hailuo 39.56%, Dreamina 36.36%, CogVideoX-5B 32.84%; OpenSoraPlan trails at 8.80% [Tab. 2]. The gap between best-commercial and best-open is much smaller in VBench motion-smoothness (~96–99% across the board) than in CoarseHOIScore (range 8.8%–42.7%), which the paper offers as evidence that HOI quality is an axis VBench misses. The MoME captioning ablation (Tab. 3) holds the base model and fine-tuning protocol fixed and swaps in PLLaVA-only vs MoME captions for the training set; the MoME-trained model gains +5.96 CoarseHOIScore (33.17 → 39.13) and ~+3 on VBench imaging quality. The dataset ablation (Tab. 4) fine-tunes three open models on HOIGen-1M with LoRA: OpenSora gains +3.52, CogVideoX-2B +7.79, CogVideoX-5B +11.20 on CoarseHOIScore; FineHOIScore also rises (e.g. CogVideoX-5B 94.25 → 96.04). Qualitative comparisons (Fig. 6) report cleaner hands and more reasonable object contact post-finetuning, with HOIScore lifting from 0%/50% to substantially higher on the displayed cases.

Why it’s interesting

For Luma the immediate read is two-fold. First, HOIGen-1M is a candidate fine-tuning corpus for character-centric or virtual-try-on-style generation work — it is the only million-scale, manually verified HOI video dataset with long captions in the open world right now, and the +11.20 CoarseHOIScore gain on CogVideoX-5B post-LoRA is direct evidence that dropping it into a fine-tuning pipeline shifts the HOI-quality axis meaningfully even on an off-the-shelf open model. The Jiaming note flags it for “human stuff”, which lines up cleanly with the use case. Second, the MoME captioning recipe is the more transferable contribution — a two-expert + judge + on-disagreement-third-expert pattern is exactly the cost-effective verification gate the wiki’s Synthetic Training Data concept page already tracks across Action100M: A Large-scale Video Action Dataset (Tree-of-Captions + GPT-OSS-120B Self-Refine) and Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning (hindsight curation). HOIGen-1M lands earlier in that lineage (March 2025) and is closer in spirit to Action100M’s hierarchical-text-then-LLM-aggregation pattern, but with a smaller VLM stack (PLLaVA + Qwen2-VL + Llama-3.1) and explicit hallucination detection via cross-verification rather than Self-Refine. On the evaluation side, CoarseHOIScore and FineHOIScore are a useful sibling to HumanScore: Benchmarking Human Motions in Generated Videos — HumanScore measures biomechanical plausibility of solo human motion via OpenSim skeleton fitting, HOIGen-1M’s metrics measure whether the human is actually in contact with an object, and the two are complementary rather than competing axes for reward modeling. Caveat: the HOI-triplet detector backing CoarseHOIScore is itself imperfect on generated frames (the paper notes some models score 0% mostly because hand detection fails), so CoarseHOIScore tracks generator-detector co-modality more than pure HOI plausibility.