HumanNet: Scaling Human-centric Video Learning to One Million Hours

TL;DR

HumanNet (Peking University DAGroup, Daquan Zhou et al.) is a one-million-hour human-centric video corpus designed as scalable infrastructure for embodied pretraining — fine-grained activities, human–object interactions, tool use, and long-horizon behaviors, split across first- and third-person viewpoints. Beyond raw scale, the paper frames data curation (human-centric filtering, temporal structuring, viewpoint diversity, motion/caption/hand–body annotation enrichment) as first-class design principles. A first-step controlled VLA ablation shows that continued training of a Qwen VLM on 1,000 hours of HumanNet egocentric video beats continued training on 100 hours of real-robot data from Magic Cobot under fixed validation, the headline argument for human video as a scalable substitute for robot data.

Key claims

HumanNet is a one-million-hour human-centric video corpus spanning first-person and third-person views, organized by a multi-axis taxonomy over source type, viewpoint, task structure, environment, interaction style, motion category, and metadata availability [Abstract; §Taxonomy].
Interaction-centric annotations are bundled with the video: captions, motion descriptions, and hand- and body-related signals enabling motion-aware and interaction-aware learning [Abstract].
The data-curation paradigm treats human-centric filtering, temporal structuring, viewpoint diversity, and annotation enrichment as first-class design principles rather than after-the-fact post-processing [Abstract].
Controlled VLA ablation: continued training from the Qwen VLM on 1,000 hours of egocentric HumanNet video surpasses continued training on 100 hours of real-robot data from Magic Cobot, under a fixed validation set [Abstract; §VLA Validation].
A 1,000-hour-HumanNet initialization matches or modestly surpasses a 100-hour real-robot baseline and “substantially closes the gap” to a 20,000-hour real-robot baseline under the LingBot-VLA post-training regime (100 tasks × 20 episodes, 34 hours total) [README §Validation].
The release is open: dataset on Hugging Face (DAGroup-PKU/HumanNet), code and project page (dagroup-pku.github.io/HumanNet), with a HF_ENDPOINT mirror path documented for mainland-China users [README §Download].

Method

HumanNet is a data + infrastructure paper, so the “method” is a curation pipeline rather than a model. The corpus is assembled to ~967k–1M hours from unstructured internet video, filtered for human-centric content, then organized along a multi-axis taxonomy: source type, viewpoint (first-person / third-person), task structure, environment, interaction style, motion category, and metadata. Annotation enrichment runs on top: caption labels, motion descriptions, hand and body signals, plus motion-centric representations. The egocentric subset is the one used for the headline VLA validation; the third-person subset is positioned as substrate for activity understanding and motion generation.

The validation experiment piggybacks on the same group’s LingBot-VLA: A Pragmatic VLA Foundation Model / Advancing Open-source World Models (LingBot-World) stack. A fixed downstream LingBot-VLA post-training protocol (100 tasks, 20 episodes per task, 34 hours of downstream data) is held constant; only the pretraining source is varied. The four conditions are (i) random init, (ii) 100 hours of Magic Cobot real-robot data, (iii) 1,000 hours of egocentric HumanNet video, and (iv) a 20,000-hour real-robot baseline. The claim is that condition (iii) matches or beats (ii) and substantially closes the gap to (iv).

Results

Corpus: ~1,000,000 hours of human-centric video; first- and third-person views; multi-axis taxonomy over source / viewpoint / task / environment / interaction / motion / metadata [Abstract].
Annotations: captions, motion descriptions, hand-and-body signals, motion-centric representations [Abstract].
VLA validation under fixed downstream protocol (LingBot-VLA, 100 tasks × 20 episodes, 34 h): 1,000 h egocentric HumanNet pretraining ≥ 100 h real-robot pretraining, and substantially narrows the gap to the 20,000 h real-robot baseline [README §Validation].
Specific per-task SR / PS numbers and the headline corpus statistic versus the “967k hours” project-page figure were not retrievable at filing time; the abstract characterizes the comparison qualitatively.

Why it’s interesting

This is the second human-egocentric-video-for-robot-policy corpus filed this week, after the consortium-style EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World (1,362 h, 1,965 tasks, 2,087 demonstrators, Aria Gen 1 across five academic labs + Scale/Mecka). HumanNet sits at the opposite end of the scale-vs-curation axis: ~700× more hours but assembled from unstructured internet video (filtered) rather than protocol-controlled lab capture, and validated against a single in-house VLA stack rather than a cross-lab multi-embodiment study. The Build AI egocentric line (Egocentric-1M — largest egocentric video dataset (Build AI / Eddy Xu announcement), Build AI's early egocentric release — 400k action labels, 2.5k clips, 2× open-source dataset size (Eddy Xu tweet, Oct 22 2025)) is the closest scale-side comparison, but Build AI captures single-domain factory POV from custom glasses while HumanNet aggregates first- and third-person across activities, environments, and viewpoints — a different bet on what generalizes.

The headline result — 1,000 h human video ≥ 100 h teleop — is the same trade-off EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World is built to measure cross-lab, and the same argument the broader human-video-as-pretraining literature (Introducing the V-JEPA 2 world model and new benchmarks for physical reasoning, Action100M: A Large-scale Video Action Dataset) has been edging toward. The Ant-DAGroup pipeline (LingBot-VLA + LingBot-World + HumanNet) is now a fully vertically integrated open stack: data corpus → world model → VLA policy. Worth tracking whether the 1M-hour internet-video curation actually produces transfer at the rate the abstract claims, given that the comparison is against only 100 h of real-robot data (a low bar) and the closer-gap comparison against 20k h is “substantial” rather than matched.