EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World

TL;DR

EgoVerse is a consortium-built ecosystem (Georgia Tech, Stanford, UC San Diego, ETH Zürich, MIT CSAIL, Meta Reality Labs, Mecka AI, Scale AI) for collecting, hosting, and learning from egocentric human-demonstration data for robot policy training. The current release totals 1,362 hours / ~80k episodes / 1,965 tasks / 240 scenes / 2,087 demonstrators, split into a controlled academic component (EgoVerse-A, Project Aria Gen 1 glasses, six standardized flagship tasks, shared protocols across labs) and an industry-scale in-the-wild component (EgoVerse-I, Scale + Mecka contributions). The backing infrastructure — EgoDB — exposes the corpus as a living dataset with continuous ingestion, plus a commodity-phone capture path that lowers the contribution barrier. Beyond the data, the paper conducts a cross-lab, multi-embodiment study finding that policy performance generally improves with more human data, but that effective scaling depends on alignment between the human data and the robot learning objective.

Key claims

The release totals 1,362 hours of human demonstrations across 240 scenes, 1,965 tasks, and 2,087 unique demonstrators [§EgoVerse Dataset / overview].
The dataset has two intentionally distinct components: EgoVerse-A (controlled, reproducible, mirrored protocols across academic labs) and EgoVerse-I (scale, diversity, in-the-wild captures from industry partners) [§EgoVerse Dataset].
Academic data collection uses Project Aria Gen 1 glasses — a 75 g head-worn rig with a wide-FoV RGB camera plus two side-facing monochrome SLAM/hand-tracking cameras — yielding accurate camera poses, 3D head tracking, and dense language annotation out of the capture pipeline [§EgoVerse-A; Aria sensor description].
Six standardized flagship tasks are defined and executed across the EgoVerse-A labs, each task performed across 8–12 scenes per lab with randomized objects and varied demonstrator height/posture/movement [§EgoVerse-A].
EgoDB is positioned as a living dataset infrastructure with continuous ingestion, standardized processing, unified storage, controlled access, visualization, and downstream-learning interfaces — explicitly contrasted with prior static datasets [§EgoDB].
An iPhone-based capture pipeline is included alongside lab- and partner-operated hardware, lowering the barrier for individual / low-resource contributors [§EgoDB].
The consortium-scale study reports that policy performance generally improves with increased human data, but that effective scaling depends on alignment between the human data and the robot learning objective [Abstract; §Consortium-Scale Study].
Co-training is the concrete instantiation tested: human data and robot data are placed in a unified action space the policy learns from jointly, validated across multiple labs / tasks / robot embodiments under shared protocols [Abstract].
The work is framed in contrast to teleop-only corpora (Open X-Embodiment, DROID, RH20T) — examining human egocentric data as a first-class data source alongside robot data rather than a fallback when teleop is unavailable [§Related Work / Scaling Robot Learning].

Method

EgoVerse is closer to a data + infrastructure paper than a model paper, so the “method” decomposes into three pieces:

Capture stack. EgoVerse-A standardizes on Project Aria Gen 1 glasses across the academic labs (Georgia Tech RL2, Stanford REAL, UC San Diego Wang Lab, ETH Zürich CVG & SRL). The Aria rig provides RGB plus a stereo monochrome SLAM pair, yielding raw streams the EgoDB pipeline post-processes into camera poses, 3D head trajectories, hand tracking, and per-frame dense language tags. The same six flagship tasks are run at every academic lab across 8–12 scenes, with object randomization and demonstrator variation supplying naturally diverse egocentric viewpoints without explicit augmentation. EgoVerse-I uses different captures from Scale and Mecka in the wild; a smartphone-based pipeline is also exposed so individual contributors can add data.

EgoDB infrastructure. A scalable data-management layer underneath both components, providing standardized processing, unified storage formats, controlled access, visualization, and downstream-learning interfaces. The framing as a living dataset (continuous ingestion + versioning) is explicit — old episode hashes are treated as stale and must be refreshed from the SQL episode table before downloading, with Zarr paths and preview MP4s regenerated on reprocessing (per the public GitHub release notes).

Consortium study. Co-training experiments are run across multiple labs, multiple tasks, and multiple robot embodiments under shared protocols. Human demonstrations and robot teleop data are unified in a single action space the policy ingests, and scaling curves are measured against varied human-to-robot data ratios. The companion EgoMimic codebase (GaTech-RL2/EgoVerse on GitHub) ships ACT, EgoMimic (HPT-based), and π-style algorithm implementations on a Hydra + PyTorch-Lightning training stack with DDP.

Results

Headline corpus stats: 1,362 hours, ~80k episodes, 1,965 tasks, 240 scenes, 2,087 demonstrators [Abstract].
The consortium-scale co-training study reports that policy performance generally improves with increased human data across labs / tasks / embodiments, but effective scaling depends on alignment between human data and the robot learning objective — i.e. the gain is not unconditional in dataset size [Abstract].
Per the project page, validated robot embodiments span Georgia Tech RL2 (object-in-container, ID/OOD), Stanford REAL (cup-on-saucer, bimanual precision), UC San Diego Wang Lab (bag-grocery, long-horizon), and ETH Zürich CVG & SRL (bag-grocery, long-horizon) [egoverse.ai project page].
Specific quantitative gains from the co-training study were not retrievable at filing time due to arXiv rate limiting; the abstract characterizes the effect qualitatively. The paper is positioned as “one of the first large, cross-lab validations that combining human and robot data improves robot performance” per the Scale AI release blog.

Why it’s interesting

For Luma’s world-model and embodied-AI tracks, EgoVerse is the first filed consortium-style human-data initiative — distinct in shape from the unilateral commercial scaling plays it shares the room with. Build AI’s Egocentric-1M — largest egocentric video dataset (Build AI / Eddy Xu announcement) (Egocentric-1M, ~1M hours of factory POV from custom glasses, single-company) and Build AI's early egocentric release — 400k action labels, 2.5k clips, 2× open-source dataset size (Eddy Xu tweet, Oct 22 2025) (the earlier 400k-action release) compete on raw scale; EgoVerse is two orders of magnitude smaller (1,362 h) but trades scale for policy-ready annotations (camera poses, 3D head tracking, 21-keypoint hand skeletons, dense language) and a controlled cross-lab experimental protocol that lets the field actually measure human-to-robot transfer rather than just train a model and hope. The EgoVerse-A/I split is the explicit reification of the scale-vs-control trade-off Build AI dodges by collapsing onto scale. Complements Action100M: A Large-scale Video Action Dataset which scales VLM-derived action annotation on internet video — Action100M and Egocentric-1M both aim at the substrate (passive observation at scale), while EgoVerse is built around transfer experiments (does this human data actually train better policies?). Closest neighbor in the multi-embodiment-transfer space is the teleop-side Unitree open-sources UnifoLM-WBT-Dataset — humanoid whole-body teleoperation dataset (UnifoLM-WBT humanoid whole-body teleop), and the explicit-simulation-physics counterpart is the new The Role of Simulation in Scalable Robotics, Genesis World 1.0, and the Path Forward / Genesis World — Simulation platform for general-purpose robotics & embodied AI learning stack. The “living dataset with phone-based capture pipeline” framing also touches the data-layer thesis in Rerun — The Data Layer for Physical AI.