ACE-Data-0: Human-Centric Ambient Capture as Embodied Data Engine

TL;DR

ACE-Data-0 (S-Lab NTU + ACE Robotics; Yukang Cao, Ziwei Liu, and collaborators overlapping with ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining and Kairos-HomeWorld — whole-home 3D scene generation with object-level interactivity (ACE Robotics + CUHK MMLab + Shenzhen Loop Area Institute)) is a large-scale human-centric embodied dataset that records ego + multi-view exo video, whole-body + hand motion, object pose, audio, and full-palm tactile data on one clock and in one world frame across two capture sites — a ~30m² desk workspace and a ~200m² furnished apartment. The bet is that fragmenting embodied data across viewpoints, modalities, or scales throws away the joint distribution that policies need to learn from, and that a single-pass synchronized capture pipeline with millisecond-level temporal residuals and <3-px reprojection error gives measured, not estimated annotations that survive occlusion and extreme viewpoints. The planned release covers 150+ hours, 17M+ frames, 75K+ interaction episodes, 200+ task categories, and 50+ participants, gated to non-commercial academic research; a benchmark on 30+ methods across tactile inference, human motion recovery, and hand motion shows existing methods degrade sharply under contact, occlusion, and long horizons.

Key claims

ACE-Data-0 records ego + exo video, human motion (41-joint + articulated hands + SMPL-X), object 6-DoF pose @ 60 Hz, audio, and full-palm tactile grids on a single unified timeline and shared world coordinate frame, so any tracked 3D point can be projected onto any pixel of any view without rerunning the capture pipeline [§What each take contains].
Two capture sites: a ~30 m² desk table-scale setup (8× GoPro @ 0.3–0.5 m, 16× OptiTrack PrimeX 22) and a ~200 m² furnished apartment room-scale setup (8× ZED One, 12× OptiTrack PrimeX 22, at least 4 views on any point) [§How it was captured].
Participants receive goal-level instructions (“prepare a cup of tea and serve it at the table”) rather than step-by-step scripts, so planning, hesitation, and improvisation enter the data by themselves [§What this is].
Human states, object states, and contact are metrically tracked or directly sensed — annotations are measured rather than estimated, staying correct under furniture occlusion, extreme viewpoints, and motion blur where image-based detectors fail [§What this is].
Planned release: 150+ hours, 17M+ video frames, 75K+ interaction episodes, 200+ task categories, 50+ participants, 8+ exocentric plus 4 egocentric fisheye views per moment, with take lengths up to 20–30 minutes for long-horizon chains [§What this is, Table].
Takes come in three families: atomic HOI (1–3 household tasks, ~3 min), chains of HOI (~20–30 min continuous activity ending with the scene tidied back), and human-scene interaction (whole-body motion + furniture contact, almost no objects, ~5 min) [§What each take contains].
Temporal synchronization: all devices registered to the OptiTrack 60 Hz clock by photographing a nanosecond-resolution clock displayed on the mocap host, giving millisecond-level residuals within a single mocap frame [§How it was captured].
Spatial calibration: ArUco board with retroreflective corners bridges exocentric cameras that share no field of view (median reprojection error < 3 px); headset is solved by hand-eye calibration against its tracked rig (~2 px), so egocentric camera poses are measured and do not drift [§How it was captured].
Hand pose in table-scale is triangulated from 8 exo views + manually refined; in room-scale it uses Manus mocap gloves @ 60 Hz [§How it was captured, Table].
Benchmark holds out 10 hours as test set and evaluates 30+ published methods across tactile inference from video, human motion recovery, and hand motion from ego + exo views; existing methods degrade sharply under contact, occlusion, egomotion, and long horizons [§Benchmark].
Empirical finding from the benchmark: strong per-frame pose accuracy does not imply an accurate world-frame trajectory, and egomotion (not finger articulation) dominates the error in egocentric hand reconstruction [§Benchmark].
Access is gated and non-commercial academic only, personal and non-transferable; data files are not yet published at the HF page but access is granted now on institutional-email requests [§Access].
Two-site coverage is the acknowledged main limitation — limited variation in layouts, furnishings, and lighting; articulated-mechanism state, fluids, and deformables are not annotated; capture gear is visible in the recordings [§Limitations].

Method

ACE-Data-0 is a dataset release, not a model, so the method is the capture and calibration pipeline. Participants wear an ACE-Ego-Head-V02 Lite headset (4 fisheye cameras, IMU, 5 tracked markers), a 41-marker mocap suit, and full-palm tactile gloves; they operate inside a room-scale or table-scale rig instrumented with 8+ synchronized exocentric cameras (GoPros at the table, ZED Ones in the apartment) and 12–16 OptiTrack PrimeX 22 mocap cameras. Tracked objects are scanned and marked in advance so 6-DoF pose is directly sensed rather than estimated. All streams — egocentric fisheye @ 20 FPS + IMU + headset pose, exocentric video @ 30 FPS + intrinsics + world pose, 41-joint skeletons + articulated hand poses + SMPL-X, object pose @ 60 Hz + 2D/3D boxes in every view, multi-source audio, full-palm pressure grids — are registered to the OptiTrack 60 Hz clock via a nanosecond-clock photographing trick, and to a shared world frame via an ArUco board that bridges non-overlapping exo cameras. The release ships in two directory trees: Room-scale/ (per-take motions/, raw/, timeline/, videos_and_annotations/, caption/) and Table-scale/ (with HOI/ and Tactile/ sub-trees, LeRobot-compatible parquet chunks under data/, object meshes under objs/, and raw sensor records under raw/). The parquet data provides timeline indices and raw-row ranges; raw sensor records stay in raw/; video bytes stay in videos/. The companion technical report (arXiv 2607.28625) covers the capture protocol, calibration, and annotation pipeline in full; per-modality error tables and calibration ablations are in the report and were not retrievable in full from the HF card at filing time.

Results

Scale: 150+ hours of recorded activity, 17M+ frames, 75K+ interaction episodes, 200+ task categories, 50+ participants, 2 capture environments, 8+ exocentric plus 4 egocentric fisheye views per moment (planned release; to be verified in final release manifest) [Table §What this is].
Time synchronization: millisecond-level residuals within a single mocap frame across all devices [§How it was captured].
Space calibration: median reprojection error < 3 px for exocentric cameras via ArUco bridging; ~2 px for the headset via hand-eye calibration [§How it was captured].
Benchmark scope: 10-hour held-out test set, 30+ published methods evaluated across tactile inference from video, human motion recovery, and hand motion from ego + exo views [§Benchmark].
Benchmark headline: existing methods degrade sharply under contact, occlusion, egomotion, and long horizons; strong per-frame pose accuracy does not imply accurate world-frame trajectory; egomotion — not finger articulation — dominates the error in egocentric hand reconstruction [§Benchmark].
Per-method scores, per-modality error tables, and exact per-task category breakdown are in the technical report and were not retrievable at filing time.
HF downloads at filing: 27,658 in the last month (dataset card, no file contents yet) [HF page footer].

Why it’s interesting

ACE-Data-0 is the substrate for the training recipe ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining proposed and complements the group’s world-modeling side (Kairos-HomeWorld — whole-home 3D scene generation with object-level interactivity (ACE Robotics + CUHK MMLab + Shenzhen Loop Area Institute) Kairos-HomeWorld). The distinguishing lever is not scale — the volume is smaller than HumanNet: Scaling Human-centric Video Learning to One Million Hours‘s million-hour corpus, Lightwheel AI open-sources EgoSuite-Open100K — 100K-hour egocentric human dataset with hand and body pose‘s EgoSuite-Open100K, or Egocentric-1M — largest egocentric video dataset (Build AI / Eddy Xu announcement)‘s Egocentric-1M — but synchronized multimodal density on measured, not estimated, annotations. Every prior egocentric-video-for-robotics dataset in the wiki (HumanNet, Ego4D, EgoSuite-Open100K, Egocentric-1M, EgoScale’s 20,854 hours from EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data) pays for scale by giving up either tactile ground truth, or object 6-DoF ground truth, or exo views, or all three; ACE-Data-0 pays for those by giving up scale and by constraining to two sites. The benchmark result that egomotion dominates egocentric hand-reconstruction error is a direct empirical vindication of the ActiveMimic: Egocentric Video Pretraining with Active Perception argument that viewpoint should be a first-class action, and of Do as I Do: Dexterous Manipulation Data from Everyday Human Videos‘s bet that upstream kinematic reconstruction quality is the bottleneck. It also complements the tactile-side effort in Open-X-Tactile — community effort to build the world's largest heterogeneous tactile manipulation dataset Open-X-Tactile: where Open-X-Tactile targets a heterogeneous pooled tactile corpus across labs, ACE-Data-0 gives a unified-with-vision-and-motion tactile stream at a single lab — the two datasets probably want to be combined rather than compared.