HOT3D — Egocentric Multi-View Dataset for 3D Hand-Object Interaction Understanding

TL;DR

HOT3D is Meta’s egocentric benchmark for 3D hand–object interaction: over one million multi-view frames synchronously captured from Project Aria glasses and Quest 3, paired with high-fidelity PBR 3D models of 33 diverse objects, motion-capture ground-truth 3D poses of both hands and objects, and Aria’s native gaze signal. It is an official dataset of the BOP challenge (2024) and covers both model-based and model-free 6-DoF object pose estimation as well as joint hand+object tracking from a first-person viewpoint. For robotics research it is one of the largest publicly-available substrates for training and evaluating hand-object tracking, retargeting, and grasp models on real head-mounted-device streams rather than in simulation.

Key claims

Over one million multi-view egocentric frames of hand–object interactions, captured synchronously from Project Aria glasses and Quest 3 [§Dataset Content].
33 diverse objects released as high-fidelity PBR-material 3D scans from an in-house 3D scanner, enabling model-based 6-DoF pose estimation research [§High-fidelity 3D object models].
Ground-truth 3D poses of both hands and objects captured via optical markers tracked by a professional motion-capture rig — supports joint hand+object tracking supervision [§Accurate ground-truth 3D poses of hands and objects].
Aria stream includes wearer gaze, opening pose-estimation-with-intent research (intent prediction, gaze-driven ROI cropping) that headset-only pipelines cannot address [§Precise eye tracking, indicating wearer gaze].
Recognised as an official dataset for the BOP Challenge in 2024, covering model-based and model-free object detection and pose estimation tasks [§Unlocking new challenges].
Python tools + code samples ship for loading and visualising the multi-view streams together with the ground-truth 3D hand+object poses [§Comprehensive tools].

Method

The dataset is captured from two head-mounted platforms simultaneously (Project Aria glasses for research-grade multimodal sensing including gaze, and consumer Quest 3 for high-throughput multi-view video), so a single interaction can be studied under both sensor stacks. Ground truth is bootstrapped by attaching small optical markers to hands and manipulated objects and tracking them with a professional motion-capture system, then registered against the high-fidelity object scans to produce per-frame 6-DoF hand-and-object poses. Object models are reconstructed with an in-house 3D scanner producing PBR-textured meshes suitable for model-based pose-estimation baselines. Loading + visualisation tooling is exposed as a Python API around the multi-view image streams and 3D ground truth.

Results

The page is a dataset landing rather than a benchmark leaderboard; no numeric baselines are published on it. HOT3D’s role in the wider ecosystem is external: it is a fixture in the BOP Challenge (2024) [§Unlocking new challenges], and downstream work in the wiki uses it as a standard benchmark — e.g. MolmoMotion-AR reports 0.109 m 3D average displacement error on HOT3D vs 0.129 for ObjectForesight and 0.200 for Wan2.2-5B (MolmoMotion: Language-guided 3D motion forecasting), and ReViV reports SOTA across a suite including HOT3D (ReViV: Reconstructing the Viewer and the View in 4D from Monocular Egocentric Video).

Why it’s interesting

HOT3D fills the “real multi-view egocentric + accurate 3D ground truth” slot for hand-object interaction work — a slot that is otherwise served by simulation or by monocular in-the-wild video with weak supervision. It’s a natural upstream for the egocentric-pretraining recipes tracked under Human-to-Robot Retargeting (e.g. Do as I Do: Dexterous Manipulation Data from Everyday Human Videos, HUG: Human Universal Grasping, EgoInfinity: A Web-Scale 4D Hand-Object Interaction Data Engine for Any-View Robot Retargeting and Video-to-Action Robot Learning) — all of which need high-quality 3D hand-object ground truth for supervision or evaluation — and complements the perception-side pose-estimation cluster (Pose Estimation and Motion Capture) where several filed papers (ReViV: Reconstructing the Viewer and the View in 4D from Monocular Egocentric Video, MolmoMotion: Language-guided 3D motion forecasting) already benchmark on it. The dual-headset (Aria + Quest 3) capture is unusual: it lets research disentangle what changes when you swap sensor stacks for the same interaction — a signal that most single-headset egocentric corpora cannot provide.