Yu Xiang — 12 hand-object interaction datasets worth knowing for robot learning

TL;DR

Yu Xiang (UT Dallas IRVL, ex-NVIDIA) posts a 12-item curated directory of hand-object interaction (HOI) datasets useful for robot learning, spanning single-hand RGB pose (FreiHAND), synthetic training (RHD, ObMan), multi-view calibrated capture (HO-Cap, PALM), RGB-D + egocentric real capture (HO3D, H2O-3D, DexYCB), long-horizon action data (HOI4D), grasp/object-aware pose (OakInk v1/v2), bimanual mesh-annotated interaction (ARCTIC), and tool-use / action-centric interaction (TACO). Each entry lists download + code links. The framing argument in the OP is that accurate hand pose is the load-bearing bridge between pixels, objects, contact, and motion for learning from human interaction, and that the hard part is not collection but making pose formats, camera models, coordinate frames, and object annotations comparable across datasets.

Key claims

Hands are the locus where intent becomes action for robot learning (grasping, tool use, bimanual coordination, object handoff, contact recovery), so accurate hand pose is treated as the bridge between pixels, objects, contact, and motion rather than as an annotation detail [OP tweet].
The 12-dataset directory covers distinct substrate slots: FreiHAND (RGB pose baseline with MANO supervision), RHD (synthetic viewpoint/appearance variation), HO-Cap (calibrated multi-view HOI), HO3D + H2O-3D (RGB-D + egocentric real HOI with occlusion), ObMan (large synthetic hand-object with dense supervision), DexYCB (real RGB-D YCB manipulation), HOI4D (large-scale time-unfolding actions), OakInk v1 (object-aware pose + grasp), OakInk v2 (longer object-centric manipulation sequences), ARCTIC (two-hand mesh-annotated), PALM (multi-view HOI for calibrated projection), TACO (tool-use / action-centric with affordance) [tweets 1/12–12/12].
Interoperability — not collection — is claimed to be the hard part: pose formats, camera models, coordinate frames, and object annotations across these datasets are currently not directly comparable, and solving image-based hand pose estimation is argued to be a load-bearing input for robot learning from human video [tweet 13/12 wrap].
Follow-up reply (Raj Patel, HumanArchive) argues mm-level trajectory error for hand pose + SLAM is required for downstream utility, pointing at HA-SLAM and ego500 evaluations as external calibration on this bar [reply].

Method

Not applicable — this is a curated resource thread rather than a system. The organizing schema across the 12 entries is (dataset name, positioning claim, download URL, code URL, exemplar image).

Results

Not applicable — no benchmark numbers reported. The thread’s value is as a scoped substrate directory: 12 named datasets, each with download and code links, positioned within a shared coordinate-and-annotation-comparability problem statement.

Why it’s interesting

The Human-to-Robot Retargeting concept page already tracks a substantial retargeting-recipe debate whose empirical footing is precisely the datasets in this thread — DexYCB and TACO show up as headline evaluation sets in C2Dex: Contact-Consistent Reconstruction and Retargeting for Dexterous Manipulation from Monocular Video (57.78% vs 17.78% on DexYCB, 26.67% vs 10.00% on TACO), HOI4D powers GPS: Geometric Primary Structure for Articulated Parts Perception in Robot Manipulation, and HO3D anchors ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation. Xiang’s list makes explicit the substrate that these papers implicitly compete over. It also complements HOT3D — Egocentric Multi-View Dataset for 3D Hand-Object Interaction Understanding (HOT3D, filed Aug 13) by covering the pre-Aria HOI dataset lineage; HOT3D is a superset in fidelity but this thread names the smaller-but-established benchmarks that comparison numbers still live in. The interoperability claim (“pose formats, camera models, coordinate frames, and object annotations comparable enough to learn from them together”) is a concrete open problem for anyone building multi-dataset hand-pose pretraining substrates cited under Human-to-Robot Retargeting.