CosmoH2G: A Hand-to-Gripper Transfer Dataset and Baseline Method for Object Manipulation with Complex Spatial Movements

TL;DR

CosmoH2G attacks the failure mode of existing hand-to-gripper transfer recipes on complex spatial movements (rotations, flips, intricate 3D trajectories) — regimes where existing kinematic retargeting collapses because minor deviations compound. The paper contributes (a) a scalable acquisition pipeline for hand–gripper paired demonstrations centered on a handheld gripper (mirroring the collection-tool-embodiment-invariance recipe used by UMI / mimic / XR-1), yielding a 6,189-episode / 1,254-unique-object dataset with explicitly higher motion complexity than existing benchmarks, and (b) a two-stage baseline framework: Stage I predicts sparse gripper keyframes (initial + terminal); Stage II generates the full continuous action sequence conditioned on those keyframes; a post-optimization step keeps orientation learned but corrects translation drift via grasping heuristics + kinematic consistency. Reported to outperform traditional baselines in both simulation and real-robot experiments on complex spatial manipulation.

Key claims

Naive end-to-end generation of full gripper pose sequences is insufficient for complex spatial trajectories — minor deviations compound rapidly under intricate dynamics, motivating the sparse-keyframe-then-fill decomposition [§Abstract].
A two-stage sparse-keyframe → continuous-action decomposition simplifies the mapping objective: Stage I only predicts initial + terminal gripper keyframes; Stage II generates the continuous action sequence conditioned on those keyframes [§Abstract].
Cumulative drift is mitigated by keeping the gripper’s orientation learned end-to-end while post-optimizing its translation using grasping heuristics + kinematic consistency — a hybrid rather than fully learned or fully classical pose track [§Abstract].
The acquisition pipeline is motion-complexity-first: rigorous protocol prioritizes trajectory complexity (rotations / flips) and leverages a handheld gripper for direct action mimicry, yielding 6,189 episodes across 1,254 unique objects [§Abstract].
Framework enables “stable and precise” hand-to-gripper transfer of complex spatial manipulations in both simulation and real-robot experiments, significantly outperforming traditional baselines [§Abstract].

Method

The recipe is a two-stage generative framework operating on a paired hand–gripper dataset collected with a handheld gripper (the demonstrator drives the actual gripper end-effector rather than mimicking with a bare hand, so action mimicry is seamless). Stage I predicts sparse gripper keyframes — specifically the initial and terminal poses — collapsing the mapping problem to a low-dimensional endpoint prediction. Stage II then generates the full continuous gripper action sequence conditioned on those keyframes. A drift-mitigation step keeps the gripper’s orientation channel end-to-end learned but post-optimizes its translation channel using grasping heuristics and kinematic consistency — the paper’s split matches its observation that translation errors compound faster than orientation errors under complex spatial motion.

The dataset (6,189 episodes, 1,254 unique objects) is explicitly higher-complexity than existing hand-to-gripper corpora and is validated in both simulation and real-robot experiments.

Results

The abstract reports:

Framework “significantly outperforms traditional baselines” in both simulation and real-robot experiments on complex spatial manipulation transfer [§Abstract].
Dataset complexity: 6,189 episodes / 1,254 unique objects, “significantly higher spatial complexity” than existing hand-to-gripper benchmarks [§Abstract].

No per-task numbers, ablations, or baseline names are disclosed in the abstract. Code + dataset marked “coming soon” as of filing.

Why it’s interesting

CosmoH2G stakes out a new position on the Human-to-Robot Retargeting interface-location axis that no filed paper yet occupies: shift the retargeting difficulty from full-trajectory to sparse keyframes + drift-corrected fill, with an asymmetric orientation-vs-translation split (learn one, optimize the other). This is a distinct answer from the recipes already on the concept page — physics-in-loop bilevel (ReActor: Reinforcement Learning for Physics-Aware Motion Retargeting ReActor), 4D kinematic reconstruction (Do as I Do: Dexterous Manipulation Data from Everyday Human Videos Do as I Do), 3-DoF wrist reduction (Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots), or object-side contact topology (C2Dex: Contact-Consistent Reconstruction and Retargeting for Dexterous Manipulation from Monocular Video C2Dex) — and directly addresses the failure mode (compounding drift on complex spatial trajectories) that motion-retargeting recipes typically abstract away. On the collection-tool side it sits in the same lineage as Solving Dexterity: A Full-Stack Approach (mimic hand M1 + wearable U1) mimic and Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training XR-1’s UMI-scale UMI pretraining — collect with the gripper rather than retargeting after the fact — but scoped to a smaller, motion-complexity-curated corpus with an explicit baseline method rather than at pretraining scale.