RoboTok: An Internet-Scale Data Engine for Human Demonstration Retrieval and Dexterous Manipulation Learning

TL;DR

RoboTok (Rice RobotPI Lab) is a retrieval engine over web human-manipulation video for supplying training demonstrations to dexterous-robot policies. Given one query human-demonstration video, it returns other videos that share underlying manipulation motion rather than visual appearance or text labels, by comparing canonicalized 3D hand trajectories expressed in an estimated actor-centered reference frame. On retrieval benchmarks it recovers more manipulation-relevant demonstrations than prior robot-data retrieval methods, and using the retrieved videos to guide policy training improves simulated dexterous-task success. The framing is that internet video is a continuously-growing supervision source that already exists — the missing piece is a motion-aware index over it.

Key claims

Representing each video as canonicalized 3D hand trajectories in an actor-centered reference frame makes manipulation behaviors comparable across camera viewpoint, scene appearance, and partial actor occlusion, in contrast to label- or visual-embedding-based retrieval [§Abstract].
The learned latent motion space over 3D hand trajectories is compact enough to support efficient search and continual indexing over internet-scale video collections [§Abstract].
RoboTok retrieves more manipulation-relevant demonstrations than existing robot-data retrieval approaches on the paper’s retrieval benchmarks [§Abstract].
Simulated robots trained with RoboTok-retrieved demonstrations complete manipulation tasks with higher success than the baselines evaluated [§Abstract].

Method

Videos are represented by 3D hand trajectories reconstructed from monocular RGB and re-expressed in a per-clip actor-centered reference frame — the estimated coordinate system of the person performing the manipulation, so trajectory shape describes what the hand does relative to the body rather than the camera. These canonicalized trajectories are embedded into a latent motion space in which nearest-neighbor lookup returns videos that share how the hand moves over time, and the embedding is compact enough for continual indexing as new web videos arrive. Query and corpus videos both go through the same reconstruction + canonicalization + embedding stack; retrieval is by motion similarity in the learned space.

Results

Reported on retrieval benchmarks: RoboTok recovers more manipulation-relevant demonstrations than existing robot-data retrieval methods (specific baselines and numbers not disclosed in the abstract). Downstream policy training on RoboTok-retrieved video improves simulated-robot task success over the same baselines. Code (github.com/Rice-RobotPI-Lab/RoboTok-Code), data + models (huggingface.co/Rice-RobotPI-Lab/robotok-public), and project page (rice-robotpi-lab.github.io/RoboTok) released.

Why it’s interesting

RoboTok sits in the data-engine slot of the human-to-robot pipeline rather than the interface-location debate. EgoInfinity: A Web-Scale 4D Hand-Object Interaction Data Engine for Any-View Robot Retargeting and Video-to-Action Robot Learning EgoInfinity stakes the same “web-scale autonomous video-to-action conversion” position but as a forward pipeline that converts everything upstream; RoboTok complements it by making the substrate searchable — you retrieve motion-matched videos on demand rather than curating a fixed corpus like Introducing Index: Building The World's Largest and Most Diverse Physical Dataset Figure Index, HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining HumanScale, or EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data EgoScale. The 3D-hand-trajectory-as-key choice sits between Do as I Do: Dexterous Manipulation Data from Everyday Human Videos Do-as-I-Do’s kinematic-reconstruction-as-supervision recipe (they use the trajectory as the training signal) and MimicLabs’ finding in What Matters in Learning from Large-Scale Datasets for Robot Manipulation that targeted alignment of retrieved demonstrations beats naive scale — RoboTok is a concrete algorithmic answer to “targeted on what?” (motion similarity in actor-centered 3D-hand space), and the data-side counterpart to LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition LUCID’s motion-first intent abstraction on the model side. The actor-centered canonicalization is the load-bearing move: it inherits the same insight as Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots‘s head-camera-frame wrist reduction — collapse the retargeting-relevant channels into a low-dimensional embodiment-invariant frame — but applies it as a retrieval key rather than an action target.