OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction

ListenOakInk: Teaching Machines How Hands Hold Things11:56

Show notes

A look at OakInk, the SJTU MVIG knowledge repository from CVPR 2022 that pairs an object-affordance base (Oak) with an interaction base (Ink) to reach 50,000 distinct, intent-oriented hand-object interactions. We unpack the multi-sensor capture rig, the Tink transfer pipeline that turns 1,000 recorded grasps into 49,000 virtual ones, and why "contactness" stress patterns and human-to-human handover make this dataset distinctive. We close on what a contact-first data recipe means for anyone building visual generative models of interaction. Sources: - OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

OakInk (SJTU MVIG, CVPR 2022) is a hand-object interaction knowledge repository built around two interrelated bases: Oak, an object-affordance knowledge base of 1,800 scanned household objects tagged with part-level segmentation, taxonomy (maniptool vs functool), and 30 attribute phrases; and Ink, an interaction knowledge base of 50,000 distinct affordance-aware, intent-oriented hand-object interactions with MANO hand poses and 6-DoF object poses. The image half (OakInk-Image) is a 230K-frame real capture from 12 subjects performing 5 intents (use / hold / lift-up / hand-out / receive) across 100 objects under a synced 4-RGB-D + 8-camera MoCap rig; the geometry half (OakInk-Shape) extends the 1K recorded interactions to 49K virtual counterparts via Tink, a learn-fitting pipeline that transfers hand-object interactions across objects of similar affordance using DeepSDF shape interpolation, iterative contact mapping, and pose refinement with an interpenetration cost.

Key claims

OakInk-Image is the first HOI dataset to jointly provide dynamic interaction, human-to-human handover, hand-object contact regions, intent labels, and full 6-DoF object pose across 100 objects × 12 subjects × up to 5 intents [Table 1].
Existing real HOI datasets are either object-poor (DexYCB: 20 objects) or interaction-sparse (ContactPose: 2.3K distinct interactions in 2.9M frames, 0.08%); synthetic ones (ObMan) don’t reflect human grasp distribution. OakInk closes both gaps by combining 100 recorded objects with 1,700 affordance-transferred virtual counterparts to reach 50K distinct interactions [§1, §3.4, Table 1].
Cross-dataset training on OakInk-Image + HO3D consistently reduces hand MPJPE on DexYCB vs HO3D-only training, indicating OakInk-Image carries pose-supervision signal complementary to HO3D [§3.4 Cross Validation, Table 2].
Tink’s three-module pipeline — DeepSDF latent interpolation (N=10 landmarks) + explicit contact mapping via ICP + iterative pose refinement with contactness, anatomical rotation constraints, and SDF-based interpenetration cost — produces hand-object transfers that pass perceptual plausibility evaluation by 5 volunteers [§3.3, Fig. 5].
Contact is recorded as stress patterns: a ring-shape decreasing “contactness” value seeded at anchor–vertex closest points (per 17 hand parts, 5 mm threshold), yielding a spread field rather than binary contact — motivated as an imitation of elastic deformation that MANO + rigid objects cannot represent directly [§3.2.3].
Hand pose annotation is crowd-sourced 2D keypoints (21 per view, 4 views) then optimized to 3D via multi-view re-projection, geometric consistency, temporal smoothing, and silhouette constraints against a differentiable MANO layer [§3.2.3, Eq. 1].

Method

The pipeline is three-stage. Oak base collects 1,800 household objects from online vendors, ShapeNet, YCB, and ContactDB, organizes them under a two-level taxonomy (maniptool/functool × 32 WordNet categories), and tags each part with attributes drawn from a 30-phrase pool built by consensus across 10 volunteers. Ink base records human demonstrations in a 3 m³ cuboidal rig with 4 RealSense D435 cameras + 8 OptiTrack Prime 13W cameras; 12 subjects perform each of up to 5 intents against 100 objects for 5 s. Hand pose is recovered from crowd-sourced multi-view 2D keypoints + a MANO-layer optimization; object pose from surface-attached reflective markers tracked in MoCap and transformed into the multi-camera frame. Contact and stress (“contactness”) are then computed per hand part from the paired poses. Tink transfers the ~1K recorded interactions onto the remaining 1,700 virtual objects: (i) train a per-category DeepSDF, interpolate 10 SDF landmarks between source and target; (ii) propagate contact labels stepwise via ICP along the landmark chain; (iii) optimize hand pose + shape + wrist to match transferred contactness while respecting anatomical rotation-axis limits (five MCP joints) and penalizing SDF-negative penetrations, Adam × 1000 iterations. Perceptual plausibility is verified by 5 human raters.

Results

OakInk-Image: 230,064 RGB-D frames, 12 subjects, 100 objects across 32 categories, 4 views, up to 5 intents per object; ~1K distinct interactions [§3.4].
OakInk-Shape: 50,000 distinct hand-object interactions after Tink extends the 1K recordings to 1,700 virtual counterparts [§3.4].
Cross-dataset pose estimation: training on OakInk-Image + HO3D beats HO3D-only on DexYCB test MPJPE [Table 2].
Table 1 comparison against 9 prior HOI datasets (ObMan, YCBAfford, FPHAB, HO3D, ContactPose, GRAB, DexYCB, H2O, plus OakInk-Image / OakInk-Shape): OakInk is the only entry with all five of {intent labels, object pose, dynamic interaction, handover, contact} set.
Handover: paired giver–receiver sequences with the giver setting the receiver’s expected use pose — the first HOI dataset to record human-to-human handover under mocap-grade supervision.

Why it’s interesting

OakInk sits squarely in the substrate layer that Human-to-Robot Retargeting tracks — Yu Xiang’s curated 12-dataset thread (Yu Xiang — 12 hand-object interaction datasets worth knowing for robot learning) names OakInk v1 and v2 as one of two “grasp/object-aware pose” slots that retargeting recipes compete over. The combination of scanned 3D models (which alexm explicitly uses in this thread), dynamic multi-intent trajectories, and affordance-transferred grasps at 50K scale makes it distinct from lab-captured competitors: DexYCB is 20-object with no handover (C2Dex: Contact-Consistent Reconstruction and Retargeting for Dexterous Manipulation from Monocular Video reports headline numbers on DexYCB and TACO), HOT3D (HOT3D — Egocentric Multi-View Dataset for 3D Hand-Object Interaction Understanding) is higher-fidelity egocentric but PBR-scanned-object-restricted, and HOI4D (GPS: Geometric Primary Structure for Articulated Parts Perception in Robot Manipulation) covers longer horizons without handover. The Tink methodology — transfer contact-first, then optimize pose against a shared object-space contact map — is a direct methodological ancestor of C2Dex: Contact-Consistent Reconstruction and Retargeting for Dexterous Manipulation from Monocular Video‘s “object-side contact topology in canonical object space” interface, filed six months earlier as a next-recipe candidate. The Oak affordance graph (30-phrase part-attribute vocabulary) is also a rare structured supervision signal that neither egocentric-video nor UMI-style corpora provide.