FINGR: Learning Dexterous Hand Control for Real-World Rubik's Cube Solving

TL;DR

FINGR trains a single-hand dexterous policy that solves scrambled 2×2×2 Rubik’s cubes in the real world by combining finger-relative point-cloud geometry with auxiliary prediction of future interaction signals. A shared point encoder expresses the cube in each fingertip’s frame and aggregates points without any cubie indexing; learned future tokens read from the same encoder and are supervised at multiple time scales on contact-force changes, layer-turn progress, and finger-joint displacement. The resulting representation conditions a flow policy that outputs finger actions, hitting 99.0% success over 300 turn attempts on real hardware (vs 79.7% for the base flow policy) and solving all ten scrambled cubes end-to-end in ~137 seconds.

Key claims

Expressing the cube’s point cloud relative to each fingertip and aggregating points through a shared encoder — rather than relying on per-cubie indexing — gives the policy an embodiment-tied geometric substrate for contact-rich turn execution [Abstract, §Method].
Learned future tokens that share the observation encoder and receive supervision for contact-force changes, layer-turn progress, and finger joint displacement at multiple time scales act as an auxiliary objective that shapes the shared representation beyond what the flow action loss alone induces [Abstract].
A flow policy conditioned on this representation reaches 99.0% success over 300 single-turn attempts on a real dexterous hand, versus 79.7% for the base flow policy — a 19.3-point absolute gap attributable to the finger-relative geometry + future-interaction-prediction recipe [Abstract].
Integrated with grasping and table-assisted regrasping, the policy solves all ten scrambled 2×2×2 cubes end-to-end in a mean complete-system time of ~137 seconds [Abstract].

Method

FINGR (Future-supervised Interaction Network with Geometric Representations) targets the specific skill of executing successive layer turns on a Rubik’s cube held in a single dexterous hand — a sustained contact-rich problem where some fingers must support the cube while others push a moving layer, release, and reset. The architecture has two coupled parts. A shared point encoder reads the cube’s point cloud and transforms it into each fingertip’s frame, then aggregates points without depending on cubie labels, giving the downstream policy a geometric view indexed by the hand rather than by the object. Learned future tokens share this encoder’s observation path and receive auxiliary supervision on three future-interaction signals — contact-force changes, layer-turn progress, and finger joint displacement — at multiple time scales. The flow policy consumes the resulting representation and generates finger actions directly.

Results

Single-turn success: 99.0% over 300 turn attempts on a real dexterous hand, compared with 79.7% for the base flow policy baseline [Abstract].
Full-system solving: all ten scrambled 2×2×2 cubes solved when combined with grasping and table-assisted regrasping, mean complete-system time ≈ 137 seconds [Abstract].

The abstract does not disclose the specific hand hardware, point-cloud sensor, number of demonstrations, or ablations isolating each future-prediction target individually; a reader interested in those will need the full PDF.

Why it’s interesting

The future-interaction-prediction auxiliary objective is a close relative of the auxiliary-loss recipes already circulating on the wiki — Act with Intent: Distilling Behavior Intent for Vision-Language-Action Models INDI (intent distillation) and From Foundation to Application: Improving VLA Models in Practice (LingBot-VLA 2.0) LingBot-VLA 2.0 (WFM-as-auxiliary-proxy) — but applied to a narrow contact-rich sustained-manipulation domain rather than a general VLA, and with the predicted targets being physical interaction variables (force deltas, turn progress) rather than video pixels or latent world states. The finger-relative point-cloud substrate is a sibling of Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction T-Rex’s per-finger temporal VQ-VAE and T-Rex: Tactile-Reactive Dexterous Manipulation T-Rex’s per-finger encoders — all three recipes commit to a per-fingertip observation decomposition rather than a hand-global state — but FINGR gets its contact-aware shaping from predicted force futures instead of ingested tactile signals, making it the first filed instance on the wiki of a geometry-only policy that recovers contact-rich performance via supervision on future contact. Contrast with The Cartesian Hand: In-Hand Manipulation with All-Linear Fingers The Cartesian Hand, which attacks the same single-hand in-hand manipulation regime from the opposite end — redesigning the mechanism (all-linear fingers) rather than the learning signal.