VTAP Gripper: Synergizing Fingertip Sensing and a Visuo-Tactile Active Palm for Dexterous In-Hand Manipulation

TL;DR

VTAP is a three-finger tactile-reactive gripper that pairs FlexiTac tactile-array fingers with an actuated bi-modal visuo-tactile active palm — a palm that switches between long-range vision (before contact) and contact-rich tactile sensing (after occlusion). The design argues that high dexterity does not require replicating the human hand: coordinated finger–palm interaction plus multi-modal sensing can substitute for high-DoF anthropomorphic designs. Alongside the hardware, the paper contributes a staged, gesture-conditioned retargeting framework that maps human hand motion onto the heterogeneous three-finger gripper for teleoperation. Validated on YCB / fragile grasping (93.3%), in-hand syringe reorientation, singulation of clustered objects down to 3 mm, and autonomous vision-then-tactile peg-in-hole insertion (14 mm peg into 15 mm hole with 1 mm tolerance, 7/10 trials). Accepted to IROS 2026.

Key claims

A visuo-tactile active palm — combining a camera view before contact with tactile feedback after occlusion — extends manipulation capability beyond what fingertip-only tactile sensing supports, particularly for tasks requiring pre-contact localization followed by contact-rich feedback [§Abstract, §Method].
Tactile-reactive grasping across a YCB set and fragile objects reaches 93.3% success while remaining damage-free on fragile items — supporting the claim that per-taxel finger tactile feedback plus palm coordination is sufficient for reactive grasping without a fully anthropomorphic hand [§Experiments].
Autonomous vision-to-tactile peg-in-hole insertion succeeds 7/10 trials with 1 mm tolerance (14 mm peg, 15 mm hole) by using the palm camera to localize/grasp the prism, then switching to tactile scanning with edge detection + circle fitting to find the hole [§Experiments].
In-hand singulation of clustered small objects down to 3 mm diameter is achievable by mapping coordinated thumb–index rolling and shearing motions onto the tactile-array fingers, generalizing across shape and stiffness [§Experiments].
A staged, gesture-conditioned retargeting framework closes the embodiment gap between a five-fingered human hand and the three-finger VTAP: gestures first select a grasp mode, then an optimization maps hand motion to gripper actuation — enabling teleoperated syringe manipulation and in-hand singulation that direct joint-to-joint retargeting cannot express [§Method].
Whole-system evaluation covers reactive grasping, fragile-object handling, multi-axis in-hand reorientation, syringe reorientation + plunger actuation, and autonomous visuo-tactile peg insertion — offered as a reference architecture for dexterous gripper design and contact-rich data collection to support downstream learning-based approaches [§Experiments, §Discussion].

Method

VTAP integrates three components on a single gripper. (1) Compliant, reconfigurable fingers equipped with tactile array sensors (FlexiTac-style per-taxel force/pressure readouts) — providing per-taxel contact feedback on the distal contact patches. (2) A Visuo-Tactile Active Palm — an actuated palm surface carrying both a camera and a tactile sensing area, that can present the camera view for pre-contact localization and then rotate/translate to a tactile-sensing configuration once contact begins and occlusion hides the object from external cameras. (3) A staged, gesture-conditioned teleoperation retargeting pipeline: a gesture classifier over the human hand chooses one of several grasp modes (pinch, three-finger power, palm cup, etc.), each of which comes with its own optimization objective that maps human wrist + finger motion to VTAP joint targets. Autonomous behaviors compose these primitives — e.g., peg insertion uses vision for coarse localization + grasp, then hands off to a tactile-driven local search with edge detection and circle fitting.

Results

Tactile-reactive grasping: 93.3% success across YCB + fragile objects, damage-free [§Experiments].
Autonomous vision-then-tactile peg-in-hole insertion: 7/10 trials with 14 mm peg into 15 mm hole (1 mm tolerance) [§Experiments].
In-hand singulation: works down to 3 mm object diameter across varied shapes and stiffnesses [§Experiments].
Teleoperated syringe reorientation + plunger actuation demonstrated end-to-end under the gesture-conditioned retargeting framework [§Experiments].
Multi-axis in-hand reorientation demonstrated [§Experiments].
Headline framing: “high dexterity does not require replicating the human hand” — offered as a design-space counterpoint to high-DoF anthropomorphic hands like the mimic M1 or Shadow Hand [§Discussion].

Why it’s interesting

VTAP is the first filed hardware system that operationalizes two of the design principles the wiki has been triangulating from separate directions. On the tactile side, it sits alongside Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks — whose simulator ablation argued that placement dominates sensor type and specifically that palm/proximal taxels are a bigger marginal win than upgrading fingertip resolution — by shipping a real hardware instance where the palm is an active tactile surface, not just a placement extension. On the retargeting side, VTAP’s staged, gesture-conditioned mapping is a fifth position on the interface-location axis catalogued in Human-to-Robot Retargeting: unlike Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots (drop rotation to 3-DoF) or Cross-Embodiment Robot Manipulation via a Unified Hand Action Space (canonical fingertip-sphere interface), VTAP switches retargeting objective per grasp mode — a discrete-mode-select then continuous-optimize decomposition motivated by the categorical mismatch between five-finger human and three-finger gripper morphologies. Complements Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction (Yunzhu Li is an author on both) by moving the “tactile-reactive manipulation” agenda from a fully-actuated anthropomorphic hand (Tactile-Rex) to a lower-DoF three-finger gripper — a data point that argues DoF count is negotiable if palm/finger coordination is designed in.