Tactile sensing for manipulation

TL;DR

Recipes for making touch a first-class ingredient of dexterous manipulation policies — either by ingesting high-frequency contact signals directly, by inferring contact from vision, or by simulating tactile sensors at scale so hardware and policy choices can be ablated. The three filed instances split along that design axis: Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction handles the policy side (variable-rate MoT + temporal tactile VQ-VAE on real hardware), PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images handles the vision-only substitute (weakly-supervised per-pixel fingertip pressure from RGB), and Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks handles the upstream tooling question (GPU-parallel unified tactile simulator with 8 sensor abstractions + explicit placement / resolution / noise / abstraction ablations across three dexterous tasks). Directly addresses the sensorimotor-axis counter-position the VLA Models cluster keeps circling back to — that pure VLM-plus-action-head recipes under-weight contact dynamics.

Key claims

Proprioception is insufficient for contact-rich dexterous tasks — even the cheapest binary-contact tactile abstraction beats a no-tactile baseline on all three tested tasks (Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks §4.1, Fig. 7).
Placement dominates sensor type: whole-hand coverage substantially outperforms fingertip-only, and adding taxels on the palm and proximal phalanges is a bigger marginal win than upgrading the fingertip sensor (Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks §4.1). This directly contradicts the de-facto convention of commercial tactile hardware, which concentrates spatial resolution at the distal fingertip pad.
Per-taxel force/torque is the recommended default abstraction across dexterous tasks, matching or outperforming binary contact, contact depth, elastomer displacement, and proximity in aggregate — with proximity edging it out only on tasks where the object approaches the hand rather than being already grasped (Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks §4.1).
Resolution matters far less than coverage: ~200 taxels distributed across the whole hand suffice for the three tested dexterous tasks (Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks §4.1).
Elastomer marker displacement (the GelSight-style abstraction) underperforms force/torque when local-force reading is what the task needs — displacement at one taxel is a function of indentation and shear at neighbouring taxels, blurring the local force vector (Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks §4.1).
The rate mismatch between high-frequency touch (e.g. kHz-class fingertip sensors) and slower vision/language streams can be handled by a variable-rate Mix-of-Transformer with a per-modality tactile encoder, without sacrificing the capabilities of an existing VLA (Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction project page).
Discrete tokenisation via a temporal tactile VQ-VAE turns raw touch into a code stream the policy can attend over — contrasted on the project page with prior “static” tactile encoders — and delivers >30% absolute average success-rate improvement over the strongest baseline across 12 delicate-force / deformable-object tasks (Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction).
Fingertip pressure can be estimated from a single RGB image by an SE-ResNeXt-50 + FPN encoder-decoder trained with weak “press finger X at force Y” contact labels plus adversarial domain alignment — reaching 41.9% contact IoU / 27.5% volumetric IoU on textured overlays and human-annotator-parity 80.5% contact accuracy on in-the-wild surfaces where no pressure sensor could be mounted (PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images Table 2).
Weakly-supervised prompts (“press ring finger at low force”) can substitute for instrumented ground truth as pressure supervision on 100+ natural surfaces — a scaling recipe that sidesteps the “each sensor defines a new robot” bottleneck the simulator work also flags (PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images §3.1).
Sim-to-real transfer works for tactile policies at the fingertip-aggregate level: a policy trained on the agg_bool observation type in simulation matches its simulation success rate when deployed on a real XHand1 whose only tactile channel is fingertip-aggregate force (Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks §4.1).
Temperature is a plausible tactile modality but current real-hardware temperature sensors are too insensitive to learn even a hot-ball-among-8-distractors discrimination task from touch alone — the sensitivity gap between sim and hardware is the blocker, not the algorithm (Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks §3.3, Fig. 6).

Recent contributions

[2026-07-16] Solving Dexterity: A Full-Stack Approach (mimic hand M1 + wearable U1): mimic hand M1 pursues a hardware recipe orthogonal to Tactile Genesis’s placement-dominates-type finding: a dual motor+joint encoder scheme derives ≤0.1 N unidirectional contact-force estimates from motor current on every joint (effectively whole-hand implicit force sensing), then adds dedicated fingertip tactile sensors for tangential shear and higher spatial resolution only at the distal contact points. The argument is that <0.05 Nm joint backdrivability lets every actuator double as a force sensor, sidestepping the taxel-placement problem — but no head-to-head with dedicated palm / proximal-phalanx taxels is offered. Complements Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks‘s simulator-driven placement study by testing the alternative hypothesis that backdrivable actuation obviates whole-hand taxel coverage.
[2026-07-14] Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks: GPU-parallel tactile simulator integrated into Genesis World with 8 sensor abstractions (contact / depth / kinematic force-torque / elastomer / proximity / temperature / audio) under a common interface, 20,000+ parallel envs and 1,000+ taxels on a single GPU (3-20× throughput over TacSL / Tacmap / HydroShear / FOTS). Ablations across three dexterous tasks × three placements × three resolutions × clean/noisy noise conditions yield the concrete design directives: placement > type, per-taxel force/torque is the default, 200 whole-hand taxels suffice. Sim-to-real validation on the real XHand1 confirms the fingertip-aggregate abstraction transfers.
[2026-06-20] Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction: Tactile-Reactive Dexterous Hand — first policy-side answer on the wiki. Variable-rate Mix-of-Transformer + temporal tactile VQ-VAE fuses kHz-class touch with slower vision/language, on top of an existing VLA. 100 h motor-primitive-prioritized teleop dataset; >30% absolute average success rate over the strongest baseline across 12 delicate-force / deformable-object tasks. Only a project page at filing time — no paper or code yet.
[2026-06-18] PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images: PressureVision++ — the vision-only substitute. Weak “press finger X at force Y” contact labels + adversarial domain alignment supervise a per-pixel pressure predictor from a single RGB image; 41.9% contact IoU / 27.5% volumetric IoU on textured overlays, 80.5% in-the-wild contact accuracy matching human annotators. 2.9M-frame ContactLabelDB with 51 participants × 106 surfaces released.

Open questions

Does the policy-side answer converge on variable-rate multimodal fusion (Tactile-Rex’s MoT) or on a unified token stream with per-modality position encodings? No filed paper compares the two.
Which downstream tasks actually benefit from per-taxel force/torque vs. per-link aggregate force? Tactile Genesis shows near-continuous-contact tasks (in_hand_repose) reward per-taxel, but the real-XHand1 hardware only exposes aggregate — so the win is real-world-bounded by what hardware exists.
Is vision-only contact inference (PressureVision++) sufficient as the VLA’s tactile channel, or is the latency and occlusion sensitivity a hard blocker that keeps first-class touch sensors on the critical path? No filed head-to-head.
Does the “coverage over resolution” finding from Tactile Genesis generalize to bimanual manipulation and to tasks that involve tool-mediated contact (e.g. screwdriver, chopsticks) where the informative contact point sits outside the hand? Screwdriver-task results in the paper hint the missing channel there may be temporal integration, not spatial density.
Elastomer marker displacement is a first-class output of many vision-based tactile sensors (GelSight, DIGIT). If it underperforms force/torque per-taxel on the dexterous tasks Tactile Genesis tested, does that mean the current elastomer-simulator development effort is targeting the wrong downstream metric, or that the tasks need to change?
What is the right observation-space contract between a tactile front-end and a VLA backbone? Tactile-Rex’s temporal VQ-VAE yields discrete codes; PressureVision++ yields per-pixel dense pressure maps; Tactile Genesis exposes per-taxel force vectors. None of these have been unified across policies.
Does simulator-first tactile hardware design (i.e., using Tactile Genesis’s ablations to spec whole-hand coverage before building the hardware) actually shorten the hardware-iteration loop, or does the sim-to-real gap on non-fingertip surfaces re-open the loop anyway?

Papers

2026-06

Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks — Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks (published 2026-06-21)
Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction — Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction (published 2026-06)

2023-01

PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images — PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images (published 2023-01-05)