Tactile sensing for manipulation

TL;DR

Recipes for making touch a first-class ingredient of dexterous manipulation policies — either by ingesting high-frequency contact signals directly, by inferring contact from vision, or by simulating tactile sensors at scale so hardware and policy choices can be ablated. The three filed instances split along that design axis: Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction handles the policy side (variable-rate MoT + temporal tactile VQ-VAE on real hardware), PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images handles the vision-only substitute (weakly-supervised per-pixel fingertip pressure from RGB), and Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks handles the upstream tooling question (GPU-parallel unified tactile simulator with 8 sensor abstractions + explicit placement / resolution / noise / abstraction ablations across three dexterous tasks). Directly addresses the sensorimotor-axis counter-position the VLA Models cluster keeps circling back to — that pure VLM-plus-action-head recipes under-weight contact dynamics.

Key claims

• Proprioception is insufficient for contact-rich dexterous tasks — even the cheapest binary-contact tactile abstraction beats a no-tactile baseline on all three tested tasks (Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks §4.1, Fig. 7).
• Placement dominates sensor type: whole-hand coverage substantially outperforms fingertip-only, and adding taxels on the palm and proximal phalanges is a bigger marginal win than upgrading the fingertip sensor (Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks §4.1). This directly contradicts the de-facto convention of commercial tactile hardware, which concentrates spatial resolution at the distal fingertip pad.
• Per-taxel force/torque is the recommended default abstraction across dexterous tasks, matching or outperforming binary contact, contact depth, elastomer displacement, and proximity in aggregate — with proximity edging it out only on tasks where the object approaches the hand rather than being already grasped (Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks §4.1).
• Resolution matters far less than coverage: ~200 taxels distributed across the whole hand suffice for the three tested dexterous tasks (Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks §4.1).
• Elastomer marker displacement (the GelSight-style abstraction) underperforms force/torque when local-force reading is what the task needs — displacement at one taxel is a function of indentation and shear at neighbouring taxels, blurring the local force vector (Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks §4.1).
• The rate mismatch between high-frequency touch (e.g. kHz-class fingertip sensors) and slower vision/language streams can be handled by a variable-rate Mix-of-Transformer with a per-modality tactile encoder, without sacrificing the capabilities of an existing VLA (Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction project page).
• Discrete tokenisation via a temporal tactile VQ-VAE turns raw touch into a code stream the policy can attend over — contrasted on the project page with prior “static” tactile encoders — and delivers >30% absolute average success-rate improvement over the strongest baseline across 12 delicate-force / deformable-object tasks (Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction).
• Fingertip pressure can be estimated from a single RGB image by an SE-ResNeXt-50 + FPN encoder-decoder trained with weak “press finger X at force Y” contact labels plus adversarial domain alignment — reaching 41.9% contact IoU / 27.5% volumetric IoU on textured overlays and human-annotator-parity 80.5% contact accuracy on in-the-wild surfaces where no pressure sensor could be mounted (PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images Table 2).
• Weakly-supervised prompts (“press ring finger at low force”) can substitute for instrumented ground truth as pressure supervision on 100+ natural surfaces — a scaling recipe that sidesteps the “each sensor defines a new robot” bottleneck the simulator work also flags (PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images §3.1).
• Sim-to-real transfer works for tactile policies at the fingertip-aggregate level: a policy trained on the agg_bool observation type in simulation matches its simulation success rate when deployed on a real XHand1 whose only tactile channel is fingertip-aggregate force (Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks §4.1).
• Temperature is a plausible tactile modality but current real-hardware temperature sensors are too insensitive to learn even a hot-ball-among-8-distractors discrimination task from touch alone — the sensitivity gap between sim and hardware is the blocker, not the algorithm (Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks §3.3, Fig. 6).

Recent contributions

• [2026-08-27] Project SuperDex — A unified simulation platform for dexterous manipulation: Project SuperDex (Meta) advertises tactile sensors as a first-class feature of its unified multi-physics engine (alongside soft/rigid articulations, non-convex collision, rods/tendons, shells/cloth) — a Meta-scale counter-position to Tactile Genesis’s eight-abstraction interface and Tacmap’s single-abstraction commitment. No sensor abstractions, taxel counts, or sim-to-real numbers disclosed on the landing page.
• [2026-08-21] T-Rex: Tactile-Reactive Dexterous Manipulation: T-Rex (Berkeley + NVIDIA GEAR, full arxiv version of Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction) delivers the quantitative counter-example the concept needed: naive tactile addition regresses a strong VLA (π0.5 + tactile 6% vs π0.5 17% across 12 tasks) — tactile integration must be architecturally deliberate, not appended. The recipe is a variable-rate Mixture-of-Transformer-Experts with an async cascaded flow-matching split at τ=0.4 (slow visuomotor expert + fast tactile expert firing 4× per vision tick), a per-finger temporal VQ-VAE (K=64 codebook over 16-frame force windows) with EMA + magnitude-weighted loss to prevent no-contact collapse, and the three-stage EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data pretraining recipe extended with a 100 h motor-primitive-organized mid-training dataset. Reports 65% avg vs 35% EgoScale on 12 real-world contact-rich tasks (30 pt gap comparable to EgoScale’s own 54 pt pretraining gap), with training-stage ablation showing from-scratch 18 → +human-pretrain 34 → +tactile-mid-train 45 → +full 65. Answers the concept’s “observation-space contract between tactile front-end and VLA backbone” open question with per-finger VQ-VAE codes + instantaneous force projection + ResNet deformation-map features fused via a separate high-rate expert.
• [2026-08-20] ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learning: ADEPT (NVIDIA + U. Michigan) is the first filed instance on the wiki of a fully sim-trained RL policy consuming five vision-based tactile sensors on a 29-DoF Flexiv-Sharpa hand and zero-shot transferring to real hardware — a concrete existence proof for the sim-first tactile-integration route the Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks / Tacmap: Bridging the Tactile Sim-to-Real Gap via Geometry-Consistent Penetration Depth Map tooling papers have been arguing for. Same recipe deploys separately on a tactile-free 23-DoF Kuka-Allegro (two RGB cameras only), so the tactile channel is architecturally optional rather than load-bearing to the recipe.
• [2026-08-17] ReForce — force-aware retargeting via closed-loop residual force tracker: ReForce (@wu876768 tweet) is a policy-composition counterpart to FACT: Force-Aware Contact-Rich Manipulation via Timestep Modulation — demystifying when and why VLAs fail on contact-rich tasks FACT on the force-integration axis: rather than modulating a VLA per-layer by current force, ReForce trains a closed-loop residual force tracker in simulation whose input is a motion+force reference plus live tactile state, and adds its residual joint command atop the reference at deployment. Reports 30%→70% force-safe success and 7/10→0/10 severe missing-contact failures on paper-cup grasping with a learned ACT reference. Answers the concept’s “how should the force channel enter the policy” question with residual-atop-reference rather than input-concatenation or per-layer modulation. No sensor abstraction, simulator, or robot spec disclosed in the thread.
• [2026-08-04] FACT: Force-Aware Contact-Rich Manipulation via Timestep Modulation — demystifying when and why VLAs fail on contact-rich tasks: FACT (Parés-Morlans et al., tweet thread) argues force integration in current VLAs fails not because force sensing is absent but because force is appended as an input rather than modulating the policy — proposes per-layer modulation by current force, 2 s of force history as tokens, and contact-gated learning; reports 66% vs 41% baseline across 5 contact-rich tasks and ~2,500 real rollouts. Sits between taxel-level tactile (VTAP Gripper: Synergizing Fingertip Sensing and a Visuo-Tactile Active Palm for Dexterous In-Hand Manipulation) and F/T-only recipes on the sensor-abstraction axis this page tracks, on the F/T end with per-layer modulation rather than input-concatenation as the integration pattern.
• [2026-08-03] Aero Hand Open: low-cost, open-source, lightweight, anthropomorphic dexterous hand: Aero Hand Open takes the sparsest position on the sensor-placement axis: no fingertip tactile arrays at all, using inline tendon-force sensing as the contact/load proxy — a hardware data point complementary to Tactile Genesis’s placement-dominates-type finding.
• [2026-08-01] Open-X-Tactile — community effort to build the world's largest heterogeneous tactile manipulation dataset: Open-X-Tactile (Michael Yuan, Tsinghua IIIS with Yang Gao) is the first filed community-coordination attempt to fill the “large-scale tactile dataset” gap that Data Pyramid for Embodied Manipulation named as open challenge #1 — 50+ institutions and companies (UC Berkeley, MIT, CMU, Tsinghua, ETH Zürich, NUS, Sharpa, Wuji, Paxini, Xense) signed on, with a unified data format and benchmark ecosystem accepting contributions of any size / sensor / embodiment / task, explicitly framed as the tactile analog of Open X-Embodiment; downstream question is which sensor-abstraction bet the format spec picks (single-abstraction à la Tacmap: Bridging the Tactile Sim-to-Real Gap via Geometry-Consistent Penetration Depth Map vs 8-abstraction superset à la Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks).
• [2026-07-30] Sharpa releases the Origami Dataset for the IROS 2026 Robotic Origami Challenge: Sharpa’s Origami Dataset release fills the “large-scale tactile dataset” gap named as open challenge #1 in Data Pyramid for Embodied Manipulation with 682 episodes / 4.76M frames of bimanual paper folding on two 22-DoF SharpaWave hands, 10 fingertips × 6-axis F/T synchronized frame-by-frame with 6 camera views at 30 FPS, exported in both raw and deform-map tactile formats (matching Tacmap: Bridging the Tactile Sim-to-Real Gap via Geometry-Consistent Penetration Depth Map‘s abstraction) with full LeRobot 3.0 coverage; the paper-folding task also stress-tests the low-force / deformable-object claims of Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction and stakes a fingertip-only sensor placement counter to Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks‘s coverage-over-fingertips finding.
• [2026-07-29] Tacmap: Bridging the Tactile Sim-to-Real Gap via Geometry-Consistent Penetration Depth Map: Tacmap (Su, Peng, Ren, Mao, Du, Zhang, Zhu; Sharpa + HKUST + NVIDIA) is the sim-to-real alignment half of the tactile stack — commits to a single tactile abstraction, geometry-consistent penetration-depth (“deform map”), computed by GPU ray-casting in Isaac Lab / MuJoCo and recovered from real tactile images by a ResNet inversion trained against ground-truth depth measured by an automated indenter rig. Reports sub-mm contact position error, ~0.5 N net-force error, >85% deform-map IoU on square+cylinder indenters, and a zero-shot sim-to-real PPO in-hand rotation policy on the SharpaWave hand. Complements Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks (unifies 8 abstractions, argues per-taxel force/torque as default) by taking the opposite bet — commit to one abstraction and monopolize the alignment story around it. Answers the concept’s “observation-space contract between tactile front-end and policy” open question with a specific proposal: normal-projection penetration-depth pixel field. Explicitly punts on tangential shear (§VI limitations) — the HydroShear: Non-Holonomic Hydroelastic Tactile Simulator / HydroShear: Hydroelastic Shear Simulation for Tactile Sim-to-Real Reinforcement Learning axis.
• [2026-07-29] HydroShear: Non-Holonomic Hydroelastic Tactile Simulator: HydroShear (project page) is the upstream reference for the shear-simulation baselines Tactile Genesis benchmarks against and whose elastomer model Tactile Genesis extends. Non-holonomic hydroelastic simulator that models stick-slip, path-dependent force/shear build-up, and full SE(3) object-sensor interactions via SDF-tracked on-surface indenter displacements. Reports 93% avg zero-shot sim-to-real RL success across peg insertion / bin packing / book shelving / drawer pulling — vs 34% for tactile-image-trained baselines and 58-61% for alternative shear simulators. Concrete quantitative case that shear-fidelity, not tactile-image rendering, is the load-bearing piece for contact-rich RL transfer. Closes a citation gap in the concept page since Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks‘s §3.1 elastomer model builds on it directly.
• [2026-07-28] Data Pyramid for Embodied Manipulation: Data Pyramid names large-scale tactile datasets as the first of six open challenges for embodied-manipulation data recipes — external corroboration that the concept-page thesis (tactile is the load-bearing missing modality for dexterous manipulation) is now the consensus survey position, not a fringe critique. Complements the three existing filed instances (Tactile Genesis simulator, Tactile-Reactive Dexterous Hand policy, PressureVision++ vision-only substitute) with the recognition that the dataset layer is what still doesn’t exist at pretraining scale.
• [2026-07-28] Being-H0.8: A Latent Tactile World-Action Model at Scale: Being-H0.8 (BeingBeyond) is the first filed foundation-model-scale instance of “infer touch from vision” as the primary tactile channel — TactoHand converts 500,000+ hours of egocentric human video into dense pseudo-tactile supervision (binary contact + continuous proximity on canonical MANO) via geometry-supervised prediction, and a universal tactile encoder (query-based Perceiver over a coarse-to-fine pyramid) unifies these pseudo-labels with measured taxel/pressure signals under one interface. Extends the FELT: Generating Tactile Signals from Vision for Visuo-Tactile Manipulation / PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images vision-only-tactile recipe from per-task policy training to embodied pretraining, and answers the “observation-space contract between tactile front-end and VLA backbone” open question with a Perceiver-resampled fixed-token interface plus a learned missing-touch representation for tactile-absent samples.
• [2026-07-25] FELT: Generating Tactile Signals from Vision for Visuo-Tactile Manipulation: FELT (Li, Ling, Gu, Huang, Liang, Islam, Bedri, Chirikjian, Li, Nikolaidis, Seita) extends the vision-only-tactile idea from PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images to robot grippers and — critically — validates it inside a manipulation policy: a frozen vision encoder plus a lightweight dual-branch query decoder (one branch per finger, respecting sensor topology) synthesizes per-finger pressure tactile images from RGB in a feed-forward pass. Both generated tactile images and latent tactile features improve success over vision-only baselines across four contact-rich tasks, with the latent-feature variant needing no real tactile sensor at either training or deployment — a purely software tactile channel. Shared authors (Binghao Huang, Yunzhu Li) with VTAP Gripper: Synergizing Fingertip Sensing and a Visuo-Tactile Active Palm for Dexterous In-Hand Manipulation make this the third leg of the same group’s tactile-manipulation triangulation (hardware, sim, vision-to-tactile).
• [2026-07-23] VTAP Gripper: Synergizing Fingertip Sensing and a Visuo-Tactile Active Palm for Dexterous In-Hand Manipulation: VTAP Gripper (Zhou, Athar, Hu, Huang, Li, Wachs, She; IROS 2026) is the first filed hardware instance operationalizing Tactile Genesis’s placement-over-type finding — an actuated visuo-tactile active palm that switches between long-range camera view (pre-contact) and contact-rich tactile sensing (post-occlusion), paired with FlexiTac tactile-array fingers. Demonstrates 93.3% tactile-reactive grasping on YCB+fragile, autonomous vision-then-tactile peg-in-hole (14 mm peg, 15 mm hole, 1 mm tolerance, 7/10), and 3-mm-object in-hand singulation on a three-finger (not anthropomorphic) gripper. Shares author Yunzhu Li with Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction and stakes a low-DoF-gripper position against the anthropomorphic route.
• [2026-07-16] Solving Dexterity: A Full-Stack Approach (mimic hand M1 + wearable U1): mimic hand M1 pursues a hardware recipe orthogonal to Tactile Genesis’s placement-dominates-type finding: a dual motor+joint encoder scheme derives ≤0.1 N unidirectional contact-force estimates from motor current on every joint (effectively whole-hand implicit force sensing), then adds dedicated fingertip tactile sensors for tangential shear and higher spatial resolution only at the distal contact points. The argument is that <0.05 Nm joint backdrivability lets every actuator double as a force sensor, sidestepping the taxel-placement problem — but no head-to-head with dedicated palm / proximal-phalanx taxels is offered. Complements Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks‘s simulator-driven placement study by testing the alternative hypothesis that backdrivable actuation obviates whole-hand taxel coverage.
• [2026-07-14] Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks: GPU-parallel tactile simulator integrated into Genesis World with 8 sensor abstractions (contact / depth / kinematic force-torque / elastomer / proximity / temperature / audio) under a common interface, 20,000+ parallel envs and 1,000+ taxels on a single GPU (3-20× throughput over TacSL / Tacmap / HydroShear / FOTS). Ablations across three dexterous tasks × three placements × three resolutions × clean/noisy noise conditions yield the concrete design directives: placement > type, per-taxel force/torque is the default, 200 whole-hand taxels suffice. Sim-to-real validation on the real XHand1 confirms the fingertip-aggregate abstraction transfers.
• [2026-06-20] Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction: Tactile-Reactive Dexterous Hand — first policy-side answer on the wiki. Variable-rate Mix-of-Transformer + temporal tactile VQ-VAE fuses kHz-class touch with slower vision/language, on top of an existing VLA. 100 h motor-primitive-prioritized teleop dataset; >30% absolute average success rate over the strongest baseline across 12 delicate-force / deformable-object tasks. Only a project page at filing time — no paper or code yet.
• [2026-06-18] PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images: PressureVision++ — the vision-only substitute. Weak “press finger X at force Y” contact labels + adversarial domain alignment supervise a per-pixel pressure predictor from a single RGB image; 41.9% contact IoU / 27.5% volumetric IoU on textured overlays, 80.5% in-the-wild contact accuracy matching human annotators. 2.9M-frame ContactLabelDB with 51 participants × 106 surfaces released.

Open questions

• Does the policy-side answer converge on variable-rate multimodal fusion (Tactile-Rex’s MoT) or on a unified token stream with per-modality position encodings? No filed paper compares the two.
• Which downstream tasks actually benefit from per-taxel force/torque vs. per-link aggregate force? Tactile Genesis shows near-continuous-contact tasks (in_hand_repose) reward per-taxel, but the real-XHand1 hardware only exposes aggregate — so the win is real-world-bounded by what hardware exists.
• Is vision-only contact inference (PressureVision++) sufficient as the VLA’s tactile channel, or is the latency and occlusion sensitivity a hard blocker that keeps first-class touch sensors on the critical path? No filed head-to-head.
• Does the “coverage over resolution” finding from Tactile Genesis generalize to bimanual manipulation and to tasks that involve tool-mediated contact (e.g. screwdriver, chopsticks) where the informative contact point sits outside the hand? Screwdriver-task results in the paper hint the missing channel there may be temporal integration, not spatial density.
• Elastomer marker displacement is a first-class output of many vision-based tactile sensors (GelSight, DIGIT). If it underperforms force/torque per-taxel on the dexterous tasks Tactile Genesis tested, does that mean the current elastomer-simulator development effort is targeting the wrong downstream metric, or that the tasks need to change?
• What is the right observation-space contract between a tactile front-end and a VLA backbone? Tactile-Rex’s temporal VQ-VAE yields discrete codes; PressureVision++ yields per-pixel dense pressure maps; Tactile Genesis exposes per-taxel force vectors. None of these have been unified across policies.
• Does simulator-first tactile hardware design (i.e., using Tactile Genesis’s ablations to spec whole-hand coverage before building the hardware) actually shorten the hardware-iteration loop, or does the sim-to-real gap on non-fingertip surfaces re-open the loop anyway?

Papers

2026-06

• Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks — Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks (published 2026-06-21)
• Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction — Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction (published 2026-06)

2023-01

• PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images — PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images (published 2023-01-05)