Tacmap: Bridging the Tactile Sim-to-Real Gap via Geometry-Consistent Penetration Depth Map

ListenTouch, Minus the Pixels: How Tacmap Crosses the Sim-to-Real Gap10:20

Show notes

This episode digs into **Tacmap**, a GPU-parallel tactile simulator that trains robots to feel by abstracting touch into a geometry-consistent penetration-depth map instead of raw sensor images. We cover why simulating tactile pixels is the wrong target, how ray-casting along surface normals handles curved fingertips, and the zero-shot in-hand rotation result on a physical dexterous hand. We also talk about what the normal-only formulation deliberately leaves out — and what it all suggests for observation-space design in visual generative models. **Sources** - Tacmap: Bridging the Tactile Sim-to-Real Gap via Geometry-Consistent Penetration Depth Map

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

Tacmap is a GPU-parallel tactile simulator for vision-based tactile sensors (VBTS) that abstracts touch away from raw sensor images and into a geometry-consistent penetration-depth map (a “deform map”). In simulation, the deform map is computed by ray-casting between two nested sensor surfaces to measure the intersection volume with a rigid object; in the real world, a ResNet encoder-decoder is trained (with an automated indenter rig) to translate raw tactile images into the same deform-map representation. Because policy and hardware then observe the same domain-invariant geometric quantity, a PPO in-hand rotation policy trained purely in simulation transfers zero-shot to the SharpaWave dexterous hand. Tacmap is geometry-agnostic (supports curved fingertips via normal-projection), integrates into Isaac Lab (thousands of parallel envs) and MuJoCo, and sits between analytical depth-buffer methods (fast but shallow) and FEM/IPC-based simulators (accurate but too slow for RL).

Key claims

Simulating tactile images is the wrong target: the underlying penetration-depth deform map is a sensor-agnostic, optics-agnostic proxy that can serve as the shared observation space between sim and real [§I, §III-A].
The simulated deform map is generated by discretising the sensor’s virtual sensing surface into a grid and ray-casting each grid point along the surface normal to measure how far the object has intruded between the sensing surface and the physical undeformed surface [§III-B].
Working in normal-projection space makes the pipeline geometry-agnostic — curved anthropomorphic fingertips are handled natively, unlike prior simulators (TACTO, Taxim, TacSL) that assume flat sensor surfaces [§I, §III-B].
Real-world deform maps are recovered by training a ResNet encoder-decoder on pixel-wise MSE against ground-truth depth measured by a 3-axis motion stage indenting the sensor with a diverse indenter library — the raw tactile image is inverted back into the same geometric quantity the simulator produces [§III-C].
Quantitative sim-to-real fidelity on square and cylindrical indenters: contact-position error <1 mm, deform-depth error sub-mm, net-force error ~0.5 N, deform-map IoU >85% [Table I, §V-B].
Computational efficiency scales to RL-relevant parallelism: GPU memory grows near-linearly from 16 to 8,192 parallel Isaac Lab environments, and enabling Tacmap rendering leaves overall simulator throughput essentially unchanged — because ray-casting is vectorised alongside the physics step, not run as a separate FEM stage [§V-C, Fig. 6].
Zero-shot sim-to-real transfer works on a contact-rich task: a PPO policy trained exclusively in sim with only the Tacmap stream as tactile observation performs smooth, continuous in-hand rotation of a spherical object on the physical SharpaWave dexterous hand with no fine-tuning [§V-D].
Alongside the deform map, Tacmap exposes two other tactile signals: net force (physics-engine contact for sim; ResNet regression from tactile image for real) and contact position (physics-engine contact for sim; centroid of predicted deform map for real). Only the deform map is a pixel-dense geometric field [§III-A].
The normal-only formulation is an explicit limitation: Tacmap does not model tangential shear or micro-slip, both of which the paper flags as necessary for higher-level dexterity like incipient-slip prediction on delicate picking [§VI].

Method

The elastomer is modelled as the volume between two nested surfaces: the undeformed physical sensor surface $S_p$ and a virtual sensing surface $S_v$ offset outward from it. When a rigid object intrudes, its geometry intersects the gap between $S_v$ and $S_p$ . Tacmap discretises $S_v$ into an $H \times W$ grid; from each grid point it casts a ray along the local surface normal inward through the elastomer. The ray’s first intersection with the object mesh, together with $S_p$ ‘s coordinate, defines the local penetration depth — that scalar per grid cell forms the deform map $D$ . Because the projection is along the local normal rather than a global axis, curved sensors are handled natively.

For the real robot, an automated 3-axis motion stage indents the sensor with a library of geometric indenters at recorded 3D poses; the same normal-projection logic is used to compute ground-truth deform maps, giving a synchronised dataset $\{(I, D)\}$ of raw tactile image → deform map pairs. A ResNet encoder-decoder is trained with pixel-wise MSE to invert the sensor’s optics and produce $\hat D$ from a single tactile image at deployment.

At runtime in Isaac Lab, a Multi-Mesh Ray Caster pre-computes the tactile sensing points and their sensing directions on the undeformed sensor surface. Per physics step, the tactile poses are synchronised with the object poses, and the Raycaster API computes GPU-parallel penetration depth across thousands of environments; a vectorised post-processing pipeline converts intersections to the deform map. In MuJoCo, the same logic runs via mjray. The tactile representation is therefore decoupled from the physics collision mesh — you can have a high-resolution tactile grid without paying a physics-solver stability cost.

Results

Force alignment: simulated net force (from physics-engine contact sensors) and real-world estimated net force (from the ResNet regression head) are highly correlated across cylindrical and square compression tests [Fig. 4, §V-B].
Geometric alignment: side-by-side deform maps across six timestamps of a compression trajectory show near-identical contact manifolds between sim and real for both cylindrical and square indenters [Fig. 5, §V-B].
Quantitative fidelity: contact-position error <1 mm, deform-depth error sub-mm, net-force error ~0.5 N, deform-map IoU >85% across the square and cylinder indenters [Table I, §V-B].
Scaling: GPU memory grows near-linearly with parallel environment count from 16 to 8,192 in Isaac Lab; enabling Tacmap rendering yields negligible degradation to total system throughput at thousands of concurrent envs [Fig. 6, §V-C].
Zero-shot policy transfer: a PPO agent trained purely in simulation using the Tacmap stream as observation performs smooth continuous in-hand rotation of a spherical object on the physical SharpaWave dexterous hand with no real-world fine-tuning or domain adaptation [§V-D].
Ablation coverage: experiments span both Isaac Lab and MuJoCo backends and both flat and curved sensor geometries via the normal-projection formulation [§IV, §V-A].

Why it’s interesting

Tacmap is the direct sim-to-real answer to a design directive Tactile sensing for manipulation has been converging on: that the right observation space contract between the tactile front-end and the policy is geometric penetration depth, not raw sensor pixels. It sits complementary to but architecturally distinct from Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks (which unifies eight tactile abstractions inside Genesis World and pushes for per-taxel force/torque as the default) and HydroShear: Non-Holonomic Hydroelastic Tactile Simulator (which models path-dependent stick-slip shear that Tacmap explicitly punts on). If Tacmap’s normal-only geometry is enough for in-hand rotation but HydroShear’s shear model is needed for peg insertion, the tactile-simulator design space is bifurcating along a shear/no-shear axis rather than along a fidelity axis — worth watching. Also: this validates the vision-to-tactile-quantity translation recipe (FELT: Generating Tactile Signals from Vision for Visuo-Tactile Manipulation, PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images) but inverts it — instead of predicting tactile from vision, Tacmap predicts geometry from tactile, then aligns sim to that geometric target.