HydroShear: Non-Holonomic Hydroelastic Tactile Simulator

TL;DR

HydroShear is a non-holonomic hydroelastic tactile simulator that targets the sim-to-real gap for contact-rich manipulation by modeling three things prior tactile simulators skimp on: stick-slip transitions, path-dependent force/shear build-up, and full SE(3) object–sensor interactions. It extends the hydroelastic contact model with Signed Distance Functions (SDFs) that track on-surface indenter point displacements through the elastomer membrane, producing physics-based force fields from arbitrary watertight geometries while staying agnostic to the underlying physics engine. On GelSight Minis, HydroShear reproduces real tactile shear more faithfully than prior methods (FOTS, TacSL variants). Trained-in-sim RL policies transfer zero-shot to real hardware across four contact-rich tasks — peg insertion, bin packing, book shelving, drawer pulling — averaging 93% success versus 34% for tactile-image-trained baselines and 58–61% for alternative shear simulators.

Key claims

Prior tactile simulators focus on vision-based sensor image rendering and use overly simplistic models of force and shear, producing a large sim-to-real gap on dexterous contact-rich tasks [project page overview].
HydroShear models three effects prior methods omit: (a) stick-slip transitions, (b) path-dependent force/shear build-up, (c) full SE(3) object–sensor interactions [project page overview].
The formulation extends hydroelastic contact with SDFs to track on-surface indenter point displacements against the sensor elastomer membrane, yielding a physics-based shear/dilation displacement field on the tactile grid [project page, method illustration].
The pipeline decomposes the marker displacement field into a dilation field (from indenter–elastomer SDF penetration) and a shear field (from the history of indenter poses in the elastomer frame), then sums them into the total field [project page, pipeline figure].
The approach is agnostic to the underlying physics engine and accepts arbitrary watertight indenter geometries [project page overview].
On real GelSight Mini comparisons, HydroShear reproduces measured tactile shear more faithfully than FOTS (reimplemented) and TacSL variants (TacSL Shear, TacSL Gray) [project page qualitative comparisons across Mario Star, Cow, Dumbbell, Torus, Sphere, Cross indenters].
Sim-to-real transfer works zero-shot across four contact-rich RL tasks — peg insertion, bin packing, book shelving, drawer pulling — with an average 93% real-world success rate [project page results].
Same-recipe policies trained on tactile images achieve only 34% average success; alternative shear simulation methods reach 58–61% — HydroShear’s advantage is attributed to the shear-fidelity gap rather than the RL recipe [project page results].
Training uses Asymmetric Actor-Critic Distillation (AACD): stage 1 trains a privileged teacher actor-critic with access to contact forces and object poses; stage 2 initializes the critic from the teacher and trains a student actor from scratch on high-dimensional inputs (EE pose, relative EE-goal pose, left/right tactile shear); stage 3 deploys the student in the real world [project page AACD illustration].
The student actor/critic uses encoder-LSTM-MLP networks optimized with PPO [project page AACD illustration].

Method

The simulator’s core is a shear+dilation decomposition on the tactile grid. Given an indenter geometry I and a sensor elastomer E, HydroShear computes marker displacement at each tactile grid point by combining (a) a dilation displacement field, derived by identifying grid points inside the indenter SDF and computing the outward push against the elastomer, and (b) a shear displacement field, derived by tracking the history of the indenter’s pose in the elastomer frame and connecting each initial on-surface contact point to its current position while it remains in penetration. The two fields sum to the total marker displacement returned to downstream policies. The formulation is deliberately physics-engine-agnostic and accepts any watertight indenter mesh.

Policy training uses Asymmetric Actor-Critic Distillation (AACD): a privileged teacher is trained with PPO on ground-truth contact forces and object poses; a student actor is then trained from scratch with the privileged critic frozen as initialization, but seeing only proprioception (EE pose, relative EE-goal pose) plus left/right tactile shear from HydroShear. Encoder-LSTM-MLP networks give the student memory to handle path-dependent shear accumulation. The student deploys zero-shot on the physical robot.

Results

Real GelSight Mini shear reproduction: HydroShear closer to measured shear than FOTS (reimplemented), TacSL Shear, and TacSL Gray across six indenter geometries (Mario Star, Cow, Dumbbell, Ring Torus, Sphere including rolling, Cross), evaluated from both front and side views [project page qualitative comparisons].
Zero-shot sim-to-real RL transfer succeeds across four contact-rich tasks — peg insertion, bin packing, book shelving (for insertion), and drawer pulling (fine gripper control under slip) — with 93% average success [project page results].
Baselines on the same tasks: policies trained on tactile images reach 34% average success; policies trained with alternative shear-simulation methods reach 58–61% — a >30-point gap attributed to shear fidelity [project page results].

Why it’s interesting

HydroShear is the direct upstream reference for the shear-simulation baselines in Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks. Tactile Genesis explicitly benchmarks against HydroShear and reports beating it on marker-displacement error (relative RMSE 0.329 vs 0.403 on dilation; 0.174 vs 0.217 on shear on GelSight Mini) while claiming 3–20× throughput advantage — but Tactile Genesis’s elastomer model extends HydroShear with a compressibility term and clamped boundary condition, making HydroShear the underlying formulation the newer simulator builds on. Filing it now closes a citation gap in Tactile sensing for manipulation.

The zero-shot sim-to-real story also complements FELT: Generating Tactile Signals from Vision for Visuo-Tactile Manipulation (vision-only tactile inference) and Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction (policy-side high-frequency tactile fusion) by taking the third route on the tactile-manipulation triangle: skip vision-based tactile images entirely and train directly on high-fidelity simulated shear. The 93% vs 34% gap over image-based baselines is the strongest quantitative case yet on the wiki that shear fidelity, not tactile image rendering, is the bottleneck for contact-rich RL transfer.