PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images

TL;DR

PressureVision++ trains a deep model to estimate per-pixel fingertip pressure from a single RGB image of a human hand on a surface, sidestepping the data-collection bottleneck of instrumented pressure sensors by using contact labels — weak prompts of the form “press ring finger at low force” — as supervision on unlabeled diverse-surface data. The system reports 41.9% contact IoU and 27.5% volumetric IoU on the fully labeled test set vs. PressureVision’s 15.2% / 11.3%, and matches human annotators on weak-label contact accuracy. The paper also shows a mixed-reality application where everyday tabletops become touch-sensitive keyboards, with users typing 78% faster than on Meta Quest 2’s Direct Touch baseline.

Key claims

Prompting a participant to press specific fingertips at a target force level yields weak supervision (a 6-element “contact label”) that can substitute for ground-truth pressure on surfaces where mounting a sensor would alter appearance or geometry [§3.1].
Adding the contact-label classifier head plus an adversarial domain-alignment loss between fully-labeled and weakly-labeled domains lifts volumetric IoU from 14.9% (fully-labeled only) to 27.5% — a +85% relative gain, with the contact-label loss contributing most of the lift [Table 3].
PressureVision++ reaches 89.3% contact accuracy / 41.9% contact IoU / 27.5% volumetric IoU on the fully labeled test set vs. PressureVision’s 72.7% / 15.2% / 11.3% [Table 2].
On weakly labeled in-the-wild surfaces the model reaches 80.5% contact accuracy, matching human Mechanical Turk annotators (80.5%) and far exceeding the original PressureVision (53.5%) [Table 2].
The 2.9M-frame ContactLabelDB dataset (51 participants × 106 surfaces, with both fully-labeled and weakly-labeled splits) is released; this is 50× more surfaces than PressureVisionDB’s 2 [Table 1].
A pressure-aware touch-typing keyboard projected onto a tabletop using a single external RGB camera at ~50 FPS yields 25.8 net WPM vs. 14.4 WPM for the Meta Quest 2 Direct Touch baseline; 9/10 users prefer it [Table 4].

Method

The network is an SE-ResNeXt-50 + FPN encoder-decoder that takes a MediaPipe-cropped 448×448 hand image and outputs a per-pixel pressure image discretized into logarithmically-spaced bins, trained with a structure-aware cross-entropy loss on the fully labeled split. Two auxiliary losses bridge the fully-labeled and weakly-labeled domains: (a) a contact-label classifier on the bottleneck features predicts the 6-vector (5 fingertip-contact bits + one force level), trained with binary cross-entropy on both domains, and (b) a domain discriminator with gradient reversal aligns feature distributions across the two domains. Ground-truth pressure for the fully-labeled split comes from a Sensel Morph pressure-sensing array projected into image space via homography; the weakly-labeled split uses only the prompted contact label and is collected on 100+ natural surfaces (textured, curved, deformable) without any sensor instrumentation [§4].

Results

On the fully labeled test set (textured overlays unseen in training), PressureVision++ hits 89.3% contact accuracy, 41.9% contact IoU, and 27.5% volumetric IoU — improving the prior PressureVision baseline by +16.6, +26.7, and +16.2 absolute points respectively [Table 2]. On weakly labeled diverse surfaces, contact accuracy reaches 80.5%, exactly matching human Mechanical Turk annotators and 27 absolute points above the original PressureVision [Table 2]. Ablating either the contact-label loss or domain loss degrades volumetric IoU substantially (27.5% → 25.5% without domain loss; → 17.5% without contact-label loss; → 14.9% with neither) [Table 3]. The mixed-reality typing study (n=10) reports a 79% throughput improvement over Meta Quest 2’s pose-based Direct Touch keyboard with 9/10 user preference [Table 4]. The main failure mode is occluded fingertips, where pressure is not estimated.

Why it’s interesting

Filed because Supreeth raised it in the #research-external thread as the canonical “vision-from-contact-sensing” reference while debating whether VLAs should ingest force/torque/gripper-width inputs, with the explicit qualifier that the work is in constrained settings. That qualifier is the synthesis hook: this paper backs the position in Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level that tactile/contact perception is a real axis VLA work has under-weighted, while simultaneously showing the scaling failure mode Supreeth flags — the fully-labeled corner requires a Sensel Morph array, which doesn’t generalize to arbitrary objects. The open question on VLA Models of “how should tactile/contact dynamics be incorporated into the VLA stack” has an implicit answer here: vision-only pressure estimation, supervised by easy-to-collect behavioural prompts, may be a cheaper substrate than instrumented gloves or per-object force sensors. The paper also contrasts with the recipe in ActiveMimic: Egocentric Video Pretraining with Active Perception, which treats camera motion as the weak supervision signal for an egocentric pretraining objective rather than contact prompts; both papers reach for the same lever (weak labels collected without instrumented surfaces) but for different downstream signals.