Skip to content
Bud's Wiki

FELT: Generating Tactile Signals from Vision for Visuo-Tactile Manipulation

TL;DR

Section titled “TL;DR”

FELT (Feature-Extracted Latent Tactile) is a learning-based framework that synthesizes per-finger pressure tactile images from RGB observations in a single feed-forward pass, letting a policy trained with vision-only data recover most of the benefit of instrumented touch. A large frozen visual encoder feeds a lightweight query decoder with separate left/right branches that respect the physical topology of dual-finger tactile sensor panels. At inference the policy either consumes the generated tactile images or, cheaper still, the latent tactile features — both improve success over a vision-only baseline on four contact-rich manipulation tasks, and the latent-feature variant needs no tactile sensor at deployment. It is a pragmatic answer to the “tactile data is scarce because tactile sensors are fragile” bottleneck.

Key claims

Section titled “Key claims”
  • Tactile data remains scarcer than visual data because tactile sensors are fragile, specialized, and hard to standardize — motivating a synthesis-from-vision approach [§1].
  • A frozen large visual encoder plus a lightweight query decoder can predict per-finger pressure tactile images in a single feed-forward pass from RGB alone [§3].
  • Decoding the left and right tactile sensor panels through separate branches captures the asymmetric contact patterns of wiping, insertion, and in-hand rotation better than a shared decoder [§3, Method].
  • At inference FELT requires only RGB, so vision-only demonstration datasets can be augmented with either generated tactile images or latent tactile features [§4].
  • Both generated tactile images and latent tactile features improve policy success over vision-only baselines on four contact-rich manipulation tasks [§5, Results].
  • The latent-feature variant requires no real tactile sensor during policy training or deployment, only vision — meaning the tactile channel becomes a purely software-side addition [§5].

Method

Section titled “Method”

FELT is a two-stage pipeline. A large frozen vision encoder produces features from RGB observations; a lightweight query decoder — split into two branches, one per finger of a dual-finger tactile sensor — maps those features to per-finger pressure tactile images. The topological split is the key architectural choice: because a two-finger gripper’s left and right pads see complementary contact geometries during tasks like wiping, insertion, and in-hand rotation, tying their outputs through a shared decoder would blur the asymmetric contact signal. Once trained on paired RGB + tactile data, FELT can operate in two downstream modes: (1) generated-image mode, where the synthesized tactile image is fed into a policy trained on real tactile images, and (2) latent-feature mode, where the intermediate FELT features go straight into the policy — bypassing the need for a real tactile sensor at either train or test time.

Results

Section titled “Results”

Four contact-rich manipulation tasks (wiping, insertion, in-hand rotation, and one further contact-rich task per the abstract) are used as the evaluation suite. Both FELT-generated tactile images and FELT latent tactile features improve policy success over vision-only baselines on all four tasks [Abstract, §5]. The latent-feature variant is the more aggressive claim: policies trained and deployed with no real tactile sensor at any stage still outperform the vision-only baseline, because the vision encoder has effectively absorbed enough tactile-adjacent signal from the pairing dataset to be useful downstream.

Why it’s interesting

Section titled “Why it’s interesting”

FELT sits alongside PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images as the wiki’s second “vision-only tactile substitute,” but goes further: PressureVision++ predicts per-pixel pressure on human hands for annotation and analysis, while FELT predicts robot-gripper tactile images and validates the substitute inside a manipulation policy. It also complements the sim-first placement study of Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks by attacking the same data-scarcity bottleneck from the opposite direction — instead of simulating tactile physics, hallucinate the tactile signal from RGB. Shared authors with VTAP Gripper: Synergizing Fingertip Sensing and a Visuo-Tactile Active Palm for Dexterous In-Hand Manipulation (Binghao Huang, Yunzhu Li) place this in the same research thread as the hardware-side VTAP gripper, which suggests the group is triangulating tactile-augmented manipulation from three angles at once: hardware (VTAP), simulation (Tactile Genesis), and vision-to-tactile synthesis (FELT). It’s also the first filed paper where the tactile channel is entirely software — a data-augmentation view of touch rather than a sensor-integration view.

See also

Section titled “See also”