Skip to content
Bud's Wiki

TouchScale: 500 Hours of Human Vision and Touch for Visual-Tactile Learning

TL;DR

Section titled “TL;DR”

TouchScale is a 500-hour human-interaction dataset recorded with a single unified wearable rig — head-mounted RGB-D (30 Hz), two wrist-mounted RGB cameras (30 Hz), and bimanual tactile gloves with 880 sensing taxels per hand (sub-2 mm resolution) — covering ~2K predefined task descriptions across everyday activities and structured manipulation. Holding the sensor and collection protocol fixed and varying only the amount of data used, the authors show monotonic upward trends on zero-shot tactile prediction, on action recognition after visual-encoder pretraining, and on real-robot contact-rich manipulation. Headline numbers: zero-shot contact IoU on an unseen tactile sensor rises from 0.134 to 0.383 when training uses the full TouchScale; and tactile mid-training a robot policy lifts real-world success on four contact-rich tasks from 22.5% → 57.5% average under matched 50-demo / 20-trial-per-task evaluation on an xArm6 + BrainCo Revo 2 hand. The whole stack — synchronized visual-tactile recordings and reconstructed object models — will be publicly released, filling the “no large-scale tactile dataset at consistent sensing” gap that Tactile sensing for manipulation and the Data Pyramid paper have been naming for months.

Key claims

Section titled “Key claims”
  • A single unified sensor + collection protocol across 500 h of bimanual contact-rich human interaction yields monotonically improving downstream performance as more data is used — the holding-sensor-fixed design is what lets the data-amount effect be isolated [§Abstract].
  • Training on the full TouchScale raises zero-shot contact IoU on data from an unseen tactile sensor from 0.134 to 0.383 [§Abstract] — a direct cross-sensor-transfer measurement under the “hold sensor, vary data” design.
  • Pretraining a visual encoder on TouchScale yields the highest action recognition accuracy on three benchmarks among compared visual-tactile datasets [§Abstract].
  • Used for visual-tactile mid-training of a robot policy, TouchScale lifts average real-world success on four contact-rich manipulation tasks from 22.5% → 57.5% (50 demos/task, 20 trials/task, xArm6 + BrainCo Revo 2 + ego + wrist cameras) [§Abstract, project page].
  • Each recording temporally aligns egocentric RGB-D video, wrist RGB video, and dense full-hand bimanual tactile measurements — alignment is a load-bearing design property, not a post-hoc nice-to-have [§Abstract].
  • Dataset covers ~2K predefined task descriptions across laboratories, kitchens, workbenches, and everyday environments; recordings span before-contact / during-contact / after-release phases of each interaction [project page].
  • Hardware stack: head-mounted RGB-D camera + two wrist-mounted RGB cameras at 30 Hz, bimanual tactile gloves with 880 sensing taxels per hand distributed across five fingers and palm at sub-2 mm spatial resolution [project page].
  • Robot evaluation stack: xArm6 arm + BrainCo Revo 2 robotic hand with its own tactile sensing + ego-view + wrist cameras; demonstrations collected via VR headset + motion-capture gloves [project page].

Method

Section titled “Method”

TouchScale is a dataset-first contribution. One unified wearable rig records every demonstration: a head-mounted RGB-D camera (①) captures the overall egocentric scene; two wrist-mounted RGB cameras (②) capture close-up hand-object contact; bimanual tactile gloves (③) each carry 880 sensing taxels across the five fingers and palm at sub-2 mm resolution. All streams run at 30 Hz and are time-aligned, giving each sample a frame-synchronized vision + wrist-vision + full-hand bimanual tactile tuple. The ~2K task descriptions span everyday activities (kitchens, workbenches) and structured manipulation sequences, and the dataset will also release reconstructed object models alongside the raw recordings.

On the robot side, the paper validates the “mid-training” use of TouchScale: a visual-tactile backbone is first mid-trained on human TouchScale data, then post-trained with 50 teleoperated demonstrations per task on an xArm6 + BrainCo Revo 2 hand instrumented with its own tactile sensors plus ego-view and wrist cameras. Mid-training explicitly includes a future-tactile-change prediction objective — the policy learns to anticipate how tactile signals will evolve before the real post-training phase. Four tasks are used: bottle-cap removal, test-tube transfer, whiteboard wipe, and soft/hard sorting; 20 evaluation trials per task.

Results

Section titled “Results”

Headline numbers from the abstract and project page (holding the sensor + protocol fixed and varying training data):

  • Cross-sensor zero-shot tactile prediction: contact IoU on an unseen sensor rises 0.134 → 0.383 (+0.249, roughly +186% relative) when scaled to the full TouchScale corpus [§Abstract]. This is the sharpest cross-sensor-generalization datapoint on the wiki — directly addresses the “each sensor defines a new robot” bottleneck that PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images and the Tactile Genesis / Tacmap axis have been circling.
  • Action recognition: TouchScale-pretrained visual encoder achieves the highest action recognition accuracy on three benchmarks vs. other visual-tactile dataset baselines; MECCANO specifically improves vs. EgoTouch from 20.16% → 28.97% (+43.7% relative) [project page].
  • Real-robot contact-rich manipulation: average success across four tasks lifts 22.5% → 57.5% (+35.0 pp absolute) under matched 50 demos / 20 trials per task [§Abstract, project page].
  • Trend-in-data-amount: with the sensor and collection protocol held fixed, both zero-shot tactile prediction and real-robot success show an overall upward trend as more TouchScale data is used [§Abstract] — the design that makes this measurable (one rig, one protocol, 500 h) is itself the contribution.

Reconstructed object models + full synchronized recordings will be publicly released; code is at github.com/phai-lab/touchscale and the dataset at huggingface.co/datasets/2077AIDataFoundation/TouchScale under MIT license.

Why it’s interesting

Section titled “Why it’s interesting”

TouchScale is the first filed instance on the wiki of a large-scale egocentric human-tactile corpus with the sensor + protocol held fixed across all recordings — the exact design the Tactile sensing for manipulation concept page has been naming as the missing data-side piece. The “hold sensor, vary data” design is what lets its scaling claims (0.134 → 0.383 contact IoU; 22.5% → 57.5% robot success) be read as data-amount effects rather than confounded by sensor heterogeneity, which is the critique that bit the Open-X-Tactile (Open-X-Tactile — community effort to build the world's largest heterogeneous tactile manipulation dataset) pooled-corpus approach and that Data Pyramid for Embodied Manipulation named as open challenge #1.

Sits alongside ACE-Data-0: Human-Centric Ambient Capture as Embodied Data Engine ACE-Data-0 (which provides full-palm tactile grids synchronized with mocap + exocentric video but at smaller scale) and Neocambrian AI launches Mitts — tactile gloves for high-precision robot touch data Mitts (hardware-only announcement, no corpus) as the at-scale corpus version of the glove-based human-tactile-capture axis. Complementary to Being-H0.8: A Latent Tactile World-Action Model at Scale Being-H0.8 — Being-H0.8 argues the way to make touch a first-class pretraining ingredient is to infer pseudo-tactile from vision at the 500,000+ h scale; TouchScale argues the way is to measure real tactile at consistent-sensor 500 h scale and let that drive both perception and policy mid-training. Head-to-head between these two bets at matched downstream protocol is the obvious next experiment the wiki now wants.

See also

Section titled “See also”