Skip to content
Bud's Wiki

T-Rex: Tactile-Reactive Dexterous Manipulation

TL;DR

Section titled “TL;DR”

T-Rex is the full-paper version of the earlier project-page release Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction — a Berkeley + NVIDIA GEAR collaboration that operationalizes tactile as a first-class high-frequency modality inside a VLA. The recipe has three legs: a 100-hour tactile-synchronized bimanual teleop dataset organized around 502 object × motor-primitive combinations on a 58-DoF Dexmate Vega-1 + dual Sharpa Wave hands (22 DoF each, 10 fingertip 6-axis F/T + deformation-map sensors); a variable-rate Mixture-of-Transformer-Experts architecture with a slow visuomotor action expert and a fast tactile expert bound together by asynchronous cascaded flow matching (4 fast tactile ticks per slow visuomotor tick, split at τ = 0.4); and a temporal tactile VQ-VAE that discretizes 16-frame windows of per-finger force into a K=64 codebook. Extends the EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data three-stage recipe (22,889 h human video pretraining → tactile mid-training → skill-specific post-training) and reports 65% average success across 12 contact-rich real-world tasks vs 35% for EgoScale and 17% for π0.5 — a 30-point absolute gap that answers the sensorimotor-axis critique the wiki has been tracking since Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level.

Key claims

Section titled “Key claims”
  • Naively conditioning a pretrained VLA on tactile signals hurts performance: π0.5 + tactile scores 6% average vs 17% for plain π0.5 across 12 tasks — tactile integration must be architecturally deliberate, not appended [Table 1].
  • A variable-rate Mixture-of-Transformer-Experts with three experts (latent, action, tactile) supports asynchronous fusion of high-frequency touch with slower vision/language streams via cascaded flow matching split at τ = 0.4 [§4.1, §4.2].
  • The tactile expert fires 4 times per action-expert step (at intra-chunk offsets {0, 4, 8, 12}), reusing the cached vision-language context — so per-step cost is dominated by four lightweight tactile denoising steps rather than re-running the vision tower [§4.2, Eq. 5].
  • A per-finger temporal VQ-VAE compresses 16-frame windows of raw 6-D force into a K=64 codebook via a 1D temporal conv encoder, with EMA codebook updates + periodic re-seeding + magnitude-weighted reconstruction loss to prevent collapse onto the dominant no-contact state [§4.1, App. C].
  • Two-tokens-per-finger encoding (VQ-VAE force + current unquantized force projection + ResNet-derived deformation-map features) preserves both instantaneous force and drift-robust temporal patterns [Eq. 2].
  • All three training stages contribute: from-scratch scores 18% average, +human-pretraining rises to 34%, +tactile-mid-training rises to 45%, full recipe reaches 65% on six benchmark tasks — human pretraining and tactile mid-training are complementary rather than substitutes [Table 3].
  • T-Rex beats every baseline on every one of 12 tasks: 96% Flip Page, 75% Transfer Egg, 78% Split Cup, 47% Open Lock, vs strongest baseline (EgoScale) averaging 35% [Table 1].
  • Ablations show every component earns its place: removing tactile drops 23 points (65 → 42), removing the VQ-VAE force encoder drops 7 points, using only deformation drops 11 points, removing the async cascade drops 5 points [Table 2].
  • Data efficiency: with tactile-grounded mid-training, success climbs far faster as post-training demos grow from 10 → 200; the gap over no-mid-training is largest in the low-data regime [Fig. 5].
  • The dataset explicitly organizes around motor primitives, not tasks — 207 household objects × 22 motor primitives = 502 physically-feasible combinations with ~17 demos each — as a data-efficiency move that maximizes contact-event coverage per hour [§3].
  • Training with a delay augmentation during mid-training matches the visual/tactile staleness seen at deployment, transferring cleanly to real-time closed-loop control [§4.3].
  • An auxiliary future-visual-prediction objective keeps the rapid tactile reflexes grounded in task context and prevents the tactile expert from drifting into pure reactive control [§4.3, Eq. 7].
  • The paper is the arxiv version of the earlier T-Rex project page — the dataset grew from the initial 50-hour / 5,500-episode announcement (per the Jim Fan tweet) to the 100-hour version reported in the arxiv abstract and Table 3.

Method

Section titled “Method”

T-Rex is a flow-matching VLA with a Mixture-of-Transformer-Experts backbone. The three experts share attention KV caches but have separate FFN parameters: a latent expert predicts future visual representations from RGB + language; an action expert denoises actions from pure noise down to an intermediate flow timestep τ = 0.4 over 6 Euler steps; a tactile expert takes over from τ = 0.4 → 0 over 4 fast steps, conditioned on live tactile tokens rather than re-running vision. The tactile front-end is a per-finger temporal VQ-VAE (16-frame 6-D force window → 256-D embedding → K=64 code) plus a lightweight MLP projection of the current force vector plus a frozen ResNet-derived encoder over the deformation map — three streams per finger × 10 fingers form the tactile token sequence.

The training recipe extends EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data: (1) large-scale human egocentric pretraining on 22,889 h of first-person video trains the latent + action experts on retargeted 22-DoF hand actions in a unified action space — no tactile yet; (2) tactile-grounded robot mid-training on the 100-h T-Rex dataset adapts the action expert to robot multiview observations and trains the tactile expert from scratch as a high-frequency refiner, with a delay augmentation matching deployment latency; (3) skill-specific post-training fine-tunes on ~100 demos per task. An auxiliary future-visual-latent-prediction objective on the latent expert (weight 0.1) provides context grounding for the fast tactile reflexes.

Evaluation is a 12-task real-world benchmark on the bimanual Dexmate Vega-1 with two 22-DoF Sharpa Wave hands (58 DoF total), ZED head + wrist cameras, per-finger 6-axis F/T + deformation maps, 30 Hz observations over a 300 Hz low-level control thread. Each task runs 16 randomized trials with progress rubrics for multi-stage tasks.

Results

Section titled “Results”
  • Headline (Table 1): T-Rex 65% average success across 12 tasks; strongest baseline EgoScale 35% (30 points absolute gap); π0.5 17%; Tactile-VLA 15%; RDP 6%; ViTacFormer 3%; π0.5 + tactile 6% (naive tactile bolt-on regresses).
  • Per-task highlights: Flip Page 96% (vs 68% EgoScale, 9% ViTacFormer); Transfer Egg 75% (vs 44%); Apply Paste 66% (vs 38%); Open Lock 47% (vs 19%); Refill Tablet 41% (vs 12%); Deal Poker 57% (vs 28%).
  • Tactile modality ablation (Table 2): removing all tactile drops 65 → 42 (−23 pts); MLP-force + deform drops to 58 (−7); deform-only drops to 54 (−11); MLP-force + VQ-VAE-force (no deform) drops to 59 (−6).
  • Architecture ablation (Table 2): removing async cascade drops to 60 (−5); intermediate split τ = 0.4 is best — too small starves visuomotor priors, too large starves tactile capacity [Fig. 4].
  • Training-stage ablation (Table 3): from-scratch 18 → +human-pretrain 34 → +tactile-mid-train 45 → +full 65 on six benchmark tasks. Both pretraining and mid-training contribute; neither substitutes for the other.
  • Data efficiency (Fig. 5): with tactile-grounded mid-training, success climbs far faster as post-training demos grow from 10 → 200; largest gap in low-data regime.
  • Data-mixture ablation (Fig. 6): the 100 h motor-primitive-diverse T-Rex dataset beats a matched-budget 100 h task-specific dataset (11 tasks) on zero-shot transfer.

Why it’s interesting

Section titled “Why it’s interesting”

T-Rex is the paper this wiki has been waiting for since the Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction project page dropped in June — the arxiv release quantifies every claim the project page made, doubles the dataset from 50 h to 100 h, and reveals the co-authorship is roughly the full Berkeley + NVIDIA GEAR (Malik, Abbeel, Darrell, Goldberg, Fei-Fei) + Yuke Zhu + Danfei Xu + Jim Fan constellation. The paper is a direct architectural successor to EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data — same three-stage recipe, same 22-DoF Sharpa hand, same author overlap — with the tactile expert grafted on. That lineage matters: the 30-point average gap T-Rex reports over EgoScale (65% vs 35%) is roughly the same magnitude as the 54-point gap EgoScale reported over no-pretraining, suggesting tactile is a comparably large lever to pretraining scale. The finding that naive tactile addition regresses performance (π0.5 + tactile 6% vs π0.5 17%) is the concept-page counter-example the Tactile sensing for manipulation cluster needed — it validates the concept-page open question “what is the right observation-space contract between a tactile front-end and a VLA backbone” as not merely open but load-bearing. On the VLA Models cluster axis, T-Rex is the first filed VLA where a modality other than vision drives architectural choices — the async cascade with 4 tactile ticks per vision tick is the touch-first mirror of the compute-flexible schemes Flex-π: A Multi-Stream World-Action Model with Compute Flexibility and πR²: Reactive Real-time Flow Policies have been developing on the action-frequency axis.

See also

Section titled “See also”