TacBPM: A Tactile-conditioned Behavior Prior Model for Dexterous Reorientation

TL;DR

TacBPM is a tactile-conditioned behavior prior for dexterous in-hand reorientation: multi-scale sphere-manipulation specialists are distilled into a single latent controller conditioned on tactile-proprioceptive history, and downstream policies then produce residual latent actions on top of this fixed prior rather than exploring in raw high-DoF joint space. The recipe is evaluated on arbitrary-pose transfer across anisotropic objects, commanded-axis rotation, and an arm-hand Grasp-to-AnyPose task (grasp → lift → transport → reach goal pose for novel tool geometries). Reported results show accelerated training, stable policies where matched raw-action PPO stays near failure, and successful sim-to-real transfer on in-hand and arm-hand tasks. Sits on the “tactile-as-load-bearing-modality” side of the wiki’s dexterous-manipulation debate but stakes a distinct integration pattern from the ones already filed — the tactile channel enters via a distilled prior, and adaptation happens in the prior’s latent space.

Key claims

• Distilling multi-scale sphere specialists into a shared latent behavior prior lets downstream policies reuse a fixed tactile prior via residual latent actions, reducing the exploration burden that raw joint-command policies face on high-DoF hands [Abstract].
• The prior is conditioned on tactile-proprioceptive history, so latent behavior reflects the current hand-object interaction rather than being a static skill prior [Abstract].
• Across arbitrary-pose transfer on anisotropic objects, commanded-axis rotation, and the Grasp-to-AnyPose arm-hand task, the recipe accelerates training and yields stable policies where a matched raw-action PPO baseline remains near failure [Abstract].
• Sim-to-real transfer is demonstrated on both in-hand and arm-hand tasks [Abstract].

Method

TacBPM has two stages. In the first, multiple sphere specialists are trained at different scales (a curriculum whose exact structure the abstract does not fully specify, but which spans object scales relevant to reorientation) and then distilled into a single latent controller — the behavior prior — whose input is a tactile + proprioceptive history window and whose output is a latent action code. In the second stage, downstream task policies (arbitrary-pose transfer, commanded-axis rotation, Grasp-to-AnyPose) are trained to emit residual latent actions that are added to the prior’s latent output before decoding to raw joint commands. Because the prior is frozen, downstream exploration is confined to the residual latent — a much lower-dimensional and better-shaped space than direct joint control on a high-DoF hand. The Grasp-to-AnyPose evaluation composes a full arm-hand pipeline (grasp → lift → transport → goal pose) for novel tool geometries and generalized placements, so the prior has to compose with reach/transport primitives rather than only in-hand rotation. (The abstract does not disclose the specific tactile-sensor abstraction, simulator, or hand hardware; details would come from the full PDF.)

Results

Headline claims from the abstract, without disclosed absolute numbers:

• Accelerated training vs. matched-baseline raw-action PPO across all three task families [Abstract].
• Stable policies produced by TacBPM where raw-action PPO with the same environment and reward remains near failure [Abstract].
• Successful sim-to-real transfer in both in-hand reorientation and arm-hand Grasp-to-AnyPose settings [Abstract].

(The abstract does not report task-level success rates or comparison against tactile baselines other than raw-action PPO; the full paper is needed to place this quantitatively next to filed instances such as ADEPT, T-Rex, or Tactile Genesis.)

Why it’s interesting

TacBPM lands in the middle of an ongoing debate the wiki’s Tactile sensing for manipulation concept has been tracking: given that tactile is load-bearing for contact-rich dexterous tasks, how should it enter the policy? Existing filed answers span input-concatenation (weakest), per-layer modulation (FACT: Force-Aware Contact-Rich Manipulation via Timestep Modulation — demystifying when and why VLAs fail on contact-rich tasks FACT), residual-atop-reference joint commands (ReForce — force-aware retargeting via closed-loop residual force tracker ReForce), and variable-rate multi-expert fusion (T-Rex: Tactile-Reactive Dexterous Manipulation T-Rex). TacBPM stakes a new position — put tactile into a distilled behavior prior whose latent space is the substrate for downstream residual adaptation — which is architecturally closer to the residual-latent-on-frozen-base recipe from Proxy Policy Steering: Adapting Frozen Robotic Foundation Models via Velocity-Space Proxy Residuals (Proxy Policy Steering) than to any of the tactile-integration recipes above. It also complements ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learning ADEPT’s fully-sim-trained multi-tactile-fingertip RL result by proposing a training-recipe answer to why raw-action PPO stalls in this regime.

See also

• Tactile sensing for manipulation — TacBPM is a new integration pattern (tactile-conditioned distilled prior + residual latent adaptation) on the concept’s core “how should touch enter the policy” axis.
• VLA Models — residual-latent adaptation on a frozen prior parallels the frozen-base steering thread (Proxy Policy Steering: Adapting Frozen Robotic Foundation Models via Velocity-Space Proxy Residuals) despite not being a VLM+action-head model.
• T-Rex: Tactile-Reactive Dexterous Manipulation — nearest tactile-integration sibling; both use a specialized tactile pathway but T-Rex partitions by rate (fast tactile expert vs. slow visuomotor) rather than by phase (distilled prior vs. residual head).
• ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learning — closest task neighbor: fully-sim-trained RL on a 29-DoF Sharpa hand with vision-based tactile fingertips, zero-shot sim-to-real; TacBPM proposes a training-recipe answer to the raw-RL-fails-on-high-DoF regime ADEPT also sidesteps.
• SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation — sibling recipe for zero-shot dexterous tool manipulation via object-centric goal-pose-conditioned RL; TacBPM’s Grasp-to-AnyPose task is the arm-hand generalization of the same problem class.