Universal Manipulation Exoskeleton: Learning Compliant Whole-body Policies with Real-time Torque Feedback

TL;DR

UME is an upper-limb teleoperation exoskeleton at Inclusion AI Lab (Ant Group) + Stanford BDML that closes the “torque data gap” in current data-collection pipelines: it exposes real-time transparent haptic torque feedback to the operator and records whole-arm joint positions plus joint torques, so learned policies can act with active compliance in tightly constrained contact-rich spaces. Coaxial ball-joint actuators (Damiao / Unitree quasi-direct-drive) preserve the human arm’s full wrist and most of shoulder range of motion; a robot-agnostic sub-manipulator retargeting algorithm maps kinematics and haptic torques bidirectionally onto 7-DoF OpenArm, 7-DoF Franka, and 6-DoF X-ARM; an onboard IMU adds mobile-base control at $9 vs$ 300 for a VR alternative. Learned ACT-based policies with the torque modality reach 0.85–0.95 success on box flipping, box pushing, GPU picking between two desktops (5 mm clearance), and 1.5-minute bimanual mobile fridge-drink retrieval — no-torque and UMI-style end-effector-only baselines collapse to 0–0.5 on the same tasks.

Key claims

Real-time transparent haptic torque feedback plus recorded per-joint torques let ACT policies learn actively compliant behavior that keeps working in visually occluded, force-mediated, and space-constrained regimes where end-effector-only pipelines cannot [§1, §3].
Universal sub-manipulator retargeting — decomposing both exoskeleton and follower into a 3-DoF virtual spherical shoulder + 1-DoF elbow + 3-DoF virtual spherical wrist with independent Jacobians — teleoperates 7-DoF OpenArm, 7-DoF Franka (sim), and 6-DoF X-ARM from the same wearer, and avoids the 6-D end-effector-task-space singularities that make constrained-space control brittle [§2.2].
UME + torque hits 0.90 / 0.85 / 0.95 / 0.95 success on box pushing / box flipping / GPU picking / fridge drink retrieval vs the No-torque ablation at 0.50 / 0.00 / — / 0.90 and the UMI-style end-effector-only baseline at 0.40 / 0.00 / 0.00 / 0.00 (over 20 trials each) [Table 1, §3.1–3.4].
Both No-torque and UMI baselines fail box flipping outright (0/20) because the two states “in contact but insufficient pressing force” and “sufficient friction, initiate flip” look visually identical and can only be told apart via joint torque; UME’s torque channel makes the state distinction learnable [§3.2, Fig. 4(b)].
On GPU picking between two desktop PCs (5 mm clearance after insertion), UMI-style end-effector control collapses to 0/20 because the arm bends either before or after entering the constrained space; UME learns to fully straighten the arm (the only configuration that reaches the back of the enclosure) and to slide compliantly along the glass surface via torque regulation [§3.3, Fig. 4(c)].
Coaxial ball-joint actuator arrangement — J1/J2/J3 axes intersecting at the human shoulder ball joint, J5/J6/J7 axes intersecting at the human wrist ball joint — replicates the full human wrist RoM and most of shoulder RoM without structural links blocking human movement, a departure from Dynamixel-based isomorphic leader-follower designs like ALOHA / GELLO [§2.1, Fig. 2(b), Fig. 9].
Low reduction ratio quasi-direct-drive actuators (Damiao, Unitree) plus per-joint gravity/friction/Coriolis compensation are what makes transparent torque feedback feasible at low cost, low weight, and portable form factor [§1, §2, Appendix C].
On the highly force-mediated box-flipping task, UME’s torque feedback delivers a 3.3× data-collection throughput improvement over the same UME without torque feedback, operating at 71% of unassisted human demonstration speed [§3.5, Fig. 5].
Replacing a VR headset ( $300) with an onboard IMU ($ 9) for exoskeleton orientation detection and mobile-base velocity control drops the mobile teleoperation setup cost meaningfully while retaining holonomic mobile-manipulation coverage on an upgraded Tidybot++ base [§2.3, Appendix E.4].

Method

UME is a 7-DoF upper-limb exoskeleton whose mechanical structure mirrors the human 3-1-3 arm kinematic (shoulder ball joint + elbow hinge + wrist ball joint). Actuators are arranged coaxially so J1/J2/J3 rotational axes always intersect at the human shoulder ball-joint center regardless of angle, and J5/J6/J7 axes always intersect at the human wrist center. This is the design choice that eliminates structural links blocking movement and preserves human range of motion — the paper contrasts it explicitly with Dynamixel-based leader-follower designs (ALOHA, GELLO) that rely on high-ratio planetary gearboxes and cannot backdrive transparently. UME uses low-reduction quasi-direct-drive actuators (Damiao and Unitree) that support transparent haptic torque feedback with per-joint Coriolis / centrifugal / gravity / friction compensation.

The universal retargeting algorithm decomposes both the exoskeleton and the follower robot into three sub-manipulators: virtual 3-DoF spherical shoulder, 1-DoF elbow, virtual 3-DoF spherical wrist. Feedforward position, velocity, and haptic-torque signals are retargeted per sub-manipulator using the corresponding sub-manipulator Jacobian and its dynamically consistent (mass-weighted) inverse. Decoupling isolates each Jacobian inverse, avoiding the 6-D end-effector-space singularities that plague standard task-space control near kinematic boundaries — the property that makes tightly constrained spaces workable. An onboard IMU drives both gravity-compensation torque and, via linear mapping of Euler angles to base velocity, mobile-base control on the upgraded Tidybot++ holonomic platform.

Policies are learned with ACT (Action Chunking with Transformers). The torque modality is injected by processing per-joint torques the same way as joint positions, then appending the torque embedding to the joint-proprioception + ResNet18-image embedding stream fed to the transformer. Ablations vary this by removing torque (No-torque) or by replacing whole-arm proprioception with UMI-style end-effector 6-D pose + whole-body IK at inference (UMI).

Results

Table 1 reports 20-trial success rates per task. Box Pushing: UME 0.90, No-torque 0.50, UMI 0.40. Box Flipping: UME 0.85, No-torque 0.00, UMI 0.00. GPU Picking: UME 0.95, No-torque intermediate (severe scratching, incomplete forward motion), UMI 0.00. Fridge Drink Retrieval (~1 min 30 sec, bimanual, mobile): UME 0.95, No-torque 0.90, UMI 0 successful drink lifts.

Failure diagnostics: box-flipping failure for both ablations is inability to distinguish “insufficient pressing force” from “sufficient friction, initiate flip” — visually identical, only distinguishable via joint torque [Fig. 4(b)]. UMI GPU-picking failure is systematic arm-bending collision either before or after entering the 5 mm-clearance corridor [Fig. 4(c)]. UMI fridge failure is wrist-camera or gripper collision with the door holder while trying to grasp the can [Fig. 4(d)]. Fridge policy is robust to external disturbance: removing the drink after placement triggers autonomous fetch of another; removing the drink mid-transport causes the robot to drop its arms, return to the fridge, and resume from there.

Data-collection throughput: on box flipping (force-mediated), UME’s haptic feedback delivers 3.3× more demonstrations per minute than UME-without-torque, reaching 71% of unassisted human-hand demonstration speed [Fig. 5]. Universal teleoperation: UME successfully drives bimanual X-ARMs (block pick / transfer / place-in-cup) in the real world and bimanual Franka arms in simulation from the same operator.

Why it’s interesting

UME is the sharpest concrete answer yet to the Tactile sensing for manipulation “how does force/torque enter the policy” question at the data-collection layer rather than at the policy layer — complementary to FACT: Force-Aware Contact-Rich Manipulation via Timestep Modulation — demystifying when and why VLAs fail on contact-rich tasks FACT (per-layer force modulation of the VLA), ReForce — force-aware retargeting via closed-loop residual force tracker ReForce (sim-trained residual force tracker), and Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction Tactile-Rex (variable-rate MoT + per-finger VQ-VAE). The load-bearing move is that recording joint torques during teleop (rather than only end-effector poses as UMI-lineage systems do) is what makes downstream ACT policies capable of active compliance — a 30 pp shift on box pushing and a 0 → 85 pp shift on box flipping, both measured on the same collection substrate. Sibling on the recipe axis to Getting a Grip on Robotic Data Collection — Co-Designed Koala Gripper for Force-Mediated Manipulation Koala (mirrored fingertip force sensors across handheld and actuated variants) and ModPack: An Extensible Teleoperation Interface for Bimanual Mobile Manipulation ModPack (projected joint torques giving 30 pp on box-transfer contact tasks) — all three papers converge from different starting points on the position that force/torque channels in the data are what current VLA/BC recipes are missing for force-mediated contact-rich tasks.

For Human-to-Robot Retargeting the paper stakes out a distinctive position on the interface-location axis: rather than collapsing all human demonstrators onto one canonical robot representation (mimic, HandUMI, XR-1) or reducing to a low-DoF shared action space (Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots), UME keeps the demonstrator side human-shaped (7-DoF exoskeleton matching the 3-1-3 arm kinematic) and pushes the invariance into a per-sub-manipulator retargeting algorithm that works across robots of different DoF (7-DoF OpenArm, 7-DoF Franka, 6-DoF X-ARM) using virtual spherical joints — a sibling in framing to Cross-Embodiment Robot Manipulation via a Unified Hand Action Space UHAS’s canonical fingertip sphere and Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots Translation-as-Bridging-Action’s 3-DoF reduction, but on the arm rather than the hand level, and with haptic torques flowing back through the same virtual-joint decomposition.