Pixel: A Co-Designed Full-Stack Robot System for Learning General-Purpose Mobile Manipulation

TL;DR

Pixel (CMU; Tao, Liu, Salakhutdinov, Pathak) is a full-stack co-designed robot learning system for general-purpose mobile manipulation. One wheeled humanoid platform (< $15K) shares human-aligned kinematics and compliant grippers with a sensorized human data-collection system (<$ 1K) so camera viewpoints, gripper geometry, and task-relevant motion align between demonstrator and robot. Co-designed with it: multiple teleoperation + robot-free human data-collection interfaces, a unified whole-body controller based on geometric fabrics, and a common policy learning pipeline that treats human and robot demonstrations the same way. A hierarchical learning stack enables language-conditioned visuomotor control and long-horizon tasks. The headline claim is co-design across hardware, data, control, and learning — not any single lever — as the recipe.

Key claims

A wheeled-humanoid platform under $15K + sensorized human data-collection system under$ 1K can be co-designed to share camera viewpoints, gripper geometry, and task-relevant motion, supporting transfer from human demonstrations to robot deployment [§System Co-Design / Hardware Design].
Geometric fabrics provide a unified whole-body controller usable for both teleoperation and policy rollouts, yielding smooth safe motion under one interface [§Interface Design / Whole Body Controller].
Three complementary teleoperation interfaces (compared under the same unified whole-body controller) span the data-collection design space for expressive whole-body control [§Teleoperation Interfaces].
A low-cost 3D-printed sensorized human data-collection system captures fine-grained dexterous demonstrations and supports untethered whole-body mobile data collection [§Sensorized Human Data].
A common policy-learning pipeline treats human and robot demonstrations uniformly; a hierarchical learning stack adds language conditioning and long-horizon task execution on top [§Policy Design].
Autonomous behaviors demonstrated on the real platform span dexterity, active perception, navigation, and whole-body coordination [§Tasks / Autonomous Tasks].

Method

Pixel’s design is organized as four co-designed layers. Hardware: a wheeled-humanoid platform under $15K is paired with a sensorized human data-collection rig under$ 1K; human-aligned kinematics and compliant grippers are matched across both so a human demonstrator’s workspace, camera views, and gripper geometry transfer to the robot without retargeting. Controller: a geometric-fabrics whole-body controller provides one smooth, safe interface that both teleoperators and learned policies invoke, so policy rollouts and teleoperation share dynamics. Data collection: three teleoperation interfaces (expressive whole-body) plus a wearable sensorized human rig for robot-free collection — including untethered mobile. Policy: a common pipeline consumes both demonstration sources, and a hierarchical stack adds language conditioning + long-horizon execution. This is the same architectural “make the collection tool embodiment-invariant” bet as XR-1 (Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training) and mimic (Solving Dexterity: A Full-Stack Approach (mimic hand M1 + wearable U1)), extended one level further to a wheeled-humanoid target rather than a stationary dexterous hand.

Results

Reported empirically rather than numerically in the abstract / project page: (a) mobile-manipulation behaviors on the real platform requiring dexterity, active perception, navigation, and whole-body coordination; (b) support for sensorized-human-only, robot-only, and teleoperated-task policy training under the same pipeline; (c) long-horizon manipulation and robustness demos in the project page’s “Autonomous Tasks” section. The companion paper Pixel-UMI: Learning Environment-Aware Whole-Body Manipulation from Sensorized Human Demonstrations carries the quantitative Pixel-UMI results (87.0% average task progress on five challenging tasks spanning articulated-object manipulation, deformable-object handling, constrained-space reaching, whole-body collision avoidance, and contact-rich bimanual coordination).

Why it’s interesting

Pixel pushes the “co-design the collection tool, controller, and policy together” position (Solving Dexterity: A Full-Stack Approach (mimic hand M1 + wearable U1) mimic, Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training XR-1, TwinDEX: A Scalable Path from Robot-Free Data Collection to Real-World Dexterous Manipulation TwinDEX, Getting a Grip on Robotic Data Collection — Co-Designed Koala Gripper for Force-Mediated Manipulation RAI Koala) from parallel-gripper and single-hand regimes up to a wheeled-humanoid with whole-body control, with sub-$16K BOM for the full system. Direct sibling to WARP: Whole-body Retargeting from Offline Human Motion via Closed-Form c-SEW WARP on the whole-body-mobile-manipulation axis but attacks the problem at the hardware+collection-tool layer rather than at the IK-solver layer. The companion Pixel-UMI paper (Pixel-UMI: Learning Environment-Aware Whole-Body Manipulation from Sensorized Human Demonstrations) is where the policy-learning contributions live.