UMI-Bench 1.0: An Open and Reproducible Real-World Benchmark for Tabletop Robotic Manipulation with UMI Data

TL;DR

UMI-Bench 1.0 is a local-first real-robot benchmark for evaluating policies trained on Universal Manipulation Interface (UMI) data — the hand-held, wrist-camera-only teleop format whose policies depend tightly on the coupling between wrist-view observations, action representation, data collection, and physical deployment. The benchmark bundles data collection, scene reset, policy execution, result logging, and task-factor analysis into a single reproducible protocol across nine tabletop tasks (Trash Bag, Stamp Ink, Remote Storage, Pour Beans, Pack & Carry, Turntable Pick, Mahjong Sort, Kitchen Rearrange, Pants Folding), each evaluated under both seen and unseen conditions. It is positioned as the first benchmark dedicated to real-world evaluation of UMI-based manipulation models, complementing distributed crowd-sourced platforms like RoboArena with a self-contained, auditable workstation setup.

Key claims

UMI-style policies’ performance depends on a coupling of wrist-view observations, action representation, data collection process, and physical deployment that prior real-world benchmarks were not designed around [Abstract].
UMI-Bench 1.0 is, to the authors’ knowledge, the first benchmark dedicated to real-world evaluation of UMI-based manipulation models [Abstract].
The benchmark aligns demonstration capture, UMI data samples, task setup, real-world evaluation workstations, and rollout logging into one unified protocol [project page §Data-to-evaluation Pipeline].
Each of nine tabletop tasks is evaluated under both a seen and an unseen condition, with all rollouts logged and the protocol auditable [project page §Seen and unseen rollouts].

Method

UMI-Bench is structured as a data-to-evaluation pipeline: hand-held UMI grippers (wrist-camera-only, no external observers) are used to collect demonstrations; a fixed task setup specifies the scene, objects, and reset procedure; a real-world evaluation workstation runs the trained policy; and a rollout-logging system captures execution traces and outcomes, decomposed by per-task factors (object pose, distractor, novel-instance generalization, lighting, etc.). The benchmark ships with nine tabletop manipulation tasks spanning rigid pick-and-place (Turntable Pick, Mahjong Sort), deformable / granular manipulation (Trash Bag, Pour Beans, Pants Folding), articulated objects (Remote Storage), and multi-step composite behaviors (Stamp Ink, Pack & Carry, Kitchen Rearrange). For each task, the protocol distinguishes seen conditions (matched to the training distribution) and unseen conditions (held-out object instances, layouts, or distractors), with one representative seen-condition and one unseen-condition rollout shown publicly per task.

Results

The arXiv abstract and project page describe the benchmark and protocol rather than reporting numerical results for any specific policy — UMI-Bench is presented as the evaluation substrate, not as a leaderboard at filing time. Per-task seen / unseen rollout pairs are released as web-optimized clips on the project page [project page §Seen and unseen rollouts]. Quantitative success rates, factor-decomposition tables, and policy comparisons are referenced as part of the unified protocol but are not the focus of the v1.0 release.

Why it’s interesting

UMI-Bench fills a gap in the robot-evaluation landscape that the team has been actively tracking: RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies approaches generalist-policy evaluation through distributed, crowd-sourced pairwise A/B comparisons across many labs — strong on diversity but vulnerable to integrity issues, as the RoboArena rolls back evaluations after benchmark hacking observed since April (Pranav Atreya announcement) retraction made concrete — whereas UMI-Bench takes the opposite bet of a self-contained, auditable single-workstation protocol, trading scale for reproducibility. It also complements the UMI-data-collection toolkits and stack-bundled rigs tracked in this wiki — see RAIDEN: A Toolkit for Policy Learning with YAM Bimanual Robot Arms (RAIDEN’s bimanual YAM toolkit) and ABC: Scalable Behavior Cloning with Open Data, Training, and Evaluation (ABC’s passive-leader DAgger rig + abcdl) — by sitting downstream of them on the evaluation side rather than the data-collection side. For Luma’s interest in VLA / world-action models, the per-task seen-vs-unseen split is the more useful primitive than aggregate score: it cleanly isolates generalization axes that VLA pretraining (egocentric video, world-model imagination, etc.) is supposed to unlock.