RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies

TL;DR

RoboDojo is a unified sim-and-real benchmark for generalist robot manipulation policies that spans 42 simulation tasks (Isaac Sim, heterogeneous parallel) and 18 real-world tasks, evaluating five capability dimensions in simulation (generalization, memory, precision, long-horizon execution, open-vocabulary instruction following) and exposing real deployment challenges in the real-world track. It ships alongside XPolicyLab — an integration layer that lets a policy be wired in once and run across both sim and real — and RoboDojo-RealEval, a reproducible real-world evaluation system with remote cloud access, standardized hardware, and a scene-reset + deployment interface. The authors integrated 30 policies into XPolicyLab, evaluated them on RoboDojo, and released a public leaderboard with systematic analysis of current policy performance.

Key claims

Existing benchmarks are narrow along three axes at once — task horizon, capability coverage, and evaluation modality (sim-only or real-only) — leaving generalist manipulation policies without a systematic evaluation surface [Abstract].
Simulation and real-world evaluation each cover the other’s blind spots: simulation gives scalable feedback but misses physical-deployment challenges, while real-world evaluation is costly, slow, and difficult to reproduce; a benchmark must couple both to trust the ranking it produces [Abstract].
RoboDojo’s simulation track evaluates five explicit capability dimensions — generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following — across 42 tasks; the real-world track adds 18 tasks under deployment-realistic conditions [Abstract].
Scalable simulation evaluation is achieved via heterogeneous parallel simulation on Isaac Sim, letting many task-scene combinations run concurrently on the same hardware [Abstract].
RoboDojo-RealEval externalizes the real-world evaluation infrastructure: remote cloud access to standardized hardware, standardized scene-reset protocol, evaluation protocol, and deployment interface — the reproducibility gaps that ordinarily block cross-lab real-robot comparison are moved into the platform [Abstract].
XPolicyLab is the integration substrate that lets any policy plug in once and be evaluated across simulation and real-world settings with minimal adaptation; the authors report 30 policies integrated and evaluated to seed the leaderboard [Abstract].
The paper releases a public leaderboard and a systematic analysis of the 30-policy sweep as the initial evidence base for what current generalist policies can and cannot do [Abstract].

Method

RoboDojo is structured as three coupled artifacts. First, a task suite: 42 simulation tasks in Isaac Sim, organized so that each of the five evaluation dimensions (generalization, memory, precision, long-horizon execution, open-vocabulary instruction following) is exercised by a dedicated task cluster; and 18 real-world tasks selected to expose challenges (contact dynamics, sensing noise, unmodeled friction, scene drift) that simulation elides. Second, a runtime: heterogeneous parallel simulation in Isaac Sim on the sim side, and RoboDojo-RealEval on the real side — the latter provides remote cloud access to a standardized robot cell, a fixed scene-reset protocol, a fixed evaluation protocol, and a common deployment interface so that a policy submitted by any lab can be executed and scored without on-site engineering. Third, a policy-integration layer, XPolicyLab, where a policy is wired in once and dispatched into either the simulation or the real-world runtime through the same interface. The authors integrated 30 existing generalist policies through XPolicyLab and evaluated the whole set on RoboDojo, publishing the results as a public leaderboard at robodojo-benchmark.com with code at github.com/RoboDojo-Benchmark/RoboDojo.

Results

The paper reports a systematic analysis of 30 policies evaluated across the 42-sim + 18-real task suite. Full task-level and per-dimension numbers are on the public leaderboard; the abstract-level headline is that the sweep is used to characterize current generalist-policy performance across the five capability dimensions and against real-world deployment conditions, rather than to advance a single new policy. The concrete measurable outputs are (a) the leaderboard itself, (b) a “systematic analysis of current policy performance” the authors summarize as motivating evidence for the benchmark’s design, and (c) a demonstration that the sim + real coupling ranks policies differently than either half alone would.

Why it’s interesting

RoboDojo fills a specific hole in the wiki’s evaluation-substrate map for generalist manipulation. The closest neighbours are RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies and RoboCasa365 — Large-Scale Simulation of Everyday Tasks for Generalist Robots: RoboArena is real-only + human-judged + crowd-sourced, RoboCasa365 is sim-only + kitchen-domain + auto-verified; RoboDojo is the union — sim + real, standardized hardware in the real half, capability-partitioned tasks in the sim half — and pushes the “policy-integration substrate” idea (XPolicyLab) as a first-class artifact alongside the tasks. It also contrasts with RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation, which replaces the physics simulator with a learned video world model as the sim half of the same trust problem; RoboDojo keeps Isaac Sim and instead invests in real-hardware reproducibility. The 30-policy sweep is the largest single-benchmark policy pool filed to date on the wiki (RoboArena’s paper covered 7, RoboCasa365 shipped 3 baselines) — the leaderboard is the piece most likely to be directly useful to a lab positioning a new policy.