Skip to content
Bud's Wiki

RoboLab — Task-Based Evaluation Benchmark for Robot Manipulation Policies on Isaac Lab (NVLabs)

TL;DR

Section titled “TL;DR”

RoboLab (NVLabs / RSS 2026) is a task-based evaluation benchmark for robot manipulation policies built on NVIDIA Isaac Lab, packaged as a pip-installable stack with a server-client policy architecture and multi-environment parallel evaluation. The initial release (RoboLab-120) ships 120 brand-new manipulation tasks — pick-and-place, stacking, rearrangement, tool use — each with language instructions and automated success/failure detection via composable predicates. Tasks are not tied to a specific robot embodiment (any IsaacLab-compatible robot plugs in), a self-contained web dashboard replays episode videos and does cross-experiment comparison, and Claude Code skills (/robolab-scenegen and /robolab-taskgen) generate new scenes/tasks in minutes. The associated paper is arxiv 2604.09860; the Slack context puts this at ~30 GPU-hours per 100 tasks (Sid reports 4-5 hours per checkpoint on the RoboLab-120 set).

Key claims

Section titled “Key claims”
  • 120 initial tasks span pick-and-place, stacking, rearrangement, and tool use, each with language instructions and automated success/failure detection via composable predicates [Key Features].
  • Tasks are embodiment-agnostic: any IsaacLab-compatible robot can plug in, decoupling policy evaluation from a specific robot arm [Key Features].
  • Multi-environment parallel evaluation: multiple episodes run in parallel across environments with vectorized conditionals and per-environment termination [Key Features].
  • Server-client policy architecture — the policy runs as a standalone server and RoboLab connects via a lightweight inference client — so a submitter can serve a proprietary model without releasing weights, and one policy server can fan out across evaluators [Run with a policy].
  • Two Claude Code skills (/robolab-scenegen and /robolab-taskgen) generate new scenes and tasks from natural language “in minutes”, with a curated library of objects, scenes, and backgrounds as reusable primitives [Key Features, AI-Enabled Workflows].
  • Contact-rich dynamics are not invariant across IsaacSim 5.0 vs 5.1 (different PhysX builds), so benchmark results are best compared against runs on the same stack, and recorded demonstrations replay most faithfully on the stack they were recorded with [Requirements → Note on simulator versions].
  • Recommended hardware: NVIDIA RTX GPU, 48GB+ VRAM, ~8 GB disk (7 GB assets); reported speed 30 GPU hours / 100 tasks at ~1.4 it/s assuming a ~200 ms inference step [Requirements].

Method

Section titled “Method”

RoboLab is packaged as a uv-managed Python project with a mutually-exclusive extras choice at install time — isaac50 (IsaacSim 5.0 / IsaacLab 2.2.0, default) or isaac51 (IsaacSim 5.1 / IsaacLab 2.3.2.post1) — reflecting that PhysX contact dynamics differ across the two stacks and results are only comparable within a stack. A task is defined by a Python class with (a) an environment config combining a task, robot, observation, and action spec, (b) automated success-check predicates that are composed vectorized-across-envs and evaluated at each step, (c) optional subtask progress tracking that emits a score/reason breakdown in results, and (d) an initial-state randomization routine. The eval loop is a Ray-style server-client split: the policy under test exposes an HTTP inference endpoint from a separate GPU pool, and RoboLab’s client sends observations and receives actions each control step. Vectorized IsaacLab environments run in parallel (default 10, tunable via --num-envs), and per-env termination lets successful envs return early while others continue. Episodes are recorded to HDF5 with the exact env config (env_cfg.json) so faithful replay is possible on the same stack. A robolab-dashboard command starts a self-contained web viewer over output/ that browses scenes/tasks, replays videos, and does cross-experiment analysis.

Results

Section titled “Results”

The README does not publish leaderboard numbers directly (those live at the linked NVIDIA leaderboard page), but the shipped-artifact figures are: 120 tasks (RoboLab-120), a curated library of objects / scenes / backgrounds sized for building new tasks, ~30 GPU-hours per 100 tasks throughput, and a working policy example — Pi0.5 via OpenPI running on BananaInBowlTask with 10 parallel envs. Two sanity-check tasks are called out as good starting points (BananaInBowlTask, RubiksCubeAndBananaTask), and one gripper-toggle example (run_gripper_toggle.py) tests the gripper action path in isolation. From the Slack pointer: Sid reports 4-5 hours per checkpoint to run the full RoboLab-120 set, consistent with the README’s 30-GPU-hours-per-100-tasks headline number scaled to 120 tasks on a single GPU.

Why it’s interesting

Section titled “Why it’s interesting”

RoboLab is the sharpest recent instance of the “package robotics sim as a pip-installable benchmark with a policy-agnostic HTTP interface” pattern this wiki has catalogued in RL Environment Platforms. Where Genesis World — Simulation platform for general-purpose robotics & embodied AI learning and The Role of Simulation in Scalable Robotics, Genesis World 1.0, and the Path Forward ship the underlying simulator as the research artifact (physics engine + renderer + compiler), RoboLab commits to Isaac Lab and treats the value-add as the task library + automated success detection + dashboard. It complements RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies (Isaac Sim 42-task benchmark + 18 real tasks, 30-policy leaderboard) and RoboCasa365 — Large-Scale Simulation of Everyday Tasks for Generalist Robots (365 LLM-authored kitchen tasks, 2,500+ scenes) — all three now ship an Isaac-family sim + a public leaderboard, and RoboLab’s contribution is embodiment-agnostic tasks with composable-predicate verifiers and Claude-Code skills for extension. The Slack context — Sid running RoboLab-120 on VLA checkpoints for 4-5 hours per checkpoint, paired with the Delphi: Scaling Laws That Extrapolate 300× Past the Fit “predict downstream from cheap runs” methodology — is a concrete case of the Scale Robot Policy Evaluation with Ray (Distributed Sim-Eval on Anyscale) “sim-eval as a first-class evaluation surface for scaling studies” thesis: sim-eval is the sim analog of Delphi’s downstream soft-metric fit, and the two together are the ingredients for a proper VLA scaling law over pretraining compute.

See also

Section titled “See also”