Inspect Robots — An Open-Source Evaluation Framework for Physical AI (Robocurve)

TL;DR

Inspect Robots is the MIT-licensed evaluation harness underneath the three prior Robocurve real-hardware evals filed on the wiki (Claude Opus 5 driving robot arms zero-shot via Inspect Robots (Jay Chooi / Robocurve), Gemini Robotics-ER 2 vs Claude Opus 5 clapboard eval — ER2 threw robot arms off table, hallucinated success (Robocurve / Jay Chooi), Gemini 3.7 Flash saturates Robocurve physical tool-use benchmark at 92% vs 32% for 3.6 Flash (three weeks apart)); the source code and the umbrella Public Benefit Corporation (Robocurve) were formally announced together in Jay Chooi’s Aug 28 2026 tweet thread. The framework is explicitly modeled on Inspect AI (UK AISI’s LLM eval harness) — a Task = scenes + controller + scorer decomposition, an eval() → EvalLog entry point, a plugin registry, and pluggable Policy (VLA or LLM) and Embodiment (real arm, humanoid, or simulator) slots that are compatibility-checked before any hardware moves. Ships adapters for six real-robot families (I2RT YAM, Franka FR3/Panda, AgiBot A2 Ultra, Unitree G1, SO-ARM100/101, WidowX 250S), Isaac Lab simulation, ROS-1/ROS-2 arms via rosbridge, and a dependency-free CubePick mock world; policy plugins cover 40+ served VLAs via XPolicyLab, frontier LLMs via --policy agent, code-as-policy via CaP-X (SAM3 + Contact-GraspNet + Pyroki), and voice-in-the-loop feedback.

Key claims

Two-input eval decomposition — (Policy, Embodiment) — is validated for action/observation spaces, semantics, control rate, and scene realizability before any rollout, so an incompatible pairing fails at setup time rather than mid-episode with a moving robot [README §Why Inspect Robots].
Every run yields a schema-versioned, immutable EvalLog capturing the resolved config, git revision, package versions, grader scores, LLM transcript, and (optionally) camera frames, re-readable across releases and re-scorable offline [README §Why Inspect Robots].
Real-world-first interfaces are the default: human-in-the-loop reset, no privileged success oracle, wall-clock control rate; simulators expose seeding / privileged success / rendering as opt-in capabilities on top [README §Why Inspect Robots].
The framework core depends only on NumPy; Rerun visualization, simulator backends, and VLA backends are all separately installable plugins registered via Python entry points [README §Why Inspect Robots, §Install].
VLA-native primitives are built into the controller layer: action chunking, open-loop execution, and ACT/ALOHA temporal ensembling, with action semantics (control mode, rotation representation, gripper, frame) explicit enough to make compatibility and ensembling correct across embodiments [README §Why Inspect Robots].
Six real-robot embodiment plugins are shipped by default, each pairing an Embodiment with the Policy clients that speak its action contract — YAM (14-D joint_pos, molmoact2 + gr00t), Franka FR3/Panda (8-D joint_pos, openpi), AgiBot A2 Ultra (16-D joint_pos, go1 + openpi), Unitree G1 (16-D joint_pos, gr00t), SO-ARM100/101 (6-D joint_pos, lerobot), WidowX 250S (7-D eef_delta_pose, openvla + openpi) [README §Supported embodiments].
The --policy agent plugin puts a frontier LLM (Claude, GPT, anything behind an OpenAI-compatible API) into the same policy slot as a fine-tuned VLA, driving any embodiment through motion tool calls with a programmatic pre-check hook that can return correctable rejection reasons before absolute action chunks execute [README §Drive the robot with an LLM, plugins/inspect-robots-agent/].
The --policy capx plugin evaluates CaP-X-style code-as-policy agents against joint-space embodiments; the LLM writes Python against separately served SAM3 segmentation, Contact-GraspNet planning, Pyroki IK, and speed-limited joint-motion helpers [README §Generate robot policy code with CaP-X].
The --policy xpolicylab plugin adapts XPolicyLab’s zoo of 40+ VLAs (π0 / π0.5, GR00T, OpenVLA-OFT, RDT-1B, SmolVLA, ACT, …) behind one policy interface with a WebSocket URL parameter [README §First-party plugins].
Every run streams cameras, proprioception, and actions to a Rerun viewer non-blockingly — logging drops camera frames first (whole steps only under sustained stall) so the control loop is never delayed by a slow viewer connection, and camera streams are JPEG-compressed by default [README §Quickstart, §Why Inspect Robots].
Safety guardrails (bounds clamp + per-step delta limit derived from the embodiment’s action space) are wired into every CLI run by default for every policy; disabling them requires an explicit --disable-guardrails [README §Real robots via ROS].
The EvalLog → summarize command distills failed logs into a markdown “learnings” file whose resolved path and content hash are recorded in the next agent run’s eval configuration, giving a reproducible retry-with-learning loop [README §Retry with learning].
The Inspect-AI → Inspect-Robots API mapping is one-to-one: Model → Policy + Embodiment (two inputs); Task = dataset + solver + scorer → Task = scenes + controller + scorer; Sample → Scene; Solver chain → Controller middleware; eval() → EvalLog on both sides; @task/@solver/@scorer registry → @task/@policy/@embodiment/@scorer entry points [README §How it maps to Inspect AI].
Concrete benchmarks (task sets, scenes, scorers) do not live in this repo — they live in a companion WorldEvals catalog, and backend adapters live in separate plugin packages, keeping the framework itself a thin substrate rather than a benchmark [README §How it maps to Inspect AI].

Method

Inspect Robots is a Python-first eval framework with a CLI (inspect-robots) and a matching programmatic API. A Task is scenes + controller + scorer, where scenes carry per-episode init conditions plus an instruction, controller middleware handles action chunking / ensembling / smoothing, and the scorer produces both scalar success metrics and free-form eval-log metadata. The eval(task, policy, embodiment) call returns an EvalLog — an immutable, schema-versioned record — and simultaneously streams to a .rrd file for Rerun replay.

The load-bearing design choice is the two-slot input signature: instead of a single Model (as in Inspect AI’s LLM-only case), Inspect Robots takes both a Policy (the thing being evaluated: a VLA served over WebSocket, a frontier LLM through OpenAI-compatible API, or a code-as-policy agent) and an Embodiment (the thing the policy acts through: a real robot arm, a humanoid, an Isaac Lab simulation, or the dependency-free CubePick mock). A check_compatibility(policy, embodiment) step runs before any actuation and fails fast on mismatched action spaces, control rates, or scene requirements. This is what turns “swap in a new frontier LLM and see how it does on the YAM arm” from a config-editing project into a single flag change.

Each embodiment plugin ships both halves of the eval — the embodiment adapter and the policy clients that speak its action contract — inside one installable package (inspect-robots-yam, inspect-robots-franka, etc.), registered via Python entry points so inspect-robots list picks them up automatically. Simulation and mock backends (Isaac Lab and CubePick) follow the same plugin pattern. The ros embodiment is the general escape hatch: any ROS-1/ROS-2 arm publishing standard joint + compressed-image topics works through rosbridge without a dedicated package.

The policy layer is deliberately promiscuous. --policy agent (frontier LLM through motion tool calls with a per-turn approver-checked motion chunk), --policy capx (code-as-policy against SAM3 + Contact-GraspNet + Pyroki), --policy xpolicylab (40+ served VLAs), --policy molmoact2, --policy openpi, --policy scripted (mock for testing) are all interchangeable. Voice-in-the-loop feedback via --voice transcribes locally (no keys, no network) and delivers spoken remarks at the next inference boundary; spoken input is feedback-only, so trial-end verdicts stay on the keyboard.

The inspect-robots view command renders per-log HTML reports plus a browsable directory-level index; served with --serve --open it auto-refreshes as new runs land, and agent runs update their HTML report turn-by-turn while a run is active. The inspect-robots video command renders --store-frames runs to MP4 via ffmpeg. Together with the Rerun .rrd stream, this is what makes the small-N-per-model 5-rollout methodology Robocurve has been using in Claude Opus 5 driving robot arms zero-shot via Inspect Robots (Jay Chooi / Robocurve) / Gemini Robotics-ER 2 vs Claude Opus 5 clapboard eval — ER2 threw robot arms off table, hallucinated success (Robocurve / Jay Chooi) / Gemini 3.7 Flash saturates Robocurve physical tool-use benchmark at 92% vs 32% for 3.6 Flash (three weeks apart) auditable end-to-end — every rollout is a schema-versioned log with the camera stream attached.

The summarize LOG.json command produces either a deterministic offline digest (no model) or an LLM-generated markdown “learnings” file (with a model flag against an OpenAI-compatible chat endpoint), whose resolved path and content hash are then recorded in the next agent run’s eval configuration for reproducible retry-with-learning.

Results

Not a paper — no benchmark numbers of its own. The framework’s evaluation output has already produced three filed real-hardware results on the wiki: Claude Opus 5 zero-shot bowl-stacking on YAM (Claude Opus 5 driving robot arms zero-shot via Inspect Robots (Jay Chooi / Robocurve)), Gemini Robotics-ER 2 vs Claude Opus 5 head-to-head on the clapboard task (0/5 for ER 2, with hallucinated-success and hardware-damaging failure modes; Gemini Robotics-ER 2 vs Claude Opus 5 clapboard eval — ER2 threw robot arms off table, hallucinated success (Robocurve / Jay Chooi)), and Gemini 3.7 Flash vs 3.6 Flash on a physical tool-use “cover a block with a towel” task (92/100 vs 32/100 on a 5-rollout partial-credit rubric, per-turn latency 9.0 s vs 17.5 s; Gemini 3.7 Flash saturates Robocurve physical tool-use benchmark at 92% vs 32% for 3.6 Flash (three weeks apart)).

Repo state at filing time is alpha, MIT-licensed, uv-installable via pip install "inspect-robots[rerun]" or Python-only core, mypy-strict typed, 100% test and docstring coverage per the CI badges, with docs at docs.inspectrobots.org. Cited as @software{inspect-robots, author=Robocurve, year=2026}.

Why it’s interesting

Inspect Robots is the first filed substrate datapoint on the wiki that fills the “where does the harness code actually come from” gap under RL Environment Platforms. The three prior Chooi/Robocurve tweets have all cited github.com/robocurve/inspect-robots, but the code itself is what turns the small-N 5-rollout methodology from a one-lab practice into something a second lab could reproduce. This lands next to SETA: Scaling Environments for Terminal Agents (env repo) (terminal), Toolathlon-GYM: Large-Scale Long-Horizon Environments for Tool-Use Agents (enterprise workflows), and OpenReward — 330+ RL Environments through one API (General Reasoning announcement) (OpenReward as a managed-service platform for RL environments) as the fourth open-sourced env-substrate in the cluster, and the first one whose target is real hardware. The design decisions echo the pattern: verifier-cheap / action-expensive (rubric grader is a fast function; camera-recorded 5-rollout hardware runs are the expensive action), decoupled env compute (the policy server runs on its own GPU box, the embodiment runs on the robot host, the eval client is a laptop), and standardized entry points so a new rig integration is a plugin release rather than a fork.

Second, the Inspect AI → Inspect Robots mirroring is a bet worth naming: UK AISI’s Inspect AI has become the de-facto substrate for LLM safety and capability evals inside frontier labs, and Robocurve is explicitly betting that the same separation — Task = scenes + controller + scorer, plugin-registered Policy and Embodiment, immutable EvalLog, one-shot compatibility check before actuation — is what the physical-AI eval ecosystem is missing. If that bet lands, the “safety eval / third-party eval” role currently played by inference-time harnesses for chatbots gets a hardware counterpart with the same audit properties. The complement to this is on the deployment-protocol side: Previewing the Model Hardware Standard‘s Model Hardware Standard proposes the actuator-side protocol (read / write primitives with natural-language device tags), while Inspect Robots owns the eval-side protocol; the two are non-overlapping but naturally compose (an MHS-compliant robot is a valid Embodiment implementation).

Third, this filing changes the interpretation of the three prior Chooi tweets. Reading them in isolation, the 5-rollout small-N methodology looks fragile — small numbers, one operator, one hardware setup. Reading them with the harness open-source and CI-tested, the operator variance and setup variance are recoverable from the EvalLog artifacts (every run is git-pinned, config-locked, and camera-recorded), and the small-N is a per-refresh-cycle sampling choice rather than a methodological ceiling. The VLA Models page’s discussion of the LLM-as-controller pattern, VLM-as-Evaluator‘s discussion of judge-backbone sensitivity, and VLM Perception Failures‘s discussion of the hallucinated-success mode all now have a substrate they can replicate against.