Inspect Robots: Evaluating the Capabilities and Safety of Embodied AI

TL;DR

Inspect Robots is the paper companion to the stable release of the Robocurve-maintained open-source evaluation framework for embodied AI, formalizing the two-slot (Policy, Embodiment) design introduced in the alpha (Inspect Robots — An Open-Source Evaluation Framework for Physical AI (Robocurve)) and demonstrating it by evaluating six policies based on frontier language models. The framework pairs customizable reusable abstractions for specifying physical evaluations and analyzing their results with infrastructure that automates evaluation setup, execution, and termination [§Abstract]. Framed explicitly around capabilities and safety of embodied agents — the paper’s bet is that as general-purpose language models increasingly control robotic hardware, embodied capability + safety evaluations need a shared substrate on par with Inspect AI for LLMs. Reported adoption: nearly 100,000 downloads in the three months since release [§Abstract].

Key claims

A shared, modular, open-source framework for embodied agent evaluations is positioned as necessary infrastructure now that general-purpose language models increasingly control robotic hardware [§Abstract].
Inspect Robots pairs customizable, reusable abstractions for specifying physical evaluations and analyzing their results with infrastructure that automates evaluation setup, execution, and termination [§Abstract].
The framework is demonstrated by evaluating the capabilities and safety of six policies based on frontier language models [§Abstract].
Inspect Robots has seen significant early uptake, receiving nearly 100,000 downloads in the three months since its release [§Abstract].

Method

The paper formalizes the design already shipped in the open-source stable release: a Task = scenes + controller + scorer decomposition, and an eval(task, policy, embodiment) entry point that takes two slot inputs — the policy under test (a VLA served over WebSocket, a frontier LLM through an OpenAI-compatible API, or a code-as-policy agent) and an embodiment (real arm, humanoid, simulator, or mock) — rather than the single Model slot Inspect AI uses for LLM evals. A pre-actuation compatibility check validates action/observation spaces, semantics, control rates, and scene realizability before any robot moves. Every evaluation run produces a schema-versioned immutable EvalLog capturing config, git revision, package versions, grader scores, LLM transcript, and optionally camera frames, re-readable across releases and re-scorable offline. See the stable repo Inspect Robots — An Open-Source Evaluation Framework for Physical AI (Robocurve) for the mechanical details (six real-robot embodiment plugins, 40+ XPolicyLab-served VLAs, Isaac Lab + ROS 2 + CubePick mock, frontier-LLM-as-controller + CaP-X code-as-policy plugins, Rerun streaming, EvalLog summarize-and-retry loop).

Results

The abstract reports the framework demonstrated by evaluating the capabilities and safety of six policies based on frontier language models [§Abstract] and adoption of nearly 100,000 downloads in the three months since release [§Abstract]. Per-policy or per-task numbers are not retrievable from the arXiv abstract at filing time — the three already-filed real-hardware results running on the Inspect Robots harness (Claude Opus 5 driving robot arms zero-shot via Inspect Robots (Jay Chooi / Robocurve), Gemini Robotics-ER 2 vs Claude Opus 5 clapboard eval — ER2 threw robot arms off table, hallucinated success (Robocurve / Jay Chooi), Gemini 3.7 Flash saturates Robocurve physical tool-use benchmark at 92% vs 32% for 3.6 Flash (three weeks apart), plus Comparing Fable 5.1 to Fable 5 on robotic manipulation) stand as external evidence of the framework’s shape.

Why it’s interesting

The paper companion to the alpha-stage Inspect Robots — An Open-Source Evaluation Framework for Physical AI (Robocurve) release moves Inspect Robots from “shipping code” to a peer-review-grade claim: the same two-slot design that works for LLM evals (Inspect AI) is what embodied AI evals need. This turns the real-hardware eval substrate position on RL Environment Platforms into a citable artifact other labs can replicate against — a lever change from “Robocurve has a harness” to “the harness is a published framework with a design rationale behind it”. The 100K downloads in three months [§Abstract] is the sharpest filed adoption datapoint on real-hardware eval substrates to date, bounded on the other side by RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies‘s distributed crowd-sourced protocol and Reality Check — Pokeandwiggle's Real-World Robot Manipulation Leaderboard on an FR3 Duo Farm‘s centralized single-embodiment farm.

The explicit capabilities and safety framing aligns with the Science of Physical AI Safety workshop at CoRL 2026 where the paper is presented — positions Inspect Robots as the physical-AI counterpart to the role Inspect AI plays in LLM safety evals at UK AISI and frontier labs. Direct complement to Previewing the Model Hardware Standard‘s Model Hardware Standard (actuator-side protocol) on the eval-side; the two compose (an MHS-compliant robot is a valid Embodiment plugin). Also sharpens the moving-target story under VLA Models: the three prior Chooi/Robocurve tweets’ small-N methodology now has a published audit substrate behind it that can be re-scored offline against updated graders.