Skip to content
Bud's Wiki

ORA: Open Robot-Use Agents

TL;DR

Section titled “TL;DR”

ORA (Open Robot-Use Agents, Hao AI Lab, Apache-2.0) is an open agent harness that lets open-weight language models operate real/simulated robots fully zero-shot — no VLA, no robot training, no task-specific memory. The agent is a direct loop between a model API (OpenAI-compatible or Anthropic) and the robot, with perception (SAM3 segmentation, depth geometry), grasp proposals (Contact-GraspNet), and motion planning (CuRobo) exposed as tools behind a supervisor that checks and executes every action. On RoboTwin 2.0 and LIBERO-Pro the harness reports 69.2% and 81.4% task success with DeepSeek open weights, which the authors claim is competitive with agents built on frontier models. Ships with LIBERO/LIBERO-Pro (Panda), RoboTwin 2.0 (dual-arm ALOHA), and RoboDojo (ARX, Isaac Sim 5.1) benchmarks running from one command on one GPU, plus a terminal UI (ora watch / ora interactive) and an evidence viewer for sealed replay.

Key claims

Section titled “Key claims”
  • Open-weight language models can operate robots fully zero-shot — no VLA, no robot training, no task-specific memory — through a tool-call loop over perception + grasp + motion-planning primitives, scoring 69.2% on RoboTwin and 81.4% on LIBERO-Pro [README §Overview].
  • The harness is designed around three ingredients for the physical world: exact task and environment context, tools for perception and motion, and context management [README §Overview].
  • Every action proposed by the model is checked and executed by a supervisor; the model cannot bypass it, and tool results that exceed the context window stay recoverable within an episode [README §What is in this repository].
  • The tool surface combines SAM 3 segmentation, depth geometry, Contact-GraspNet grasp proposals, and cuRobo motion planning as model-callable primitives [README §Acknowledgements, §What is in this repository].
  • Three benchmarks ship with isolated interpreters and each benchmark’s official success check: LIBERO / LIBERO-Pro (Panda), RoboTwin 2.0 (dual-arm ALOHA), and RoboDojo (ARX, Isaac Sim 5.1) [README §What is in this repository].
  • Every model request, response, tool call, action, and receipt is recorded and sealed; runs are replayable in-terminal via ora watch or in the evidence viewer, and interactive runs are marked human-assisted and not counted as autonomous [README §Watch and steer runs in the terminal].

Method

Section titled “Method”

ORA is structured as a Native agent connecting directly to the model API with no intermediate orchestration layer. Prompt bundles deliver the task description, the robot manual (hardware + action-space contract), and the tool contracts (function signatures + usage notes). The agent then runs a ReAct-style loop: emit a tool call (perception query, grasp proposal, motion plan, or direct action), the supervisor validates it, dispatches it against the live simulator or robot, and returns a condensed result that the agent reasons over before the next step. The simulator provides the official task verdict.

Tool contracts expose observations (SAM 3 segmentation, depth geometry) and bounded action proposals (Contact-GraspNet grasp candidates, cuRobo motion plans) rather than raw joint commands, which keeps the model’s output in a space the supervisor can validate before actuation. The acknowledgements explicitly cite Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents (Harness VLA, same lab lineage via the RPent team sharing eval cases + leaderboard), CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation (CaP-X), and ASPIRE: Agentic /Skills Discovery for Robotics (ASPIRE) as the design inspirations for the tool surface.

Benchmarks run in isolated Python interpreters (./scripts/ora run --env libero|robotwin|robodojo) with each benchmark’s official success check wired in, and the authors verify end-to-end on a single RTX PRO 6000 Blackwell with 32 GB+ VRAM. A no-GPU ora local path targets LIBERO on commodity hardware. The distribution is ora-agent on PyPI (pending) with Python 3.11 lockfile.

Results

Section titled “Results”
  • RoboTwin 2.0 (dual-arm ALOHA): 69.2% overall task success with open-weight DeepSeek through ORA [README §Overview].
  • LIBERO-Pro (Panda): 81.4% overall task success, same configuration [README §Overview].
  • Both scores claimed as “competitive with agents built on frontier models” (no explicit frontier-model baseline table in the README; deferred to the blog post at haoailab.com/blogs/ora/) [README §Overview].
  • Verified end-to-end on single RTX PRO 6000 Blackwell (32GB+) and through launch checks on H200; 8× H200 nodes for benchmark-scale official suites [README §Pick your machine].

Why it’s interesting

Section titled “Why it’s interesting”

ORA lands squarely on the recipe-lever cell staked out by Robot-Use Agents Isola’s “robot-use agents” positional thesis — the LLM-as-puppeteer paradigm with intelligence diffusing through the ecosystem via cloud model updates — and gives it the sharpest open-source implementation + benchmark numbers filed so far at the open-weights pole. It occupies a distinct position from the LLM-as-controller filings already tracked under Tool-Use Agents: OpenRUA: Robot-Use Agents Are Zero-Shot Visuomotor Policies OpenRUA takes a zero-abstraction position (nothing but ROS 2 native CLI as the tool surface, Claude Opus 5 as the model), while ORA keeps a curated primitive library (SAM 3, Contact-GraspNet, cuRobo) and bets on open-weights DeepSeek. Compared to Astranomy: An Empirical Study of GPT-6 Astra as a Native Robot Policy Astranomy’s finding that frontier Astra on RoboLab-120 costs ~$36/successful episode, open-weight DeepSeek via ORA is a direct test of whether the LLM-as-controller pattern can escape the two-order-of-magnitude cost premium — the headline numbers (69.2% / 81.4%) suggest it can, pending a shared-axis comparison on common tasks.

The terminal-UI and sealed-evidence design (ora watch, ora interactive, replay-in-terminal) treats robot sessions the way agentic SWE already treats coding sessions, which is the operational pattern Previewing the Model Hardware Standard MHS and Introducing Waddle: Agents that Control Robots Waddle both argue for from different angles. For VLA Models this is a concrete filing on the “no VLA at all” counter-recipe cell, direct sibling to Guava and OpenRUA, now with three benchmarks and a shippable harness rather than a paper.

See also

Section titled “See also”