Skip to content
Bud's Wiki

RobotUse: Allocating Computation, Context, and Decisions

TL;DR

Section titled “TL;DR”

RobotUse (SNU, Nojun Kwak’s group) is a robot agent harness that factors long-horizon manipulation into a Main agent that owns the task goal + persistent playbook, a Subagent that owns the within-subgoal detailed interaction trace, and a Backend that owns geometry / motion planning / control. Agents operate on the task by visually selecting targets and gripper poses rather than writing low-level motion code, with the Backend grounding those visual decisions into physical execution. A persistent playbook is updated across runs so the harness learns from execution — including under imperfect feedback — and transfers what it learns to subsequent tasks. On the RoboLab benchmark, RobotUse achieves 45% task success, beating CaP-X by 6.7 percentage points while maintaining compact decision contexts and reducing reliance on predefined action abstractions.

Key claims

Section titled “Key claims”
  • RobotUse decomposes a robot agent into three distinct responsibilities — a Main agent (overall goal + persistent playbook), a Subagent (per-subgoal detailed interaction + return summary), and a Backend (geometry, motion planning, control) — explicitly to allocate computation, context, and decisions across the stack rather than loading them all onto a single LLM loop [Abstract].
  • Agents interact with the task primarily via visual target and pose selection, with physical grounding (geometry, motion planning, control) deferred to the Backend rather than materialized in agent-authored motion code [Abstract].
  • Subagents retain the full detailed interaction trace within each subgoal and return only the information needed for subsequent decisions, keeping the Main agent’s decision context compact across long horizons [Abstract].
  • Continual harnessing — a persistent playbook updated across executions — lets agents learn from real-world execution even under imperfect feedback and transfers that learning to subsequent tasks [Abstract].
  • On RoboLab, RobotUse reaches 45% task success, outperforming CaP-X by 6.7 percentage points [Abstract].
  • RobotUse maintains compact decision contexts and reduces reliance on predefined action abstractions, relative to interfaces that bake high-level motion macros into the policy’s action space [Abstract].

Method

Section titled “Method”

RobotUse structures a robot agent as three roles with distinct allocation of computation and context. The Main agent carries the overall task goal, consults a persistent playbook that accumulates execution experience across runs, and dispatches subgoals. For each subgoal, a Subagent runs the detailed per-step interaction loop — observing the scene, visually selecting the next target and gripper pose, requesting Backend execution, inspecting the outcome, and iterating — then returns a compact summary of what happened and what the Main agent needs to know for the next decision. The Backend handles the physics-adjacent subproblems the agents are not asked to solve directly: given a visual target and gripper-pose selection, it does geometry, motion planning, and control, producing the executed trajectory. Across runs, the playbook is updated with lessons from execution (what worked, what failed, and under what conditions), providing a form of persistent memory that is read by the Main agent on subsequent attempts. The arxiv abstract is the only part of the paper surfaced through the fetch; the project page at https://robotuse-team.github.io and the Apache-2.0 code at https://github.com/robotuse-team/RobotUse contain the full method and ablation details.

Results

Section titled “Results”

Headline number from the abstract: 45% task success on RoboLab, +6.7 percentage points over CaP-X on the same benchmark. The abstract also claims qualitative improvements — compact decision contexts, reduced reliance on predefined action abstractions, and demonstrated learning transfer across tasks under imperfect real-world feedback — but does not provide per-task numbers, baseline tables beyond CaP-X, or ablations in the retrieved text.

Why it’s interesting

Section titled “Why it’s interesting”

RobotUse lands squarely in the “LLM/VLM-as-controller for manipulation” row the wiki has been tracking across Agent as Policy for Robotic Manipulation (AGP’s program-writing over a typed EE-pose interface), Show-Harness: Just a VLM Agent Can Play Robots (Show-Harness’s semantic-action-unit vocabulary with embodiment-specific interpreters), and CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation (CaP-X’s code-as-policy on Robosuite/LIBERO-PRO/BEHAVIOR with a persistent-skill-library + visual-differencing-module agent). What’s new is the explicit three-role factorization — Main / Subagent / Backend — as the design primitive, with context compaction via the Subagent return-summary interface and persistent-playbook-across-runs as the memory layer. This differentiates it from AGP (where “experience accumulation” is implicit in saved programs) and from CaP-X (where the auto-synthesized skill library is a code artifact, not a Main-agent-readable playbook), and lands it in the same Agentic Context Management lineage as Agentic Context Management: Solving Agent Memory and Cost by Treating Them as Lifecycle and Architecture Problems. The RoboLab comparison is the critical datapoint: the same benchmark Sid is using on RoboLab — Task-Based Evaluation Benchmark for Robot Manipulation Policies on Isaac Lab (NVLabs) for VLA checkpoint eval is now being used as the headline eval for an agent harness, with CaP-X as the baseline — direct quantitative comparison of harness-level recipes on an embodiment-agnostic Isaac Lab task set. The abstract’s “visual target and pose selection, Backend owns geometry” bet is the sharpest statement so far that the physical-grounding half should be deferred to classical robotics tooling rather than learned or coded by the agent — a position previously made implicitly by Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents (Harness VLA’s narrow-tool-API) and RoboClaw: An Agentic Framework for Scalable Long-Horizon Robotic Tasks (RoboClaw’s MCP-wrapped VLA skills).

See also

Section titled “See also”