Agent as Policy for Robotic Manipulation

TL;DR

Agent as Policy (AGP) instantiates the LLM-as-controller paradigm as a full paper: a general-purpose multimodal agent (GPT-6 Astra) is dropped in as the entire policy over an I2RT YAM bimanual arm — no task- or environment-specific training, no VLA, no fine-tuning. The agent observes RGB + depth + joint/gripper state on demand, writes and reruns Python programs against a free_agent runtime, submits target EE poses / joint angles / gripper openings, checks motion feedback, and decides when the task is done. Evaluated on eight configurations spanning precision manipulation (0.73 mm-clearance assembly from human video), block construction from goal images, die reorientation, targeted throwing, and bimanual towel folding, AGP hits ≥80% in 7/8 configurations (100% in five). Task experience is reusable: saved procedures and programs shorten repeat execution.

Key claims

A frontier multimodal agent can act as the sole policy for real-robot manipulation with zero task/environment training — same agent, same interface, across precision, dynamic, and deformable tasks; ≥80% success in 7 of 8 evaluated task configurations, 100% in 5 [§Abstract, §Results — Task Success and Efficiency].
Block construction 100% / 100% / 80% across three configurations, including a six-block tower where the agent uses the two-arm bridge motif for stability [§Abstract, §Results — Block Construction].
Assembly from human video succeeds at 0.73 mm clearance — the agent watches a human-video task specification and drives insertion via depth-verified EE-pose steps [§Results — Assembly; project page].
The runtime is a seven-command tabletop interface: frames, state, status, help, move_ee, move_joints, gripper, with a bridge that validates every request against workspace and motion limits before execution [Project page — Method].
Bimanual towel folding uses two arm servers on a common frame with coordination selected by the agent, not by a hardcoded plan; targeted throwing runs on a separate interface with timed motion programs (Appendix A.1) — evidence that AGP’s runtime is composable across control regimes [§Robot Platform; Appendix A.1].
Task complexity dominates execution overhead in a measurable way: potato-throwing takes ~13.9 minutes for the throw plus ~40% additional time on trajectory analysis + verification (22.8 minutes total) [§Results — Task complexity and execution overhead].
Deformable-object manipulation remains the failure mode: bimanual towel folding under a simultaneous-arms configuration gets 3/5 vs higher rates on the sequential configurations [§Results — Challenges in deformable object manipulation].
Task experience accumulation shortens repeat execution — saved procedures + reused programs cut wall-clock across repeated trials [§Abstract, §Results — Experience].
Success evaluation is intentionally not agent-self-report — final scene and recorded video are graded against criteria fixed before evaluation [Project page — Method].

Method

The paper studies AGP across ten manipulation tasks (assembly, block construction, dice flipping, targeted throwing, bimanual towel folding) on an I2RT YAM 6-DoF arm with parallel gripper (bimanual for towel folding) in a tabletop workspace. A wrist-mounted Intel RealSense D405 gives aligned RGB + depth; both cameras stream at 30 fps and the agent requests observations on demand. When a task type is first introduced, a preparation agent creates a reusable task definition (goal, references, constraints, completion criteria, interface rules, budgets, reporting requirements). A launch program then hands the saved definition plus current instance details to a fresh execution-agent session with the interface documentation and a writable workspace; tasks can be specified through video, image, or language.

The execution agent runs a four-step loop under its own control: observe (request overhead + wrist RGB + depth + joint angles + gripper pose, saved into the workspace), write/select/run programs (Python + libraries process observations, compute targets, and compose robot calls; saved programs can be reused or revised across trials), submit action targets (target EE pose, joint angles, or gripper opening via the interface; the controller executes accepted requests), and check the outcome (inspect motion feedback, request fresh observations, compare to task goal, decide next step). The tabletop interface exposes seven commands. A bridge validates all requests against workspace and motion limits before actuation; the targeted-throwing case uses a separate runtime of timed motion programs (Appendix A.1). Bimanual towel folding uses two arm servers on a shared coordinate frame with coordination chosen at runtime. The cycle ends when the agent reports the task complete or the fixed time/request budget runs out; success is graded from the final scene and recorded video against criteria fixed before the run.

Results

Block construction: 100% / 100% / 80% across three configurations; six-block tower uses the two-arm bridge for stability [§Abstract, §Results].
Assembly from human videos: succeeds at 0.73 mm clearance [§Results; project page].
≥80% in 7 of 8 evaluated task configurations, 100% in 5 [§Results].
Targeted potato throwing: ~13.9 min throw execution + ~40% additional trajectory analysis/verification for 22.8 min total [§Results — Task complexity and execution overhead].
Bimanual towel folding — simultaneous configuration: 3/5 (sharpest deformable-object failure case) [§Results — Challenges in deformable object manipulation].
Task-experience accumulation shortens repeated trials via reuse of saved procedures + programs [§Results — Experience; §Abstract].
Dataset release: 162 real-robot trials, 10 manipulation tasks, six models × three reasoning-effort levels, one row per trial replaying every step + command + robot answer + image [HuggingFace dataset card — Agent-as-Policy/agent-as-policy].

Why it’s interesting

AGP is the paper-scale version of the LLM-as-controller thread already very active on the wiki, and it upgrades that thread from single-case demos to a controlled multi-task evaluation with quantitative success rates, tool-call budgets, and time accounting. Direct sibling to Show-Harness: Just a VLM Agent Can Play Robots (Show-Harness) on the VLA Models recipe-lever board: both stake out “an LLM/VLM is the policy, the interface does the work” — but Show-Harness’s key move is the semantic action-unit vocabulary (bare symbols carrying model-known conventions, embodiment-specific interpreters), while AGP’s key move is program-writing over a typed tool interface (Python + move_ee/move_joints/gripper + validated bridge). Sits alongside Wenli Xiao — GPT-6 Astra does physical in-context learning from a single human video via a motion-planning tool harness‘s Astra + motion-planning tool harness (human-video ICL, EE-pose emit, IK tool) as the paper-scale version of the same substrate — same author-supplied “LLM emits EE targets, harness runs safe execution” pattern, now with 162 released trials.

Complements Robot-Use Agents‘s “robot-use agents” positional post with concrete numbers: 100% on block-construction, 3/5 on simultaneous-arm towel folding, 22.8 min total for potato throwing. The dexterous-contact failure mode (deformable-object simultaneous coordination) matches the specific prediction of Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level‘s sensorimotor counter-position and echoes Comparing Fable 5.1 to Fable 5 on robotic manipulation‘s Fable-5-stalls-at-first-contact result. Because AGP treats task-specific programs as first-class saved artifacts, its “experience accumulation” story is a physical-world instance of the Tool-Use Agents page’s experience-library-reduces-required-interaction-depth finding (Training-Free GRPO’s context-space update, Anthropic MHS’s “explore-then-compile-to-deterministic-script”). Does not report token/cost budgets in the released body — the closest quantitative accounting is the 22.8 min figure and the HuggingFace dataset’s per-trial replay, both of which enable downstream re-analysis in a way the qualitative Astra/Fable threads did not.