Skip to content
Bud's Wiki

A Few Words Go a Long Way: Language Guided Robot Policy Synthesis

TL;DR

Section titled “TL;DR”

ARCHITECT (Agentic Robot Code with Human In ThE loop CorrecTions) treats robot policy acquisition as interactive program synthesis: an LLM coding agent (Claude) composes modular robot programs from a fixed suite of perception and control tools (open-vocabulary detectors, grasp generators, VQA, motion planning, proprioception), and a human supervisor issues free-form natural-language corrections when execution fails. Corrections are grounded to specific tool calls / lines via execution traces, then distilled into a persistent skill library that is loaded into context on every subsequent run — long-term in-context learning over a growing repertoire of reusable behaviors. On a Franka Panda benchmark, ARCHITECT reaches 0.70 success on cluttered-box retrieval (1.75× π₀.₅, 7× ProgPrompt) and 0.80 on cloth folding, and a real-world human study shows human effort amortizing across tasks (mean corrections per trial 4.67 → 0.83; 3 of 6 participants needed zero once a personal skill library was populated) [§Abstract, tweet §7-9].

Key claims

Section titled “Key claims”
  • Robot policy acquisition is framed as interactive program synthesis, not end-to-end policy fitting: an LLM coding agent composes modular robot programs by calling a fixed suite of perception and control tools (open-vocabulary detectors, grasp generators, VQA, motion planning, proprioception) [§Abstract].
  • The modular architecture localizes failure: because the program is code over typed tools, users can isolate which tool call or line caused a failure and target feedback at that abstraction level, rather than debugging an end-to-end policy’s cascading errors [§Abstract].
  • Corrections are grounded to policy code via program execution traces, so the agent can localize exactly which tool call or line to change, instead of guessing from a top-level failure signal [tweet §4].
  • Corrections are distilled into a persistent skill library — natural-language rules and code patterns loaded into context on every future execution — implementing long-term in-context learning across sessions [§Abstract, tweet §5].
  • On the cluttered-box retrieval task, ARCHITECT reaches 0.70 success — a 1.75× relative improvement over the best VLA baseline (π₀.₅) and 7× the best code-synthesis baseline (ProgPrompt) [tweet §7].
  • On cloth folding, ARCHITECT reaches 0.80 [tweet §7].
  • In a real-world human evaluation, seeding the agent with a participant’s own prior-session skill library drops mean corrections per trial from 4.67 to 0.83, and 3 of 6 participants needed zero corrections — evidence that human effort amortizes across tasks [tweet §8].
  • Skills transfer across tasks: with Task 1’s skills (“place the cup upright on a table”) loaded and no further corrections, Task 2 (“place the box upright on a shelf”) jumps from 0/6 to 4/6 [tweet §9].
  • VLM-generated corrections (ablation replacing the human in the loop with a VLM) underperform human corrections, because humans catch failure modes that pixels don’t easily show — e.g. depth error that drops the object from too high, or “close the gripper slowly so the baseball doesn’t slip” [tweet §10].

Method

Section titled “Method”

ARCHITECT is a coding-agent loop over a Franka Panda. An LLM orchestrator (Claude) receives a natural-language task instruction and synthesizes Python code that calls a fixed suite of typed tools — open-vocabulary object detection, grasp generation, visual question answering, motion planning, and proprioception. Program execution produces a trace (per-line invocations and their observations), and a human supervisor watches the rollout. When it fails, the supervisor types a free-form correction (“the grasp was too high, try picking it up from the rim”); the correction is grounded to the specific tool call in the trace that produced the offending behavior, and the agent revises the program.

Successful executions and their surrounding corrections are distilled into a persistent skill library — a growing set of natural-language rules, patterns, and named code helpers that are retrieved into context on every future execution. This turns the correction loop into a form of long-term in-context learning: the same failure never has to be corrected twice, and skills learned from one task can be retrieved and reused when solving another. The system releases code (github.com/robo-architect/architect-franka) and a project page (robo-architect.github.io).

Results

Section titled “Results”
  • Cluttered-box retrieval: 0.70 success, vs best VLA baseline π₀.₅ at ~0.40 (1.75× relative) and best code-synthesis baseline ProgPrompt at ~0.10 (7×) [tweet §7].
  • Cloth folding: 0.80 success on the harder task (baseline numbers not in the tweet abstract) [tweet §7].
  • Real-world human study (6 participants): mean corrections per trial drop 4.67 → 0.83 once the participant’s own skill library is populated; 3 of 6 participants needed zero corrections after the first session [tweet §8].
  • Zero-shot skill transfer: Task 1 → Task 2 (upright placement, different object and surface) goes from 0/6 → 4/6 with only the Task 1 skill library loaded and no further corrections [tweet §9].
  • VLM-in-the-loop ablation: replacing the human corrector with a VLM underperforms — the paper attributes the gap to failures pixels don’t easily show (depth misestimation, grip-force nuances) [tweet §10].

Why it’s interesting

Section titled “Why it’s interesting”

ARCHITECT is now the fifth filed instance of the “LLM coding agent as top-level policy, VLA/perception/motion as callable tools, persistent skill library as long-term memory” recipe on the wiki, and the sharpest one on the human-correction interface axis. It complements Playful Agentic Robot Learning (RATs) — which grows the skill library through self-directed pre-task play rather than deployment-time human corrections — and Introducing Waddle: Agents that Control Robots (Waddle), which grows the skill library from autonomous multi-agent deployment with no human in the loop; ARCHITECT stakes out the third corner: skills accumulated from grounded human corrections against execution traces, with quantitative evidence (4.67 → 0.83 corrections/trial, 0/6 → 4/6 zero-shot transfer) that the human effort amortizes.

It also delivers the sharpest head-to-head yet against end-to-end VLAs on the code-as-policy side of the VLA Models recipe board: 1.75× π₀.₅ on cluttered-box retrieval is on the same axis as Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents (Harness VLA’s +38.6 pp on LIBERO-Pro with π₀.₅ frozen inside an orchestrator) and GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks (GaP’s 0.93-0.99 vs 0.05-0.24 for end-to-end VLAs under pose variation) — all three arguing that at long-horizon complex tasks the LLM-agent-over-tools decomposition beats the end-to-end policy at matched hardware. The load-bearing new claim vs Harness VLA / VoLo is that human natural-language corrections grounded on execution traces are strictly stronger than the closed-loop VLM-corrector ablation, which sharpens the Malik-style sensorimotor counter-position from a different angle: some failure modes (“close the gripper slowly so the baseball doesn’t slip”) are not recoverable from pixels alone even for frontier VLMs. The counter-argument that end-to-end VLAs will eventually swallow all of this — via more data or RL — remains live; ARCHITECT is the strongest existence proof to date that if you accept the modular decomposition, the interactive-correction interface makes it usable by non-experts.

See also

Section titled “See also”