EmbodiedSWE: Coding Agents for Long Horizon Dexterous Robotics

TL;DR

EmbodiedSWE studies whether frontier coding agents (Claude Code, Codex) can serve as scalable supervisors for long-horizon, contact-rich, dexterous robot policies. The paper ships three tightly coupled artifacts: EmbodiedSWE-Bench, an Isaac-Lab-based benchmark of long-horizon dexterous tasks including deformable-object manipulation and multi-embodiment settings (Franka, xArm7, Kinova Gen3 + panda hand, Unitree G1, bimanual) evaluated inside isolated Docker containers under a time budget; EmbodiedSWE-Gen, a data engine that takes a single verified agent-authored solution and diversifies it into a large trajectory dataset along five independent axes (scene, strategy, phase, dynamics, visual); and an agent-improvement loop that generates new tasks from existing ones and RLs the coding agent against verified outcomes. The headline claim is that a VLA finetuned solely on coding-agent-generated simulation demonstrations completes a long-horizon task on a real robot, and that generalization improves with more agent-diversified demonstrations.

Key claims

Frontier coding agents can solve complex long-horizon dexterous tasks in simulation when given a Python solve(env) contract, the task’s grader, and a smoke.py launcher — the paper benchmarks task performance, completion time, and inference cost against the same three-axis rubric [§Evaluation, README].
Agent solutions transfer across tasks and embodiments: prior solutions for a task family (e.g. assembly) generalize to new embodiments (Franka → xArm → attached Kinova+panda) with agent-driven adaptation rather than from-scratch rewriting [§Abstract, README Repository layout].
Purpose-built tooling — a whole-body IK controller (pink_ik with pin/pin-pink/daqp), pre-registered task rooms, deformable physics on the Newton backend as a separate suite — makes the agent’s action space narrower and more reliable, mirroring the CaP-Gym pattern where high-level primitives improve weak-model performance [README Installation, controllers layout; contrast CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation §3.3].
Individual agent solutions are typically specialized to a single task instance — they do not natively generalize across scene/dynamics/visual variations, motivating the EmbodiedSWE-Gen diversification step [§Abstract].
EmbodiedSWE-Gen expands one verified solution into diverse trajectories along five independent axes (scene, strategy, phase, dynamics, visual); each generated episode is graded and verified before promotion to the training set, then rendered to video and baked into a LeRobot-format dataset [§Data engine, README; data_engine/scripts/{init_gen,diversify,generate,render}.py].
VLA performance improves monotonically with more agent-generated demonstrations, and agent-driven diversification specifically improves generalization to held-out task variations rather than only re-fitting the seed configuration [§Abstract].
A VLA finetuned solely on coding-agent-generated simulation demonstrations transfers to a real robot on a long-horizon task — the first filed sim-only-to-real result where the sim data was authored by a coding agent rather than teleoperated or motion-planner-procedurally generated [§Abstract].
The task library spans a wider dexterous / long-horizon regime than most filed manipulation benchmarks: assembly.bulb (screwing a light bulb into a socket), assembly.ikea_table, assembly.so101 (robot self-assembly), assembly.pc_motherboard, packing.tool_packing, packing.egg_carton, deformable.tshirt, deformable.latte (capsule coffee), cutting.slice (bananas), locomanip.fruit_delivery [README Quick start].
Agent improvement is architected as a two-stage loop: sim_gen/ generates new simulation tasks from seed tasks (task-space bootstrap), and rl/ applies RL to the coding agent against verified outcomes — closing the loop between task authoring and agent policy [README Repository layout].
Real-to-sim ingestion (Gaussian-splat backgrounds, photos → sim-ready assets) is a first-class stage of the pipeline, not an afterthought — the real_to_sim/ module keeps the sim substrate close to real deployment domains [README Repository layout].

Method

The stack has four tightly coupled layers.

1. robobench (benchmark). Built on Isaac Sim 5.1 + Isaac Lab 2.3.2. Tasks are named suite.scene[.robot[.control_mode]] and register a BaseEnv + BaseScene + BaseRobot triple with a per-task grader predicate. Six suites — assembly, packing, puzzle, deformable (Newton backend, separate venv), cutting, locomanip — share a common controller zoo (joint, diff_ik, task_space OSC/impedance, pink_ik whole-body IK, composite, loco_policy) so the same task can run under different embodiments. Assets and rooms live in the HuggingFace EmbodiedSWE/robobench-assets dataset with SHA-256 verification.

2. eval (agent harness). The environment is opened in a coding agent (Claude Code, Codex) pointed at robobench/README.md and instructed to solve a named task. A solution is a Python program exposing solve(env); the contract is eval/prompts/_contract.md. Large-scale evaluation runs each agent inside an isolated Docker container under a time budget, then grades the delivered solution in fresh containers on independently randomized episodes — decoupling agent execution from grading so contamination of the grader by the agent’s stdout is impossible. Prompt conditions (hints, rules, blocked features) are yaml-authored in eval/configs/ and eval/prompts/.

3. data_engine (EmbodiedSWE-Gen). Takes one verified solve run and expands it. init_gen.py starts a campaign, diversify.py runs an agent session that authors diversity along five independent axes (scene, strategy, phase, dynamics, visual), generate.py mass-produces batched episodes with per-episode grader verification, render.py renders to video, and vla/convert/convert.py bakes the verified rendered episodes into a LeRobot-format dataset for policy training. Policy training and closed-loop VLA evaluation live in vla/.

4. sim_gen + rl (agent improvement). sim_gen/ generates new simulation tasks from seed tasks — bootstrapping the task-space itself — and rl/ applies RL to the coding agent against verified outcomes from those tasks.

Results

Self-reported / paper-abstract-level (the arxiv landing page provides only abstract and repository details, with code but no PDF text pulled at ingest):

Frontier coding agents can solve complex long-horizon tasks and transfer prior solutions across tasks and embodiments [§Abstract].
Agent solutions require substantial iterative interaction and are typically specialized to individual task instances [§Abstract].
VLA performance improves with more EmbodiedSWE-Gen-generated demonstrations [§Abstract].
Agent-aided diversification improves generalization to held-out task variations [§Abstract].
A VLA finetuned solely on coding-agent-generated simulation demonstrations completes a long-horizon task on a real robot [§Abstract].
Task list (README): 10 named tasks in the quick-start gallery spanning assembly (bulb, ikea_table, so101 self-assembly, pc_motherboard), packing (tool_packing, egg_carton), deformable (tshirt, latte / capsule coffee), cutting (slice bananas), and locomanip (fruit_delivery).

Full quantitative results (agent success rates, VLA generalization curves, real-robot transfer numbers) require reading the PDF, which was not accessible at ingest.

Why it’s interesting

This is the second filed benchmark that treats coding agents themselves as first-class robotics research objects, alongside CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation (CaP-X). The two are complementary rather than redundant: CaP-X isolates the axes of primitive abstraction × temporal interaction × visual grounding to diagnose what makes coding agents succeed at manipulation, whereas EmbodiedSWE bets on coding agents as a supervision pipeline — one verified solve becomes many trajectories via EmbodiedSWE-Gen, and the downstream measurement is VLA generalization to held-out variations, not per-task Pass@1. The five-axis diversity taxonomy (scene, strategy, phase, dynamics, visual) formalizes what CaP-Agent0’s persistent skill library did informally, and it’s directly comparable to Introducing Auto-Engineering for Robotics‘s “simulation as an engineering variable” position — both let the agent choose the substrate rather than committing to a fixed sim.

The tighter loop between coding-agent problem-solving, per-episode grading, and VLA training also puts EmbodiedSWE closer to the “sim-only-to-real via automated procedural demonstrations” thread (RoboLab — Task-Based Evaluation Benchmark for Robot Manipulation Policies on Isaac Lab (NVLabs), RoboCasa365 — Large-Scale Simulation of Everyday Tasks for Generalist Robots) than to the abstract benchmark-only lineage (LIBERO / RoboArena) — the artifact isn’t just an eval leaderboard, it’s a data engine whose outputs a downstream VLA trains on. Whether the coding-agent-authored diversity generalizes better than teleop-authored or motion-planner-authored diversity is the load-bearing empirical question the paper implicitly opens.