DexAgent: An Agentic Human2Sim2Robot Framework for Dexterous Manipulation with Self-Evolving Tool Library

TL;DR

DexAgent (Wang, Li, Fei-Fei, Wu, Huang — Stanford) is an agentic four-stage Human2Sim2Robot pipeline for dexterous manipulation that replaces the fixed procedures of prior human-video-to-robot data pipelines with an LLM agent that picks skills from a self-evolving tool library, writes new ones when the library falls short, and gates each stage on property-specific verifiers before passing outputs downstream. The pipeline converts one egocentric human video plus a task prompt through (1) semantic understanding, (2) property-based simulation reconstruction, (3) robot trajectory optimization, and (4) robot data generation — with each stage’s tools chosen per object properties (rigid, articulated, deformable) and each stage’s output validated for physical validity before propagation. Newly authored tools + verifiers persist in the library so per-video processing cost drops as the library grows. Reports 3.5× higher success rate than competing baselines across eleven real-world tasks when policies are trained on DexAgent-generated data [§Abstract].

Key claims

Existing Human2Sim2Robot pipelines use predefined procedures that struggle with diverse object properties and interactions — particularly articulated and deformable objects — because a single fixed sequence of steps cannot accommodate the required property-specific treatments [§Abstract].
DexAgent operates through four stages: semantic understanding of human videos, property-based simulation reconstruction, robot trajectory optimization, and robot data generation [§Abstract].
At each stage the agent adapts by selecting suitable skills from a persistent tool library or developing new ones when needed, rather than executing a static pipeline [§Abstract].
Property-specific verifiers assess stage outcomes for physical validity and task-specific requirements and provide feedback for refinement, preventing error propagation through the workflow [§Abstract].
In the final stage, DexAgent varies object and robot states in simulation to generate diverse robot trajectories from a single human video, then retextures the rendered observations to facilitate sim-to-real transfer [§Abstract].
Newly developed skills and verifiers are retained in the library, making the system self-evolving — per-video processing time decreases as more human videos are processed [§Abstract].
Policies trained on DexAgent-generated data achieve 3.5× higher success rate than competing baselines across eleven real-world tasks [§Abstract].

Method

DexAgent instantiates an LLM agent as the top-level controller of the standard Human2Sim2Robot data pipeline. The pipeline decomposes into four sequential stages: (1) semantic understanding parses the input egocentric human video and task prompt to determine what objects are present and what properties they have (rigid / articulated / deformable) plus the intent of the demonstrated interaction; (2) property-based simulation reconstruction builds a physics-consistent simulator scene keyed on the recovered object properties, so an articulated object gets an articulated URDF and a deformable object gets an appropriate deformation model; (3) robot trajectory optimization produces a robot-executable trajectory that reproduces the human interaction inside the reconstructed sim; (4) robot data generation varies object and robot states in the sim to expand the single-video demonstration into a diverse training set, and retextures rendered observations for sim-to-real transfer.

The load-bearing agentic move is per-stage tool selection: the agent draws from a library of skills (video understanding, sim reconstruction, trajectory optimization, rendering primitives), picks tools that match the current object properties and task, and — when the library lacks a suitable tool — writes a new one and adds it back to the library. Each stage’s output is checked by a property-specific verifier (physical validity + task-specific requirements). Verifier feedback drives refinement inside the stage rather than propagating errors forward. Newly authored skills + verifiers persist across runs, so subsequent videos benefit from what earlier videos taught the library, reducing per-video processing time. The abstract does not name the underlying LLM or the concrete skill inventory; full method + tool-list is in the paper body, which was not retrieved at filing time — a refresh once the body is parsed would sharpen the picture.

Results

3.5× higher success rate than competing baselines across eleven real-world tasks, with policies trained on DexAgent-generated data [§Abstract].
The abstract does not name the baselines, task set, target robot embodiment, or per-task success numbers. No compute/time budget for the self-evolving library is disclosed in the abstract. Filing this on abstract-only; a refresh once the paper body is parsed would give per-task, per-property breakdowns and the concrete tool inventory the library ends up carrying.

Why it’s interesting

DexAgent sits at the intersection of three otherwise-separate wiki threads. On the Human-to-Robot Retargeting axis, most filed pipelines (DO AS I DO Do as I Do: Dexterous Manipulation Data from Everyday Human Videos, Ego2Robot Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data, HuRo HuRo: Robotizing Human Videos for Scalable VLA Pretraining, C2Dex C2Dex: Contact-Consistent Reconstruction and Retargeting for Dexterous Manipulation from Monocular Video) are fixed sequences of perception + retargeting steps — DexAgent’s per-property tool selection is a structurally different position on the same axis, arguing the pipeline itself needs to be authored per video rather than reused. On the Tool-Use Agents axis, DexAgent’s persistent self-evolving skill library is the same pattern as ASPIRE (evolutionary code search over sensorimotor primitives, ASPIRE — self-evolving sensorimotor skill library that compounds across robots and the sim-to-real gap (Jim Fan / NVIDIA GEAR)), RATs (self-directed play growing a persistent skill library, Playful Agentic Robot Learning), and GRID’s Auto-Engineering (Introducing Auto-Engineering for Robotics — Video-to-Sim mode where the agent picks skill-creation sub-modes per task) — but scoped tightly to the Human2Sim2Robot data-generation subproblem rather than the whole robotics lifecycle. Closest sibling is GRID’s Video-to-Sim: same “single phone video → hand-pose + 3D segmentation → sim reconstruction → synthetic BC demos” structure, but DexAgent’s property-specific verifier gates + persistent tool library are the load-bearing formalization GRID’s blog gestures at without spelling out. Complements Do as I Do: Dexterous Manipulation Data from Everyday Human Videos (Malik group’s kinematic-only pipeline for the same H2R2 pattern) by making the pipeline agent-authored rather than fixed. The result — 3.5× on eleven real tasks — needs a body-level read to compare against Ego2Robot’s disentangled-perturbation numbers (Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data Table 1) or C2Dex’s DexYCB/TACO end-to-end margins (C2Dex: Contact-Consistent Reconstruction and Retargeting for Dexterous Manipulation from Monocular Video).