Introducing Waddle: Agents that Control Robots

TL;DR

Waddle Labs (YC) argues that end-to-end VLA / world-action models suffer from data hunger, poor steerability, and weak cross-embodiment transfer, and that the fix is to make an LLM agent the top-level policy: the agent decomposes a goal into subtasks and completes each by writing control code, viewing camera feeds, and calling VLAs as tools. Agents grow a shared library of parametrized skills (e.g. fold_grasp) that transfer across tasks and across agents. The system claims three deployed use cases — spinning up a working policy in ~20 minutes from a natural-language goal, autonomously collecting 1000× LEGO pick-and-place data overnight (then training an ACT policy from it), and running an overnight policy-tuning “autoresearch” loop that resets the scene between trials. No quantitative benchmarks are reported; the evidence is real-hardware demos and an internal comparison of Opus 4.8, Fable 5, and GPT-5.6 Sol where larger reasoning budgets solve harder tasks (t-shirt folding).

Key claims

The end-to-end VLA / WAM recipe has three deployment-limiting failure modes: reliance on huge amounts of robot data, difficulty steering the model, and weak generalization across embodiments and environments [§Problems with the current approach].
LLM agents can transfer their tool-use, reasoning, and program-writing capabilities into robotics by acting as the top-level policy that decomposes goals, writes control code, views camera feeds, and calls VLAs as one of several tools [§Our hypothesis].
Agents produce three-level hierarchical programs — platform primitives → parametrized skills → task programs — where the skill layer (e.g. fold_grasp) is the unit of transfer across tasks and across agents [§A library of skills, Fig. 2].
A skill first created by one agent (fold-grasp for flipping a package) generalizes to a different task by another agent (folding a t-shirt) without retraining [§A library of skills, Fig. 3].
Multi-robot control requires no architectural changes: a master agent spawns subagents that coordinate concurrent robots, with the same structure scaling from a bimanual pair to a fleet [§Multi-agent coordination].
Bigger reasoning models with bigger inference budgets produce better robot policies: Opus 4.8, Fable 5, and GPT-5.6 Sol all solve easy pick tasks, but only Fable 5 and GPT-5.6 (xhigh thinking) solve t-shirt folding [§Scaling with foundation models].
Auto-research and dataset-collection loops are natural downstream applications: the agent autonomously repeated LEGO pick-and-place ~1000× overnight, then trained an ACT policy on its own collected data [§Use cases].

Method

Waddle is a deployed system, not a single-shot code-as-policy generator. Robots connect to a Waddle API; an LLM agent runs continuously against each robot, decomposing user instructions into subtasks and emitting an executable program. Programs are composed of parametrized skills (grasping, aligning, folding), and skills are in turn composed of a fixed set of platform-provided primitives. When a new task requires a skill that doesn’t exist, the agent authors it; when the skill later fits another task, another agent reuses or adapts it. Every solved task adds to a persistent skill library shared across all agents on the platform. The agent inherits everything from the underlying foundation model — reasoning, tool use, program synthesis, multi-agent coordination — so no policy retraining is needed to add a robot, change a workspace, or attempt a new task. The failure loop is closed autonomously against execution feedback (in the CaP-X sense) rather than via human corrections.

Results

Policy-in-20-minutes: “write a program to place one microswitch inside each slot” produces a working robot program in ~20 minutes, no data collection or fine-tuning [§Use cases].
Autonomous data generation + policy training: instructed to “pick and place lego bricks at random positions 1000 times,” the agent ran ~1000 pick-and-place cycles overnight and then trained an ACT policy from scratch on its own data that successfully picks up LEGOs [§Use cases].
Autoresearch loop: agent handles scene reset between overnight tuning trials for a separate policy — one of the concrete auto-research use cases the post pitches [§Use cases].
Foundation-model scaling: across a manipulation suite, all three of Opus 4.8, Fable 5, and GPT-5.6 Sol solve easy pick tasks; only Fable 5 and GPT-5.6 with xhigh thinking solve t-shirt folding [§Scaling with foundation models].
No quantitative benchmark numbers reported. The post explicitly calls out benchmark standardization as a next step — success criteria and tasks are internal to Waddle’s demo suite [§Next steps].

Why it’s interesting

Waddle is the most product-shaped version of the code-as-policy revival that a chunk of the wiki has been tracking: RoboClaw: An Agentic Framework for Scalable Long-Horizon Robotic Tasks (RoboClaw’s agentic long-horizon framework), Playful Agentic Robot Learning (RATs’s pre-task play stage growing a code skill library), CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation (CaP-X’s multi-turn coding-agent benchmark), ENPIRE: Agentic Robot Policy Self-Improvement in the Real World (ENPIRE’s frontier-agent-improves-robot-policy loop), and ASPIRE — self-evolving sensorimotor skill library that compounds across robots and the sim-to-real gap (Jim Fan / NVIDIA GEAR) (ASPIRE’s evolutionary skill library) all bet that the LLM agent is the right top-level policy and the VLA is a callable primitive underneath. Waddle collapses those research prototypes into a deployed API + shared skill library and claims foundation-model-driven scaling, which is exactly the point GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks (GaP) makes for Variational Automation tasks. It also complements VIA: Visual Interface Agent for Robot Control (VIA) — which Waddle cites — on the “what interface does the agent get to control the robot” question that the Next Steps section flags as its next research target. The load-bearing weakness Ben flagged holds: with no quantitative eval against baselines, it is hard to distinguish “shipped product that works” from “cherry-picked demo suite” — Waddle itself calls for a shared agent-controlled-robot benchmark as a next step, echoing the RoboArena-style push in RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies.