Robot-Use Agents

TL;DR

Short positional essay by Phillip Isola arguing that cloud-hosted LLMs controlling robots as tools (“robot-use agents”, by analogy to computer-use agents) represents a paradigm-shift in the deployment trajectory of robotics rather than just another capability data point. The claim is not that any one Claude-controlled robot becomes more performant than a dedicated on-device policy — LLM-controlled robots remain “far less performant than dedicated solutions” — but that intelligence diffuses across the whole robot ecosystem far faster in this regime because the model runs in the cloud, is not customized per embodiment, and inherits the mature LLM tech stack. Every internet-connected robot becomes a potential tool for a frontier LLM, and updates ship as cloud model releases rather than per-device engineering efforts. The essay names limitations — latency between commands, reliability gaps vs. tried-and-true systems for high-stakes/safety-critical use — but bets these will be surmountable.

Key claims

The traditional robotics deployment paradigm runs intelligence on-device with per-embodiment customization, which makes deploying AI in robotics a slow, deliberate, engineering-heavy process [§Body]. Robot-use agents invert this: cloud model, not customized per robot, mature infrastructure, robots need no modification (no new sensors, no new onboard GPUs) to become “AI-enabled” via what amounts to a software update [§Body].
The new capability is not per-robot performance improvement — LLM-controlled robots remain less performant than dedicated solutions — but the rate at which intelligence diffuses through the ecosystem: “intelligence diffuses through the whole robotic ecosystem much more quickly than in the traditional regime” [§Body].
The historical view that LLMs are useful only for high-level planning (too high latency for real-world dynamics, poor spatial intelligence, weak causal/physical understanding) is “starting to melt away” with newer models like Fable and Astra, per recent demos [§Body, footnotes 5–10].
Remaining limitations are latency between commands (model-thinking-latency plus internet round trips) and reliability gaps vs mature dedicated systems, particularly for high-stakes and safety-critical deployments [§Body]. The essay predicts these are surmountable but does not defend that claim quantitatively.
The change generalizes beyond robots: “any physical robot, and indeed any device connected to the internet” can become an agentic-AI-accessible tool once the interface is standardized [§Body].

Method

Position essay, not an empirical paper. The argument is a comparative-scenarios analysis: contrast the traditional on-device customized-policy trajectory with the cloud-LLM-as-puppeteer trajectory, argue the second changes the rate of intelligence diffusion across the ecosystem rather than the per-device capability ceiling. Isola credits Nathan Cloos, Adam Rashid, and Antonio Norelli for discussions.

Results

No experiments. The empirical grounding is a set of demo pointers in the footnotes (Fable, Astra, Claude driving robots) and prior work on the “LLM can control robots” question spanning several years, cited as evidence that the debate is not new but the balance of evidence is shifting.

Why it’s interesting

Isola’s essay is the sharpest external framing yet for a thread already highly active on the wiki: the “LLM-as-controller” or “robot-use” recipe cell has been growing on both VLA Models and Tool-Use Agents under a fast-moving stream of empirical evidence — Jay Chooi (Robocurve) — Astra vs Fable 5.1 on real-robot manipulation via IK (Astra vs Fable 5.1 on YAM arms via IK), Comparing Fable 5.1 to Fable 5 on robotic manipulation (Fable 5.1 vs Fable 5 head-to-head on real hardware), thijs — Astra paints the Golden Gate Bridge with a real robot arm (Astra painting), Sentdex — general multimodal LLMs may sidestep VLA/WAM research for high-level robotics intelligence (open-weights GLM 5.3 Flash on an XGO quadruped), Inspect Robots — An Open-Source Evaluation Framework for Physical AI (Robocurve) (the eval harness making these comparisons apples-to-apples), and Previewing the Model Hardware Standard (MHS, the driver-level interface that makes the pattern deployable). Isola’s contribution is to name the paradigm (“robot-use agents”) and to make the load-bearing case that the important axis is diffusion rate, not per-robot ceiling — a claim that reframes the ongoing filed debate against the Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level sensorimotor counter-position (which measures the ceiling and finds LLM controllers wanting) and against the on-device VLA line (which optimizes the ceiling embodiment-by-embodiment). Sits in productive tension with Going Beyond World Models & VLAs Florence’s opposite framing — that VLM/video pretraining is a “crutch” and the right recipe is from-scratch embodied training — since Isola’s argument only holds if the cloud LLM is the actual substrate of intelligence rather than a temporary bridge to it.