Skip to content
Bud's Wiki

VIA: Visual Interface Agent for Robot Control

TL;DR

Section titled “TL;DR”

VIA recasts robot manipulation as a computer-use task: an off-the-shelf frontier foundation-model agent (Claude Code, Codex, Fable 5) drives a real manipulator through a browser-based 3D interface, taking screenshots, issuing intuitive commands, observing outcomes, and re-planning — with no robot-specific fine-tuning and no privileged state. The framework inherits general reasoning, closed-loop error recovery, and re-planning from the agent’s coding/computer-use skills. With Fable 5 it reports 96.7% success on three LIBERO-Goal tasks and 100% on a long-horizon rainbow-assembly task, with performance improving as the underlying model scales — evidence that frontier agents already possess robot-control competence given the right interface.

Key claims

Section titled “Key claims”
  • Reframing robot control as an agentic browser task lets an unmodified foundation model achieve strong zero-shot manipulation, without fine-tuning on robot data or access to privileged state [Abstract].
  • The agent interacts only through screenshots and a small set of general tools exposed by a browser-based 3D interface; perception is visual and actions are intuitive interface commands rather than low-level joint targets [Abstract].
  • With the strongest tested model (Fable 5), VIA reaches 96.7% success on three LIBERO-Goal tasks and 100% success on a long-horizon rainbow assembly task [Abstract].
  • Performance scales with the strength of the underlying foundation model: both Claude Code and Codex solve a diverse tabletop suite zero-shot, and improvements track model capability [Abstract].
  • Frontier agents inherit closed-loop error recovery and re-planning from their general software-agent training without any robot-specific adaptation [Abstract].

Method

Section titled “Method”

VIA replaces the standard VLA pipeline (VLM backbone + action head fine-tuned on robot data) with an interface between an off-the-shelf agent and a manipulator. The manipulator is exposed to the agent through a browser-based 3D interface identical in spirit to a computer-use environment: the agent takes screenshots to perceive the scene, issues intuitive commands through a small general tool set, observes the resulting frame, and adjusts. No fine-tuning is done on the foundation model, and no privileged state (object poses, contact information, joint positions) is exposed — only what a human operator would see through the same interface. The paper evaluates the framework with three off-the-shelf agents (Claude Code, Codex, Fable 5) on a diverse tabletop suite that includes LIBERO-Goal subtasks and a long-horizon rainbow assembly.

Results

Section titled “Results”
  • Fable 5 reaches 96.7% success on three LIBERO-Goal tasks and 100% success on a long-horizon rainbow assembly task [Abstract].
  • Both Claude Code and Codex solve the diverse tabletop suite zero-shot [Abstract].
  • Task success improves as the underlying foundation model becomes stronger, with Fable 5 the current best [Abstract].

Why it’s interesting

Section titled “Why it’s interesting”

VIA is the strongest datapoint yet for a lineage of counter-recipes on VLA Models that put the frontier LLM in charge and demote the robot-specific component. VoLo: A Physical Orchestrator for Open-Vocabulary Long-Horizon Manipulation (VoLo) already argued a VLM orchestrator over interruptible VLA tools beats end-to-end VLAs on long-horizon open-vocab tasks; Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering (LFP) formalized the VLA as a callable tool driven by learned language commands; How Claude Performs on Robotics Tasks showed frontier LLMs can supervise a VLA better than the VLA alone on novel LIBERO tasks. VIA takes the next step: there is no VLA at all — the browser interface exposes the manipulator directly as a computer-use surface, and a general coding agent controls it. This is a natural sibling to Cambrian-S: Towards Spatial Supersensing in Video‘s “solving spatial supersensing without spatial supersensing” pattern — capabilities that motivated a specialized training recipe turn out to be inherited from generalist training given the right interface. It sharpens the concept’s central open question (“how much of the VLA-vs-frontier-agent gap is backbone vs recipe vs data”) toward “how much of it is interface design”, and directly speaks to Tool-Use Agents‘s visual tool-use axis (VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents) — the browser interface plus screenshot loop is that axis extended to real robot control.

See also

Section titled “See also”