Skip to content
Bud's Wiki

RoboSimStudio: Smartphone Video to Multi-Physics Simulation via Coding Agents

TL;DR

Section titled “TL;DR”

RoboSimStudio (UC Berkeley + NVIDIA) is a Real2Sim2Real pipeline that converts a smartphone video of a robot performing a task into a runnable multi-physics simulation, authored by a Claude or Codex coding agent from a one-line prompt in about an hour. The output couples a Newton rigid-body simulation with a SuperDex FEM solver for deformables (demonstrated on bar-wiping with a sponge), is inspected and tuned interactively by humans and agents until the rollout matches the source video, and then varied across physical parameters and layouts to improve a GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks Graph-as-Policy via feedback. The improved policy transfers back to the real robot. The project page is a short teaser rather than a technical paper; no quantitative sim-to-real numbers are disclosed yet.

Key claims

Section titled “Key claims”
  • A smartphone-captured task video plus a one-line prompt is sufficient input for a coding-agent-authored multi-physics simulation that reproduces the scene, camera, and physics in roughly one hour [§Overview].
  • The simulation substrate is multi-physics by construction — Newton rigid bodies coupled with SuperDex FEM for deformables (illustrated on a sponge wiping a bar) — rather than a single-engine simulator [§Overview].
  • Humans and agents can inspect and tune the generated simulation interactively until a rollout matches the source video; the project page ships a browseable rollout with WASD + drag controls [§Interactive rollout].
  • The generated simulation is varied across physical parameters and layouts to provide feedback that improves a Graph-as-Policy (GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks) [§Sim2Real].
  • The improved policy runs on the real robot, closing a full Real2Sim2Real loop whose middle stage is coding-agent-authored simulation rather than human-engineered [§Sim2Real].

Method

Section titled “Method”

The page describes the system at the pipeline level only. A short smartphone video of a robot executing a task is handed to a frontier coding agent (Claude or Codex) with a natural-language prompt. The agent authors a multi-physics simulation that reconstructs the scene geometry, the camera pose, and the physics of the interaction — combining Newton for rigid bodies with SuperDex FEM for deformable contact — and ships a WebGL rollout the user can walk through. Humans and the agent iterate on the simulation until its rollouts visually match the source video. The simulation is then varied along physical parameters and layouts to generate a training distribution; feedback from these variations updates a Graph-as-Policy, and the resulting policy is deployed on the real robot. No specifics are given about perception modules, the agent prompt structure, the simulator-tuning loop, or the policy-improvement algorithm.

Results

Section titled “Results”

The project page shows a single qualitative demo (bar-wiping with a sponge) and an interactive WebGL rollout. No quantitative sim-to-real success rates, policy evaluation numbers, generation time distributions, or baselines are disclosed. The one-hour Real2Sim conversion time is the only numeric claim.

Why it’s interesting

Section titled “Why it’s interesting”

RoboSimStudio sits precisely in the junction of two concepts that have been accumulating evidence on the wiki all quarter. On the Real2Sim side, it is the sharpest instance so far of the “coding agent authors a per-episode multi-physics simulator from a short video” pattern that Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents staked out as an orchestration-level real2sim bet — same input modality (video), same substrate (coupled rigid + deformable physics), but RoboSimStudio makes the coding agent (Claude Code / Codex) the explicit protagonist rather than a VLM tool-orchestrator. On the simulator-in-the-loop side, it is the second filed project after SIMPACT: Simulation-Enabled Action Planning using Vision-Language Models where a coding agent authors the per-scene multi-physics simulator (SIMPACT uses a VLM, SIMPACT’s physics choices are also hand-selected per object), but RoboSimStudio inverts the control direction: SIMPACT uses the agent-authored simulator at inference time to refine actions online; RoboSimStudio uses it at training time to produce varied rollouts that update a graph policy offline. The direct connection to GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks (same Berkeley/NVIDIA lab, same first-author ecosystem — Goldberg / Fan / Zhu / Sastry) is load-bearing: GaP already runs its self-learning rehearsal inside a parameterized simulator; RoboSimStudio is the natural upstream — “and the simulator is itself authored by the agent from a phone video.” This complements EmbodiedSWE: Coding Agents for Long Horizon Dexterous Robotics‘s coding-agent-authored data engine (Isaac Lab + diversification along five axes) by moving the authored artifact from the solve program to the physics world itself.

See also

Section titled “See also”