Lingxiao Guo — GPT-6 does end-to-end real2sim from multi-view RGB + robot actions

TL;DR

Lingxiao Guo (CMU Robotics MS) reports feeding GPT-6 only multi-view RGB + robot action logs plus a single prompt, and getting an end-to-end real2sim pipeline back: camera calibration, object asset construction, physics system identification, MuJoCo rollout, and Blender rendering — no explicit optimizer, just the LLM agent hypothesizing physical parameters, editing the model, running dynamics, comparing against the video, and revising. The demo succeeds on rigid single-object tasks but fails on a bimanual microphone snap-fit task where rigid proxies can’t express compliant contact, so the “successful” run there is kinematic replay rather than validated dynamics. A striking side observation: GPT-6 produces multi-view-consistent geometry from pure RGB on ABC and DROID scenes where VGGT-Omega and MapAnything fail, apparently because it fits one shared scene rather than per-view geometry.

Key claims

GPT-6, given only multi-view RGB + robot actions + a simple prompt, executes the full real2sim loop (camera calibration → asset construction → physics system-ID → MuJoCo → Blender render) end-to-end [Tweet 1].
The system-ID inner loop is agentic: hypothesize a physical parameter, edit the model, run the dynamics, compare to video, revise — no optimizer is invoked [Tweet 2].
The bimanual microphone snap-fit task failed physics: no run retained all three mics; the shown video is kinematic replay, not validated dynamics, because rigid proxies can’t express compliant contact [Tweet 3].
GPT-6 produces object geometry that stays consistent across views on ABC and DROID, where VGGT-Omega and MapAnything do not, by fitting one shared scene and forcing every camera to agree rather than predicting per-view geometry [Tweet 4].
The real ceiling is the simulator, not GPT-6: physics can never exceed what the invoked simulator expresses [Tweet 5].

Method

Demonstration, not a paper. Guo prompts GPT-6 with multi-view RGB captures + robot action logs from teleoperated demos and asks for a runnable simulation. The model calls out to MuJoCo for dynamics and Blender for rendering; it treats simulator invocation as tool use, and system identification is done by iterated hypothesis-and-check against the source video rather than gradient descent. Failure mode analysis: rigid-body simulators can’t represent compliant snap-fit contact, so the agent gets kinematically-plausible motion but no dynamics validation on the bimanual mic task.

Results

Qualitative only, one demo per task. Rigid single-object tasks yield full real2sim twins (calibrated cameras, reconstructed assets, physics-consistent MuJoCo rollouts, Blender renders). Bimanual snap-fit fails physics validation — the render is kinematic replay. Multi-view geometric consistency on ABC / DROID scenes is reported qualitatively better than VGGT-Omega and MapAnything [Tweet 4]. No sim-to-real policy transfer results.

Why it’s interesting

This is a live datapoint on the “orchestration is the bottleneck” thesis that Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents made explicitly: a VLM agent, given the right toolchain (VFMs + simulator + renderer), replaces hand-composed real2sim pipelines. Agentic Real2Sim needed a bespoke framework and per-domain glue for rigid / deformable / humanoid; Guo’s tweet is the frontier-model version — GPT-6 does it zero-shot without framework code. The three most interesting details for the team are (1) the “one shared scene” trick that reportedly beats per-view feed-forward reconstructors like VGGT-Omega on multi-view consistency, echoing the SAGE-style critic-gated loop but without an explicit critic; (2) the confirmation that rigid-simulator ceilings still bind — the same limit that SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation and Introducing SimReadyGen — Agentic Simulation Generation for Physical AI work around with per-asset accuracy pipelines; and (3) that iterated hypothesize→simulate→compare→revise is an emergent capability of the base model rather than a scaffolding layer, which puts this alongside AIDE²: First Evidence of Recursive Self-Improvement as an early “the agent is the search” datapoint at frontier scale.