AHa-3D: Agentic Tool Use for Real2Sim with GPT-6 Astra

TL;DR

AHa-3D (Xu, Bian, Gao; U. Michigan + collaborators) is a harness that gives GPT-6 Astra a library of 3D vision tools — camera / geometry estimation (Pi3X), segmentation (SAM3), human motion recovery (GVHMR / Kimodo), measurement, and physics checks — and lets the agent orchestrate them to turn an ordinary indoor room video into an editable Blender scene with reusable furniture + materials, estimated per-person motion, foot-ground and object contact refinement, and a rendering that preserves the original camera and timing. It also supports generating new actions for people in the reconstructed scene and exporting the result as interactive browser demos. The pitch is explicitly that LLMs can already drive Blender from a room video, so the right question is “what if we hand the vision models to the agent as tools,” making AHa-3D the video-input instance of the agent-authored-Real2Sim pattern that complements the robot-manipulation-input RoboSimStudio: Smartphone Video to Multi-Physics Simulation via Coding Agents and Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents filings. A longer treatment appears as the LiteReality-Agent paper (NeurIPS 2026).

Key claims

A frontier coding/agentic model (GPT-6 Astra) can be made to orchestrate 3D vision tools — Pi3X for geometry, SAM3 for segmentation, GVHMR/Kimodo for human motion — to turn a single indoor room video into an editable Blender scene with reusable furniture and materials [§Project page overview].
The pipeline estimates the motion of the people in the video and places them in the shared reconstructed room with foot-ground and object contact refinement, rather than treating people and scene geometry as independent reconstruction problems [§Overview].
The reconstructed scene is rendered back with the original camera and timing, giving a direct video-to-video evaluation signal rather than only a free-viewpoint rollout [§Overview].
The agent can also generate new actions for people in the scene — the reconstruction is used as a reusable asset for downstream animation / synthesis, not only as a static digital twin [§Overview].
The runtime targets a single Linux workstation (Ubuntu / WSL2) with one NVIDIA GPU, Python 3.11, Blender 4.5 for scene assembly, and resumable per-run directories under runs/ — i.e., the harness is designed to be deployable on standard research hardware rather than requiring a cluster [GitHub README, Quick start / Tools].

Method

The project is a harness around GPT-6 Astra, not a new model. The agent is given a library of specialized 3D vision tools exposed as callable commands (Pi3X for camera / geometry estimation, SAM3 for segmentation, GVHMR and Kimodo for human motion, plus measurement and physics-check utilities) and is driven from a top-level tools/indoor CLI with discrete stages: intake (turn a scene request into a plan and list missing decisions), preflight (check recipe + inputs + optional segmentation bounds), run / batch (execute one recipe in the foreground or queue several), status (progress), preview / review-layout / accept-preview (render a preview and record reviews), check-placement (verify saved-room geometry and object stability), and acceptance / results (track completion and register deliveries). Runs are resumable, each in its own runs/ directory. Assets are organized through a shared library browseable via python tools/asset_index.py tree. Human skinning uses Blender 4.5’s body extension. The output is a Blender scene file plus a video rendered with the original camera and timing; interactive browser demos are an export option. No model-side training, loss, or benchmark numbers are described on the public page.

Results

The public artifacts are the project page, the GitHub repo (≥227 stars at the time of filing), and the LiteReality-Agent NeurIPS 2026 paper that benchmarks a closely-related system. No quantitative sim-to-real success rates, reconstruction-error metrics, or generation-time distributions are disclosed on the AHa-3D blog itself. The LiteReality-Agent abstract (ResearchGate 415154425) reports that the agentic reconstructions are “more geometrically accurate, visually realistic, and simulation-compatible” than those from frontier models such as Astra and Fable 5.1 running inside Codex / Claude Code, benchmarked on diverse real-world indoor scenes — the AHa-3D blog is the system write-up, LiteReality-Agent is the formal evaluation.

Why it’s interesting

AHa-3D is the scene-reconstruction-and-animation instance of the agent-authored Real2Sim pattern this wiki has been accumulating all quarter: RoboSimStudio: Smartphone Video to Multi-Physics Simulation via Coding Agents authors a multi-physics simulator from a robot-task smartphone video, Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents uses a VLM orchestrator to build an “episodic twin” of object-robot interaction, SIMPACT: Simulation-Enabled Action Planning using Vision-Language Models has a VLM both author the simulator and refine actions over it, and AHa-3D fills the room-scale indoor-video + human-motion cell of the same design space. Unlike SIMPACT and RoboSimStudio, the downstream task is not robot control — it is video re-rendering, action editing, and interactive scene export — which means AHa-3D is also the direct architectural neighbour of Astranomy: An Empirical Study of GPT-6 Astra as a Native Robot Policy‘s “GPT-6 Astra as native policy” probe (same Astra model, same tool-use thesis) but on a different payload. The sharpest contrast is with RoboSimStudio: Smartphone Video to Multi-Physics Simulation via Coding Agents, which keeps the agent-authored-simulator cell but swaps the input (robot smartphone video vs indoor room video), the output (training-time rollouts for GaP vs re-rendered Blender video with human motion), and the downstream consumer (policy update vs animation). It is also a concrete datapoint for the hypothesis in 3D as code that scene representation + manipulation should live in code / DCC software driven by agents rather than in end-to-end generative weights — AHa-3D is the “agent drives Blender with vision tools” instantiation of that bet.