4DCodeBench: Benchmarking Agents on Inverse Graphics of Dynamic Scenes

TL;DR

4DCodeBench is a benchmark for 4D inverse graphics via code generation: coding agents are shown a video of a dynamic physical event and must emit an executable graphics program (Blender/Python, custom simulators, keyframed scenes) that reconstructs the scene’s geometry, dynamics, and appearance. The benchmark pairs 100 real-world videos with 100 synthetic scenes spanning rigid bodies, deformables, cloth and rods, grains, fluids, fracture, and multi-material interactions, and evaluates 18 frontier agents on how well their rendered output matches the reference along geometry, appearance, and dynamics axes. The headline finding: current frontier models can reconstruct static 3D structure well, but dynamics remain a clear capability gap — and agents differ substantially in how they attack the problem (67% describe motion analytically, 19% write custom simulations, 10% use Blender physics, 3% rely on keyframing).

Key claims

4D inverse graphics via executable programs is a distinct benchmark surface from pixel-level video reconstruction: agents must translate visual observations into compact code that reproduces structure, motion, and appearance, implementing abstractions like physical simulations rather than regressing to pixels [§Abstract].
The benchmark covers 200 scenes split 100 real + 100 synthetic, spanning diverse physical phenomena including rigid bodies, deformation, cloth and rods, grains, fluid flow, fracture, and multi-material interactions [§Abstract; tweet thread 3/n].
18 frontier agents are evaluated on geometry, appearance, and dynamics; strong static reconstruction capabilities do not translate into reliable reconstruction of complex dynamics [§Abstract; HF paper page].
Agents’ chosen abstractions diverge sharply: across benchmarked solutions, 67% describe motion analytically, 19% implement custom simulations, 10% use Blender physics, and 3% rely on keyframing — with Opus 5.5 reaching for simulation in 76% of cases [tweet thread 4/n].
The benchmark is structured as a two-stage pipeline: an inference stage runs a coding agent on a reference video and saves its submission; a scoring stage renders the submission and evaluates it against the reference, using VLM-as-judge (VQA + pairwise) alongside geometric metrics [GitHub README].

Method

Each benchmark instance presents a coding agent with a reference video of a dynamic scene and asks it to produce an executable graphics program whose rendered output reconstructs the input. Agents are free to choose their own representation and abstraction — analytical motion, custom simulators, Blender physics, keyframing — rather than being constrained to a fixed graphics stack. Evaluation is run via a public world-format checker (installed in the agent image) plus a scoring image that renders the agent’s submission and scores it against the reference along geometry, appearance, and dynamics axes, with a VLM-judge module running VQA and pairwise evaluations on top of the rendered output. The benchmark ships as a reproducible harness (scripts/download_data.py, pinned environments) with the agent containerized and the scorer isolated so submissions can be re-graded identically.

The construction mirrors the two-benchmark pattern this cluster has converged on — real-world videos probe transfer and messiness, synthetic scenes isolate specific physical phenomena with ground truth. All 200 scenes are released together with the scorer and VLM-judge pipeline, placing 4DCodeBench alongside WMBench and HarnessEval-W as a community-extensible, agent-facing evaluation surface rather than a frozen leaderboard.

Results

Headline results from the paper and companion materials (full numerical tables not yet extracted at filing time):

Across the 18 frontier agents evaluated, strong static 3D reconstruction does not carry over to dynamics; dynamics are explicitly called out as the bottleneck [HF paper page].
Agents’ approaches to modeling dynamics are heterogeneous: ~67% analytical, ~19% custom simulation, ~10% Blender physics, ~3% keyframing. Opus 5.5 reaches for simulation 76% of the time — the only agent whose first-choice abstraction is a physics engine [tweet thread 4/n].
The benchmark discriminates clearly along the dynamics axis: no agent approaches ceiling on the full 200-scene suite.

Why it’s interesting

4DCodeBench is the first filed paper to operationalize the “visual AI is code” and “3D as code” positions (The Next Frontier of Visual AI Is Code, 3D as code) as a benchmark on frontier coding agents. Where the a16z and World Labs essays argue the right output of a generative visual system is a source program (SVG, Lottie, Blender, USD) rather than pixels, 4DCodeBench measures how far current coding agents can actually push that recipe on the hardest substrate — reconstructing dynamic 3D scenes from real video. The finding that static reconstruction works but dynamics don’t is a sharp empirical datapoint on the open question those essays left unanswered: can the Code → Render → Inspect → Revise loop converge on physical-law behavior, or is dynamics the regime where pixel-native generation retains the advantage?

The paper also complements the 4D Scene Generation cluster from the opposite direction. The filed scene-level methods (NeoVerse, VerseCrafter, SpaceTimePilot, 4DAnyone) all go video → explicit 4D state → rendered video via a frozen Wan2.1-style backbone with learned conditioning; 4DCodeBench goes video → executable graphics program → rendered video via a coding agent, with no backbone training at all. The agent-choice breakdown (67% analytical / 19% simulation / 10% Blender / 3% keyframing) is a concrete first measurement of which abstractions frontier models naturally reach for when the choice is theirs — a question that is implicit but unanswered in the 3D as code “neural graphics + simulation engines + 3D interchange” factorization. And it extends Agentic Software Engineering‘s benchmark surface past SWE-Bench / Terminal-Bench into physics-aware code generation, in the same spirit as EmbodiedSWE: Coding Agents for Long Horizon Dexterous Robotics (coding agents for dexterous robotics) but with executable graphics code as the deliverable rather than robot trajectories.