3D as code

TL;DR

Position essay from World Labs arguing that 3D is to physical/virtual space what code is to computation: a durable, externalized, inspectable interface that humans and machines should generate, edit, and exchange — not a transient artifact buried inside a model’s activations. The constructive claim is a factorized world-model stack where neural graphics functions as the “spatial programming language” (NeRF / Gaussian splatting), simulation engines as the “spatial chips” (physics, collision, persistence, replay), and 3D as the interchange medium between them. The essay positions World Labs’ own Marble, Chisel, RTFM, and Spark as instantiations of this stack, and explicitly contrasts it against the “pixels all the way” end-to-end approach (action-conditioned frame generation with state living only in transient neural activations).

Key claims

3D plays the same structural role in spatial systems that code plays in computational systems: a precise, externalizable, manipulable interface that supports composite human-machine workflows and machine-to-machine integration with existing pipelines (CAD, simulators, robotics stacks, GIS) [§“3D – the spatial ‘code’”].
The alternative — collapsing memory, dynamics, and rendering into a single end-to-end model that maps (history + action) → next frame — is presented as analogous to asking an LLM to be the program instead of write the program: it sacrifices reproducibility, replayability, modular composability, and the ability to distinguish editing-the-code from executing-the-code [§“Simulation engines”].
Neural graphics (NeRF, Gaussian splatting) is the “spatial programming language”: now feasible at world scale because AI-optimized GPU memory/bandwidth removes the asset-reuse/compression constraints that drove classical 3D pipelines [§“Neural graphics”].
Simulation engines are the “spatial chips” — they own state management, update rules, observations, physics, collision, lighting, synchronization, determinism, and replay; collapsing these into a learned model “weakens guarantees around physical consistency, replayability and determinism” [§“Simulation engines”].
In a factorized stack, externalized 3D artifacts (splats, meshes, videos) become inspectable, validated, versioned, testable, reusable — enabling composable pipelines analogous to code [§“3D is an interface”].
A monolithic shared-latent-space approach is workable only when all spatial reasoners share a model or latent space; in heterogeneous, modular, multi-agent settings (planner ↔ safety monitor ↔ perception ↔ renderer) structured 3D is a better lingua franca than latents or natural language [§“3D is an interface”].
3D matches human cognition: people navigate persistent objects and relationships, so an interface built on explicit spatial structure aligns with users’ mental models — and a single 3D edit propagates automatically across all rendered frames (vs 2D animation, where every frame is redrawn) [§“For people”].
World Labs’ own products instantiate the thesis: Marble (multimodal world model that reconstructs/generates/simulates 3D worlds, exports splats/meshes/video), Chisel (coarse 3D-layout conditioning interface), RTFM (learned rendering of complex visual effects from structural inputs), Spark (Gaussian-splatting WebGL renderer) [§“Toward the future”].

Method

This is a position essay structured as an extended analogy: 3D ↔ code, neural graphics ↔ programming languages, simulation engines ↔ chips. The argument proceeds in three layers. (1) Why externalized structured artifacts beat fully-internal model state, framed via a parallel to asking LLMs to sort a list inside their token stream rather than emitting a sort program. (2) Why GPU hardware trends now make data-driven 3D representations (NeRF, Gaussian splatting) practical at world scale, removing the historical memory/bandwidth constraint that forced asset reuse and hand authoring. (3) Why factorized stacks beat collapsed ones for both human–machine and machine–machine interfaces — listing concrete properties (replayability, determinism, versioning, multi-agent interchange) that explicit 3D provides and end-to-end pixel generation does not. The piece does not propose a specific architecture or report experiments; it instead positions World Labs’ product portfolio (Marble, Chisel, RTFM, Spark) as instantiations of the factorized thesis.

Results

No empirical results. The piece cites World Labs’ own products as worked examples and gestures at “pixels all the way” research directions without naming or comparing specific systems.

Why it’s interesting

This is the strongest articulation on file of the opposite position from The flavor of the bitter lesson for computer vision. Sitzmann argues that explicit 3D (NeRFs, point clouds, SDFs, camera poses) is a hand-crafted intermediate the Bitter Lesson should dissolve, and that end-to-end perception–action loops with generative-rollout world models are the right pre-training substrate. World Labs argues the inverse: 3D is the durable externalized interface, “pixels all the way” approaches lack the structural guarantees needed for production stacks, and the right system is factorized — neural graphics + 3D interchange + simulation engines. Both authors agree video/neural rendering matters; they disagree about whether 3D state should sit inside the loop or between loops. The wiki’s 4D Scene Generation cluster (NeoVerse, VerseCrafter, SpaceTimePilot — all with explicit 3D/4D Gaussian state) is exactly the design pattern World Labs’ framing endorses and Sitzmann’s framing rejects. This essay is the cleanest statement of the pro-3D side and should be read alongside Sitzmann’s for the team to make its own call.

The originating Slack note from Scott Shumaker is itself a third position worth recording: that the source of truth should be a learned latent representation and 3D should be a projection materialized on demand — closer to Sitzmann’s end-to-end view but reframed as a representation question rather than a Bitter-Lesson prediction. The essay’s weak point is that it does not engage with the strongest “pixels all the way” results (e.g. interactive Genie-3-class systems) on their own terms; it asserts the factorization wins on guarantees without showing where the collapsed alternative actually breaks in production.