A Functional Taxonomy of World Models

TL;DR

Position essay from Fei-Fei Li and the World Labs team proposing a three-way functional split of the overloaded “world model” term, grounded in the partially-observable Markov decision process (POMDP) loop: a renderer outputs observations (pixels for human eyes; optimizes for visual fidelity), a simulator outputs state (geometry, physics, dynamics; serves both humans and downstream programs like RL agents), and a planner outputs actions (the inverse of the renderer, closing the perception-action loop). The essay argues simulation is the linchpin — the structural backbone from which both rendering and planning can be derived — but is the least commercially mature of the three. The longer-term thesis is that the boundaries between renderer / simulator / planner will collapse into a single unified world model, and Marble (Gaussian splats + collision meshes from one model) is named as World Labs’ first move toward that endpoint.

Key claims

“World model” is overloaded across CV, robotics, RL, and generative AI; the unifying frame is the POMDP loop where an agent takes actions, world state updates, and the agent receives partial observations [§“The loop beneath the taxonomy”].
Three functional categories partition most of what is shipping today: renderer (action → observation pixels), simulator (→ explicit state: geometry/physics/dynamics), planner (observation → action) [§“Three functions of a world model”].
Renderers are the most commercially mature — image/video products (Nano Banana cited as a hundreds-of-millions-of-users example) optimize for visual plausibility but cannot be trusted to design a building or train a robot [§“Why simulation is the linchpin”].
Planners are the most nascent — current robotic demos are confined to “heavily constrained laboratory setups, with narrow object sets and short task horizons” and have not been validated at deployment-grade complexity, variability, or duration [§“Why simulation is the linchpin”].
Simulation is positioned as the bridge: a model that masters simulation can project understanding into pixels (for humans) and into action consequences (for embodied agents); a model that masters only rendering or only planning cannot do either [§“Why simulation is the linchpin”].
3D data with explicit geometry, material properties, and physical annotations is “orders of magnitude scarcer” than the internet video that renderers train on, and multi-physics simulation (rigid + deformable + fluid + cloth) at scale remains “orders of magnitude more expensive” than single-domain simulation [§“Why simulation is the linchpin”].
Generative simulators add a new failure mode on top of the sim-to-real gap: AI-generated geometry can look correct while containing self-intersections or wrong scale that produce nonsensical physics [§“Why simulation is the linchpin”].
Marble is named as World Labs’ first move into the simulator category — multimodal prompt (text / image / video / spatial sketch) → Gaussian splats plus collision meshes that a physics engine can operate on (i.e., a single model already crossing the renderer↔simulator boundary) [§“Why simulation is the linchpin”].
The forward-looking thesis: “The logical endpoint is a unified world model: one foundation model that can render photorealistic views, produce physically accurate structure, and plan action sequences, switching between output modalities depending on what the downstream consumer needs.” Reconciling visual-beauty vs. physical-precision data tensions inside a single architecture is named as “the defining open problem in world model research today” [§“Where the boundaries are collapsing and what comes next”].
Strategic market sizing claim: NVIDIA’s Omniverse alone targets >$1T addressable market across factories, warehouses, supply chains, and digital twins — used as evidence for simulation’s commercial weight [§“Why simulation is the linchpin”].

Method

The piece is a position essay, not a technical paper — its “method” is conceptual cartography. It anchors the taxonomy in the Sutton & Barto agent–action–state–observation loop (i.e., the POMDP), reinterprets the renderer / simulator / planner trichotomy as three different projections of that loop (which output channel of the loop the model produces), and then argues those projections are converging into one underlying model of geometry + physics + dynamics. Historical anchoring is via Kenneth Craik’s 1943 “small-scale models” of reality, and the modern term “world model” is traced to neural-network use in the late 1980s / early 1990s. The closing argument is a unification claim: the same knowledge (how a cup sits on a table, what happens when it is pushed, how a hand picks it up) underlies all three projections, so a sufficiently good simulator should be derivable into both a renderer and a planner.

Results

No experimental results. Concrete artifacts pointed to as evidence for the framing:

Renderer examples: Google’s Genie 3, World Labs’ RTFM (real-time interactive frame generation), generic text-to-video products, Nano Banana.
Simulator examples: NVIDIA Omniverse ecosystem; Marble (Gaussian splats + collision meshes from one model).
Planner examples: Vision-Language-Action (VLA) models, model-based RL systems, the “World Action Model” framing.
Boundary-collapsing examples cited as evidence the unification is already underway: pretrained video renderers being used as backbones for joint world-and-action prediction in robotics labs (renderer → planner); Marble producing splats + meshes from one model (renderer + simulator); action-conditioned renderers and editable/controllable simulators across the board.

Why it’s interesting

This is the clearest public articulation to date from a major lab of why the world-model space is splintering and where it should converge — and it complements rather than competes with the existing position essays the wiki has filed on this cluster. It fits between two prior World Labs pieces (3D as code — “3D is to physical space what code is to computation, durable + inspectable interface”) and the long-form Vincent Sitzmann argument (The flavor of the bitter lesson for computer vision — “explicit 3D / SE(3) / NeRF intermediates will dissolve into end-to-end generative-rollout WFMs by the Bitter Lesson”). Fei-Fei’s taxonomy explicitly resists Sitzmann’s collapse: she keeps explicit state (the simulator’s contract is structural — geometry + physics + dynamics) as a first-class output channel, alongside pixels and actions, and argues Marble already crosses the splat↔mesh boundary while preserving the structural side. It also clarifies, in a way no filed paper has, why the WFM cluster’s empirical / normative debate has felt confused: people are arguing about three different output channels of the same loop. Useful as a vocabulary anchor for the World Foundation Models concept page going forward — the cluster’s existing key claims already partition along very similar lines (generative-rollout vs. predictive-latent vs. action-conditional), and this essay names the partition explicitly.