The flavor of the bitter lesson for computer vision

TL;DR

Position essay by MIT’s Vincent Sitzmann (Scene Representation Group) arguing that the conventional scoping of computer vision — image → intermediate representation (class labels, masks, point clouds, NeRFs, camera poses) — is a historical compromise that the Bitter Lesson is now dissolving. The constructive claim is that vision will be subsumed into end-to-end perception–action loops with world models as the scalable pre-training objective, and that explicit 3D representations (point clouds, radiance fields, SDFs, camera poses) will follow hand-crafted features into obsolescence. The piece is notable for being written by an author whose own group built much of the modern 3D-representation stack (SIREN, scene representation networks, light-field networks), arguing against its long-term relevance for embodied AI. Useful as a framing document for why a video-generation/world-model research team should resist re-adopting explicit 3D as a backbone.

Key claims

Computer vision was historically scoped as “knowing what is where” — a mapping from images to intermediate representations — because end-to-end perception–action learning was intractable, not because the factorization is principled [§historical framing].
The Bitter Lesson applies to representations as well as algorithms: hand-crafted intermediate representations (3D point clouds, NeRFs, SDFs, voxels) are clever bottlenecks that scalable end-to-end training should be expected to outperform [§“Case study”].
Existing few-shot novel-view-synthesis SOTA is already held by generative world models (e.g. History-Guided Video Diffusion on RealEstate10K) rather than by 3D-differentiable-rendering approaches [§“Case study”].
SE(3) camera poses are themselves a redundant intermediate: novel view synthesis can be formalized as a pure representation-learning problem without multi-view geometry (xfactor / “True Self-Supervised NVS is Transferable”) [§“SE(3) Camera Poses…”].
Ego-motion is just the simplest action an agent takes — it does not warrant a dedicated architectural primitive distinct from general action modeling [§“SE(3) Camera Poses…”].
For 3D engineering tasks (CAD, manufacturing), explicit 3D meshes are reasonable at the human–machine interface layer, but should not be baked into training architectures; models should learn task-relevant internal structure on raw data and be fine-tuned to emit user-facing 3D representations only at the end [§“To get models competent…”].
The central bottleneck for embodied intelligence is the lack of paired perception–action data at scale; teleoperation scales for self-driving but not for dexterous humanoids, and current intrinsic-motivation RL is too sample-inefficient for real-world autonomous data collection [§“The key challenge…”].
World models offer two angles on this bottleneck: (i) video generative modeling as a scalable pre-training objective whose latents implicitly encode physics + human skill structure, and (ii) action-conditional video models as simulators for data amplification — but (ii) has a chicken-and-egg problem because training interactive world models requires the very paired data the field lacks [§“The key challenge…”].
Video generative models are not claimed to be a complete solution to embodied intelligence; they are one early attempt at finding a scalable perception–action pre-training task, and may not be in the final stack [§“The key challenge…”].
Open sub-problems explicitly named: intrinsic motivation, exploration, long-horizon memory, continual learning, real-time control with large models [§“The key challenge…”].

Method

This is a position essay, structured as three nested arguments. First, a historical/sociological argument that the boundaries of computer vision were drawn for tractability, not principle. Second, a Bitter-Lesson-style argument that intermediate representations (3D, camera poses) are not load-bearing — supported by a worked example (NVS now led by generative video models) and a thought experiment (“what real problem could you solve with a perfect 3D reconstruction of your room that you couldn’t solve with images directly?”). Third, a constructive claim that the unit of progress should be the perception–action loop, with world models as the most promising current pre-training objective, while explicitly listing the open problems (paired-data scarcity, intrinsic motivation, long-horizon memory) that this program faces.

The essay is also notable for what it doesn’t argue: it doesn’t claim video generation will solve embodied intelligence, doesn’t propose a specific architecture, and doesn’t critique any specific competing program — it stays at the level of which problems the field should be willing to scope.

Results

No empirical results. The essay cites the author’s group’s own work as worked examples: History-Guided Video Diffusion (Boyuan Chen, Kiwhan Song et al.) for few-shot NVS without 3D structure; xfactor / “True Self-Supervised Novel View Synthesis is Transferable” (Mitchel et al.) for NVS without camera poses or multi-view geometry; Diffusion Forcing and the Oasis interactive world model as auto-regressive rollout demonstrations; “Large Video Planner” as evidence that video generative models are already useful for robotics today; and Generative View Stitching for long-video generation under fixed camera trajectories. The references include foundational pointers: Ha & Schmidhuber’s world-models work, the Dreamer v1–v4 line (Hafner), the Neural Scene Representation and Rendering paper (Eslami et al., GQN), Girshick’s “Parable of the Parser,” and Jon Barron’s recent talk on radiance fields in the age of video models.

Why it’s interesting

The essay lands directly on a position the wiki has been accumulating evidence for. World Foundation Models currently holds papers split across “predictive-latent” WFMs (VL-JEPA / VJEPA-2) and “generative-rollout” WFMs (Cosmos Predict, Genie 3, Waymo World Model, LingBot-World); Sitzmann’s framing is the generative-rollout-as-pretraining variant, with the strong additional claim that no hand-crafted 3D representation should sit inside the loop. This matters for Luma specifically: large parts of the field (and historically Luma’s own research portfolio) still treat 3D reconstruction (NeRF, Gaussian Splatting, VGGT-derived feed-forward 3DGS / 4DGS) as a load-bearing intermediate; this essay is the clearest articulation of the counter-position. The author’s credibility on this is unusually high — his group authored several of the canonical 3D-representation papers (SIREN, scene representation networks, light-field networks) — so the argument that 3D should be discarded carries more weight than the same argument from an end-to-end-only researcher.

The wiki’s 4D-generation cluster (4D Scene Generation) is currently 100% reconstruction-then-clean-up / explicit-state-and-edit (NeoVerse, VerseCrafter, SpaceTimePilot — all Wan2.1-based, all with explicit 3D or 4D Gaussian state). Sitzmann’s essay is the strongest articulation on file of why that whole design family may be a local optimum. It does not refute those papers (they’re at the product layer, not the foundation layer) but it does argue the foundation layer should not look like them. Worth re-reading when reviewing any proposal that adds a 3D-structured module to a generative video stack.

Weaknesses worth flagging: the essay’s strongest empirical claim — that few-shot NVS SOTA is held by generative video models rather than 3D-differentiable-rendering — is sourced to the author’s own group’s work (History-Guided Video Diffusion); the wiki has not filed a direct head-to-head with a strong feed-forward 3DGS reconstructor, so this is an asserted rather than externally-validated empirical lead. And the essay handwaves the chicken-and-egg problem of action-conditional world models (you need paired data to train them; they’re meant to amplify the paired data you lack) without offering a path through.