The Next Frontier of Visual AI Is Code

TL;DR

Position essay from a16z (Yoko Li) arguing that for any visual task whose users want to keep iterating — UI design, vector graphics, motion graphics, 3D assets — the right output of a generative model is the source program that produces the pixels (SVG, HTML/CSS/React, Lottie JSON, Blender/USD scenes), not the pixels themselves. The thesis hinges on a Code → Render → Inspect → Revise test-time loop: because the artifact is structured, feedback can map onto specific source-level edits and every retry improves the underlying program, unlike pixel-native sampling which mostly re-rolls the dice. The piece concedes the end-state will be hybrid (pixel-native for realism/exploration, code-native for structure/iteration/production) and flags 3D as the next domain where reframing generation as code is most valuable because consistency across views, parts, and interactions is hard to fake.

Key claims

Visual generation splits into two stacks — pixel-native (latent diffusion, video models) and code-native (model emits SVG/HTML/Lottie/Blender/USD; an external engine renders) — and production workflows reward code-native because the artifact stays editable, versionable, and integrable [§“The two stacks of visual generation”].
Code-native generation gives test-time compute a precise feedback loop: Code → Render → Inspect → Revise, where each iteration patches the source program rather than producing a fresh sample, so the loop has a chance to converge on a fix rather than re-roll [§“Code is a good substrate for visual problems”].
Diffusion-side inference-time scaling exists (cites Inference-Time Scaling of Diffusion Models through Classical Search) but reward can only tell the model one output beats another; it cannot map feedback onto a specific source-level edit, which is presented as the structural advantage of the code-native loop [§“Code is a good substrate for visual problems”].
The visual-code stack has three load-bearing pieces: a coding model that authors/edits the artifact; a symbolic source-of-truth representation (DOM, Lottie layers/keyframes, scene graph) that makes edits meaningful; and a renderer/engine (browser, SVG renderer, Blender, game engine, simulator) that becomes the feedback environment [§“The visual generation stack with code”].
OmniLottie is cited as the worked example: making Lottie model-native (raw JSON → compact command/parameter sequences) is what lets a model generate and edit animations reliably, because feedback can then map to source-level edits (timing curve, vector path, shape morph) rather than to a flat video [§“The visual generation stack with code”].
3D is the frontier where the framing pays off most: a rendered image of a chair is not a chair, and useful assets need consistent geometry, materials, part hierarchy, and functional constraints (doors open, hinges rotate, wheels spin) — none of which pixel-native generation supplies [§“Why 3D is the next important frontier”].
3D test-time loops only converge if the agent has the right tools — change camera views, query scene state, isolate objects, compare against target, remember prior attempts — not just “render Blender again until it looks better”; cites VIGA (Blender-as-feedback-env with semantic observation/modification tools and attempt memory) and Articraft3D (articulated 3D generation as writing programs that define parts/joints/tests) as the current points on the frontier [§“Why 3D is the next important frontier”].
The future is hybrid, not zero-sum: pixel-native models remain best for realism/texture/exploration, code-native systems are better for structure/iteration/production, and the winning products will own the loop — generate the artifact, render it, inspect what broke, revise the source [§“Future implications and unsolved problems”].

Method

Position essay. The argument is structured as: (1) define the two stacks (pixel-native vs code-native); (2) argue that production users want the artifact not the output, so editability is the load-bearing property; (3) re-cast test-time compute as a closed-loop debugger over a visual program rather than rejection sampling over images; (4) sketch the stack (model + symbolic representation + renderer) and the market-map heuristic that each runtime (browser, SVG, Lottie, Blender, game engines, simulators) creates a separate wedge with its own source representation and feedback loop; (5) flag 3D as the highest-value frontier because consistency across views/parts/interactions cannot be faked by pixels; (6) close with a hybrid prediction and a recruitment pitch (yli@a16z.com). The essay names two concrete academic touchpoints — OmniLottie (already filed) and VIGA/Articraft3D (project pages, not yet on the wiki) — but otherwise relies on examples and analogy.

Results

No empirical results. The piece is a market-map argument; its citations function as existence proofs that the proposed framing is already showing up in research (OmniLottie for Lottie, VIGA/Articraft3D for 3D).

Why it’s interesting

This is the generalized version of the thesis the wiki already has in concrete form. 3D as code argues 3D-as-code for spatial/world systems specifically; this a16z piece extends the same move (“the source of truth is the structured program, not the pixels it renders to”) across UI, graphics, motion, and 3D in one frame. Read together, the two essays describe an axis on which the team’s own decisions matter: every “should the model output pixels or output a representation?” question (latent layers vs RGB layers, scene-graph vs video, SVG vs PNG) is an instance of the same trade-off. The wiki’s Vector Graphics Generation cluster — OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens (the paper this essay cites by name), VecGlypher: Unified Vector Glyph Generation with Language Models, and Vector Prism: Animating Vector Graphics by Stratifying Semantic Structure — is the strongest existing empirical evidence for the code-native loop the essay describes. The flavor of the bitter lesson for computer vision is the cleanest counterpoint: Sitzmann argues hand-crafted intermediates (including code-as-representation) should dissolve as scale grows, which would predict the hybrid eventually collapses toward pixel-native. The essay’s weak point is that it cites one inference-time-scaling-for-diffusion paper and then dismisses the whole pixel-side loop as “just sampling more images” — but Inference-Time Scaling on this wiki already documents diffusion-side loops with non-trivial feedback structure (e.g. MCTS over rollouts, VLM-graded search), so the structural-superiority claim for code-native is overstated rather than airtight.