Implications of Large-Scale Test-Time Compute (Noam Brown)

TL;DR

Noam Brown (OpenAI VP of Research) argues that as LLMs become more capable, benchmark performance is increasingly a function of test-time compute, and that the current practice of comparing models via single scalar scores is broken. Two years after o1, labs still report scalar evals, safety orgs are still surprised when a scaffold beats a base model via 100× inference, and Responsible Scaling Policies (RSPs) still ignore inference budget when setting capability thresholds. The proposal: replace scalar scores with performance-vs-cost / tokens / time curves, treat parallel multi-agent inference as a first-class deployment mode, and make third-party benchmark orgs impose or track explicit inference budgets so evaluations are comparable.

Key claims

• Benchmark performance for modern frontier LLMs is increasingly a function of test-time compute, and the capability ceiling is often unknown because runs are stopped at time/infra limits rather than at a genuine performance plateau [tl;dr / body].
• Evaluating a model by a single number on a bar chart materially misrepresents capability when the same model spans an ~10⁵× cost range (from $10 to$1M of inference per query) with monotonically improving scores [body].
• Safety evaluations conducted at low inference budgets can miss capabilities that emerge under sustained higher-budget inference by a determined actor — the “harmless at $10 of compute, capable at$10⁶” failure mode [body].
• Some recent models continue to improve past 100M generated tokens per query, with evaluations “stopped due to time and infrastructure constraints” rather than because a performance plateau was observed [body].
• Multi-agent parallel inference (many agents solving the same problem, best answer selected) is a distinct scaling axis from single-agent long-thinking: less cost-efficient per unit of quality but reduces wall-clock latency and can raise ceilings [body].
• Proposed replacement metric: performance curves with cost / tokens / time on the x-axis and score on the y-axis, not scalar leaderboards [body].
• Proposed benchmark-org discipline: track inference-compute consumed during evaluation, or impose explicit token/cost/time budgets, analogous to time limits on human exams [body].
• Proposed RSP discipline: capability thresholds must explicitly reference the test-time-compute scale at which they were measured [body].

Method

Not a paper — an X Article (tl;dr posted 2026-06-09 as the linked tweet) later delivered as a keynote at the Global AI Frontier Symposium 2026 in Seoul on 2026-07-03. The argument structure is a critique of current practice + a proposed replacement framework, illustrated with GPT-5.5 as a concrete example (bench-score deltas over prior generations look small, but user-perceived-quality deltas and long-token-budget deltas are large). The bar-chart-vs-curve reframing is the load-bearing move.

Results

No experiments; the piece is a policy / evaluation-framework argument. Concrete anchors:

• GPT-5.5 held up as a case where scalar-score deltas understate perceived capability deltas.
• 100M-tokens-per-query as the empirical high-water mark past which some current models are still improving when runs are cut off for infrastructure reasons.
• The $10 vs$1M-per-query dynamic-range figure is used to argue that a single-number report is uninformative at frontier scale.

Why it’s interesting

Directly frames the wiki’s Inference-Time Scaling concept as an evaluation-methodology problem, not just a training/serving one. The six-axis taxonomy on that page (scaffold-side, interaction-depth, sequence-budget compression, learned parallel-agent orchestration, per-step iterative refinement, test-time weight updates) is exactly the space Brown is arguing single-scalar scores obscure — his 100M-token / $10-vs-$1M dynamic range is a first-person confirmation from OpenAI that the axis matters at frontier scale. Complements On the Slow Death of Scaling (which argues pretraining scaling is slowing) by identifying inference-compute as the axis that’s not slowing and demands new eval hygiene. Sharpens the safety-eval framing in Natural Emergent Misalignment from Reward Hacking in Production RL and Anthropic’s Claude Opus 4.8 effort-control release (Introducing Claude Opus 4.8): if the effort knob is a user-facing product surface, it must also be a safety-eval surface.

See also

• Inference-Time Scaling — the concept page this piece frames as an evaluation problem
• Inference-Time Hyper-Scaling with KV Cache Compression — DMS: concrete implementation of the “spend more inference tokens at fixed memory” axis Brown argues eval must account for
• MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling — interaction-depth as an independent scaling axis (600+ tool calls per task)
• Recursive Language Models — scaffold-side scaling; direct evidence that a scaffold can turn 0.04 into 58.0 F1 on the same base model, which is exactly the “scaffold beats base by 100×” phenomenon Brown says safety orgs still miss
• Introducing Claude Opus 4.8 — Claude Opus 4.8 ships a user-facing effort control (low/high/xhigh); productization of the axis Brown says eval must reflect
• Kimi K2.5: Visual Agentic Intelligence — Kimi K2.5 PARL: learned multi-agent parallel orchestration, the fourth scaling axis on the concept page and the one Brown specifically calls out as a wall-clock-latency lever
• On the Slow Death of Scaling — companion piece on why pretraining scaling is slowing, complementary to Brown’s argument that inference-compute scaling is not
• Introducing GPT-5.5 — GPT-5.5, the concrete model Brown cites as the “scalar score understates capability” example