On the Slow Death of Scaling

TL;DR

Position essay (SSRN, Dec 2025) arguing that the compute↔performance relationship that drove the last decade of AI is breaking down, and that frontier progress is increasingly determined by data quality, algorithmic technique, architecture, and inference-time compute rather than parameter count. Hooker (ex-Cohere Labs, ex-Google Brain/DeepMind) marshals examples — Falcon 180B beaten by Llama-3 8B, BLOOM 176B beaten by Aya Expanse 8B — and argues scaling laws hold for pretraining loss in controlled regimes but fail to predict downstream capabilities or survive medium-horizon extrapolation. The constructive half names four “new optimization spaces” worth investing in: gradient-free inference-time compute, malleable synthetic-data pipelines, design/interface as part of the model system, and post-transformer architectures. Useful as a framing piece for why a research team should split its budget across non-scaling axes.

Key claims

The accepted formula — scale model size + training data — has reshaped both the science and the funding structure of AI, marginalizing academia and shutting down publication at industry labs [§1].
Bigger is no longer reliably better at fixed quality: Falcon 180B is outperformed by Llama-3 8B, Command R 35B, Gemma 3 27B; BLOOM 176B is outperformed by Aya 23 8B and Aya Expanse 8B (~4.5% of the parameters) [§2, Fig. 3].
Open-LLM-Leaderboard data shows a systematic trend of small (<13B) models beating larger ones over time, not a handful of cherry-picked cases [§2, Fig. 3].
Diminishing returns to model size are partly explained by representation redundancy: most weights can be removed post-training with minimal loss, and a small set of weights can predict ~95% of the rest (Denil et al., 2014); but training without those weights from scratch fails [§3.1].
DNNs are inefficient long-tail learners — under average-error minimization on uniformly-iterated batches, the bulk of training compute is spent memorizing infrequent features; “a ladder to the moon” [§3.1].
Data curation (de-duping, pruning, prioritization) can compensate for model size, so parameter count is not the binding constraint on performance [§3.2].
Algorithmic techniques — instruction-finetuning, distillation, CoT reasoning, long context, RAG, RLHF/DPO — produce gains comparable in magnitude to compute scaling at fixed compute [§3.3].
Architecture sets the ceiling: CNNs unlocked vision, Transformers unlocked language; a new architecture can render any existing scaling law irrelevant [§3.4].
Scaling laws are empirically well-supported only for pretraining test loss in controlled regimes; downstream-capability scaling is “murky or inconsistent” and the “emergent properties” framing is a tacit admission that scaling laws don’t predict what’s coming [§4].
Most published scaling-law fits are based on <100 model data points; small-sample power-law claims are statistically weak and don’t survive medium-horizon extrapolation [§4].
The most interesting forward axes are new optimization spaces: gradient-free inference-time compute (search, tool use, agent swarms, adaptive compute — estimated 5×–20× gains over base post-training), malleable synthetic-data pipelines (steering distributions, breaking IID assumptions), system/interface design as a first-class concern, and post-transformer architectures with better continual-learning behavior [§5.1, §5.2].
The “slow death of scaling” is about training compute, not total AI energy: inference/serving costs dominate the energy footprint and will keep rising even as model sizes trend smaller [§5.2].

Method

This is a position essay, not an empirical study, so “method” is the structure of the argument. §1–§2 establish the historical setup (GPU repurposing → bigger-is-better → bitter lesson) and present the counterevidence (small-model leaderboard plots, named model-vs-model comparisons). §3 decomposes the compute-rate-of-return into four levers — model size, data quality, algorithmic technique, architecture — and argues each can move performance independently of compute. §4 critiques scaling laws on three grounds: they predict pretraining loss not downstream capability, they have small-sample statistical issues, and they fail under architecture/data shifts. §5 lays out the constructive program: four “new optimization spaces” (inference-time compute, malleable data, design/interface, post-transformer architectures) plus a caveat that total AI energy use will keep rising.

The essay draws on Hooker’s prior work (the Hardware Lottery, 2021; her earlier essays from 2020 and 2024) and is cited as a synthesis of talks she has given over several years. The references are heavy on Cohere Labs and adjacent research (Aya family, Command R, multilinguality), reflecting the author’s vantage.

Results

Not an empirical paper — no headline numbers of its own. The numerical claims it reuses from cited work:

Inference-time compute techniques estimated to give 5×–20× gains over base post-training performance (Davidson et al., 2023) [§5.1].
Denil et al. (2014): a small subset of weights predicts ~95% of weights in the network [§3.1].
Aya 23 8B / Aya Expanse 8B outperform BLOOM 176B at ~4.5% of the parameters [§2].
Code generation shows clean power-law scaling across ~10 orders of magnitude of compute; other capabilities scale erratically (Srivastava et al., 2023; Caballero et al., 2023; Hu et al., 2024) [§4].

Why it’s interesting

The essay is the strongest single articulation of a position several of the wiki’s filed papers already implicitly bet on. Inference-Time Scaling (recursive LMs, MiroThinker’s interaction-depth axis) is exactly Hooker’s “gradient-free exploration” lever; Synthetic Training Data (L2P, InfTool, Video-Thinker, SpaceTimePilot, NeoVerse) is exactly her “malleable data space” claim — five papers on the wiki already substitute generator-stack + verification-gate for human labels. So the essay is useful less as new evidence than as a clear frame for talking about what those papers have in common, and what they are an alternative to.

The argument has weaknesses worth flagging for a research-team audience: §3.2–§3.3’s “data and algorithms compensate for compute” claim doesn’t actually rule out that those techniques also scale with compute (instruction-tuning at 70B beats it at 7B; the relevant question is the joint surface, not whether algorithms-alone is a viable axis). And the Falcon-180B-vs-Llama-3-8B comparison conflates parameters with compute (Llama 3 was trained on far more tokens at much higher effective compute); the leaderboard plots make a real point about parameter scaling but a weaker point about compute scaling.

Still, the categorization in §5.1 — gradient-free inference compute, malleable data, design/interface, post-transformer architecture — is a useful taxonomy for what the team is reading and filing. If half the recent filings cluster under two of those four, that’s signal worth acting on.