How GPT, Claude, and Gemini are actually trained and served – Reiner Pope

TL;DR

A blackboard-lecture interview with Reiner Pope (CEO of MatX, formerly TPU architecture at Google), framed as a roofline model of how frontier LLMs are actually trained and served on real clusters (Blackwell NVL72 as the worked example). The thesis: every decode operation is bounded below by either memory-bandwidth (weight + KV loads) or compute on active parameters, and most of modern API pricing, MoE layout, pipelining, and pretraining-token counts fall out as corollaries. The 1:19:00 segment Kyle flagged derives an “equal-cost heuristic” — at the lifetime optimum, pre-training compute, RL compute, and inference compute should each be of roughly the same order, which (back-solving) explains why frontier models appear ~100× over-trained relative to classical Chinchilla. The talk is the practitioner-side complement to filed academic scaling-law work; the equal-cost equation it gives is something Luma could fit numerically against its own deployment assumptions.

Key claims

Frontier-model inference is mostly memory-bandwidth-bound, not compute-bound, with output-token cost typically 3–5× input-token cost; the crossover to compute-dominated decode requires per-rack concurrent-sequence batch sizes around 2,000–3,000 for DeepSeek-V3-scale MoE models — a threshold few providers consistently hit [§“How batch size affects token cost and speed”, 00:00:00].
The total compute over a model’s lifetime decomposes as $C_\text{total} = C_\text{pretrain} + C_\text{RL} + C_\text{inference}$ with $C_\text{pretrain} = 6 N_\text{active} D_\text{pretrain}$ , $C_\text{RL} \approx (2\text{–}6) \cdot N_\text{active} D_\text{RL} \cdot \text{inefficiency}$ , and $C_\text{inference} \approx 2 N_\text{active} D_\text{inference} \cdot \text{inefficiency}$ ; if the three trade off against quality, the optimum is approximately where each is the same order [§“Because of RL, models may be 100x over-trained beyond Chinchilla-optimal”, 01:18:59].
Inefficiency factors: RL is forward-or-forward+backward at low decode MFU (multiplier 2–6×); inference is forward only at low decode MFU (multiplier ~1.5–2×); equalizing $C_\text{pretrain}$ with ( $C_\text{RL} + C_\text{inference}$ ) thus implies $D_\text{pretrain}$ must be ~two orders of magnitude above naive Chinchilla, matching the ~150 T-token rumored counts for ~100 B-active models [§“Because of RL, models may be 100x over-trained beyond Chinchilla-optimal”, 01:18:59].
All-to-all MoE communication (“expert parallelism”) strongly favors full connectivity inside one rack; with the trend toward smaller experts, tensor parallelism becomes much less relevant and can be ignored for frontier MoE layout [§“How MoE models are laid out across GPU racks”, 00:32:09].
The reason a bigger scale-up domain matters is memory bandwidth across the domain, not memory capacity — pipelining cheaply solves capacity, but bandwidth is what bounds how fast weights can be loaded and therefore how low decode latency can go [§Ilya’s “pipelining is not wise” framing, 01:03:37].
Failing to batch user traffic at large model decode destroys the cost structure relative to a well-filled batch — the gap is orders of magnitude, not incremental [§“How batch size affects token cost and speed”, 00:00:00].
Context lengths have stalled at ~200K because the KV-cache memory cost scales linearly with context and the bandwidth budget per request was already nearly spent on weights; long-context API pricing can be back-solved from the roofline equations [§“Deducing long context memory costs from API pricing”, 01:33:02].

Method

The talk is structured as a derivation rather than a result. Pope sets up the roofline-style inequality $\text{time} \geq \max(\text{compute time}, \text{memory-fetch time})$ for a single transformer decode step, plugs in the active-parameter count, KV-cache size, and per-rack memory bandwidth of a Blackwell NVL72, and then reads off the consequences for batch size, expert layout, pipelining, context length, and pricing. The pre-train / RL / inference compute partition (the segment Kyle flagged) is derived the same way: write each phase’s cost as $\text{multiplier} \times N_\text{active} \times D_\text{phase} \times \text{inefficiency}$ , then ask what allocation minimizes $C_\text{total}$ subject to quality being some monotone function of each $D_\text{phase}$ . Under the assumption that the three sources are sufficiently substitutable, the Lagrangian optimum sits where all three are equal, modulo the inefficiency multipliers. There is no transcript on YouTube; a community transcript exists at https://gist.github.com/dwarkeshsp/79100f0fdeed69d76241903bb0604dbe and an unofficial summary at https://www.dwarkesh.com/p/reiner-pope.

Results

The framework is itself the result. Two concrete back-solved numbers stand out. (1) The “300 rule” — a single dimensionless constant that relates DeepSeek-V3 active params, NVL72 memory bandwidth, and FLOP throughput to give the batch size above which decode flips from memory-bandwidth-bound to compute-bound; meeting it requires ~2,000–3,000 concurrent sequences per rack. (2) The over-training factor — a Chinchilla-optimal 100B-active model would see ~2 T pretraining tokens, but the equal-cost heuristic pushes pretraining tokens up to the ~150 T-token range, i.e. ~75–100× over Chinchilla, which matches reported frontier-model token counts. The talk asserts but does not measure that quality scales sublinearly enough in each phase’s $D$ for the equal-cost optimum to be a real optimum.

Why it’s interesting

This is the deployment-economics scaffolding underneath the team’s existing scaling-law thread. Test-Time Scaling Makes Overtraining Compute-Optimal makes the same qualitative argument from the academic side — once inference (specifically repeated sampling) is folded in, compute-optimal pretraining shifts deep into the overtrained regime, well past Chinchilla — and Pope here derives the practitioner-grade equation that produces the 75–100× factor directly from rack-level roofline arithmetic plus an RL term. The RL term is what Test-Time Scaling Makes Overtraining Compute-Optimal does not model: T2T folds in inference via pass-@k but does not include an RL training phase. Combining the two would give a four-term $(N, D_\text{pretrain}, D_\text{RL}, k)$ scaling law that nothing on the wiki currently has. The MoE-layout, pipelining, and bandwidth-vs-capacity threads also connect to Scalable Training of Mixture-of-Experts Models with Megatron Core (Megatron MoE training) and Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design (expert-vs-attention FLOPs ratio rising with compute), which both implicitly assume the kind of memory-bandwidth regime Pope explicitly models.