>10x More Efficient Pretraining

TL;DR

Magic reports a pretraining recipe that is >10× more compute-efficient than leading open-weight base models. They match DeepSeek V4 Pro Base using ~~50× fewer FLOPs (~~ $0.5M on GB200, roughly half of GPT-3's pretraining compute) and continued scaling 10× (~$ 4M) to exceed all publicly-available base models on heldout bits-per-byte perplexity. The gain is the multiplicative result of tens of changes across architecture, optimizer, training objective, and data curation — plus bug fixes treated as compute multipliers. Evaluation is done via BPB scaling-law fits across 167 domain-decomposed heldout sets, with per-generation new “knowledge evals” to avoid overfitting.

Key claims

The recipe matches DeepSeek V4 Pro Base at ~50× fewer FLOPs; 10× scale-up outperforms all evaluated open base models (DeepSeek V4 Pro/Flash, Kimi K2, Nemotron 3 Ultra) on heldout BPB [Fig. 1, footnote 1].
Bits-per-byte on heldout data is used in place of prompt-based sampling evals because base models are prompt-sensitive; BPB also smooths otherwise-emergent abilities [footnote 2].
Code evals use private codebases; reasoning evals are Kimi K3-generated CoT walkthroughs on private math problems filtered for correctness; text/research uses recent low-citation papers — with 96-character normalized-window and Jaccard-similarity decontamination against training data [§ Evaluating generalization].
Knowledge evals are LLM-rewritten/summarized to avoid rewarding sequence memorization; created once per generation to avoid feedback-loop overfitting [§ Evaluating knowledge, footnote 4].
Nemotron 3 outperforms DeepSeek V4 Pro Base across domains and inference engines despite weaker post-RL benchmarks, suggesting Nemotron’s weak headline benchmarks reflect weaker post-training rather than a weaker pretrain [footnote 2].
Small-model results (NanoGPT speedruns) do not transfer cleanly — many tiny-model wins fail at scale, and some features present in most LLMs can be deleted without harming large-scale performance [§ No shortcuts].
Change-selection protocol: train 3 models spanning 2 orders of magnitude of compute per candidate; keep a change if its power-law fit predicts help at scale. Scale to 1/10th of hero scale every few weeks; full hero run every few months [§ No shortcuts].
Pretraining-loss improvements translate to post-RL performance in a short math RL sanity check (16k CoT budget, RL directly from base, no SFT/distillation); e24 model’s private-heldout AIME-harder-than-AIME26 pass@1 first crossed 90% (pass@16 100%) at ~0.2% of pretraining compute using ~6k CoT tokens [Fig. 5, footnote 5].
Nemotron shows source memorization outliers even after LLM rewriting of documents — predicting specific numbers/phrases from medicine/health source documents unexpectedly well [footnote 4].
Verified logprobs against Fireworks’ in-house engine after finding correctness bugs in both vLLM and SGLang inference backends [§ intro, GitHub issue links].

Method

The full recipe is not disclosed but is framed as (a) building a stable foundation first — smooth convergence, low-precision training matching FP32, stable infra, correct hyperparameter scaling — then (b) hunting bugs, then (c) compounding tens of individually-small changes across model architecture, optimizer, training objective, and data curation. Evaluation is BPB on heldout data with a scaling-law fit projecting compute-to-capability. Domain decomposition of heldout research text drives targeted data-curation decisions (some domains — coding, AI/ML — are prioritized; others — sports/events, local news, notable-person facts — are deprioritized). 167 domain scaling laws are fit separately. FLOPs are reported as 6·N·D (activated params × tokens) to compare against public models whose sequence-dimension cost is inconsistently reported.

Results

$0.5M on GB200 reaches DeepSeek V4 Pro Base parity (9.67e24 → 1.12e22-1.58e23 FLOPs, per the footnote-1 table).
$4M continues 10× further and beats all evaluated public base models.
DeepSeek V4 Pro Base scaling extrapolation predicts > $100M to match Magic's$ 4M run at parity (“training a model this capable would cost >$100M under DeepSeek V4 Pro’s recipe”).
Per-domain compute-efficiency gains reported across 167 domains (Figure not quantified in extracted text).
On a private heldout competition-math eval (400 problems, harder than AIME), the e24 model’s AIME26 pass@1 crosses 90% (pass@16 100%) at ~0.2% of pretraining compute using ~6k CoT tokens on average, with continued log-linear climb [footnote 5].

Why it’s interesting

Sits at the intersection of Hyperparameter scaling laws and Training stability at scale as the most aggressive publicly-claimed compute-efficiency gain of 2026 — the recipe explicitly names “stable foundation + hunt bugs + compound many small changes” as the mechanism, echoing the “remove degrees of freedom from the HP search” theme of Delphi: Scaling Laws That Extrapolate 300× Past the Fit Delphi and Controlled LLM Training on Spectral Sphere SSO but at a much larger end-to-end scale (frontier-competitive $4M run). The “small-model wins don’t transfer, and some ubiquitous LLM features can be deleted at scale” claim directly reinforces How to Set the Learning Rate for Large-Scale Pre-training?‘s finding that µP-motivated module-specific LRs stop paying off under modern normalization — i.e., that the frontier-scale architecture surface has quietly diverged from the NanoGPT-speedrun surface. Also notable: uses inline BPB scaling-law fits per domain as the change-selection oracle (train-3-models-across-2-OOMs, keep if power-law predicts help), which is Fitting-paradigm in the sense of How to Set the Learning Rate for Large-Scale Pre-training? rather than µTransfer-paradigm.