Delphi: Scaling Laws That Extrapolate 300× Past the Fit

TL;DR

Delphi is the Marin team’s first open scaling suite: a scaling recipe mapping compute budgets to full training configurations, a suite of checkpoints trained under that recipe on the Google TPU Research Cloud, and a scaling law that uses smaller Delphi models to forecast the larger ones. The headline result is that a pre-registered scaling-law fit on IsoFLOP buckets from 3e18–3e20 FLOPs predicted the final validation loss of a 1e23 FLOP / 25B-parameter / 600B-token run within 0.2% — a 300× extrapolation past the largest fitted run. The load-bearing recipe changes over an initial attempt (which diverged at 1e23) were a token-horizon correction $(T_0/T)^{0.3}$ on learning rate and switching to the AdamH optimizer (a Frobenius-norm-sphere retraction that removes weight decay from the hyperparameter search). Checkpoints, training mixture pipelines, recipe code, development methodology, and per-figure plot data are all released.

Key claims

A scaling law fit on IsoFLOP buckets from 3e18 to 3e20 FLOPs forecasts a 1e23-FLOP held-out run to within 0.2% of measured final loss, with 1e21 and 1e22 held-out runs landing within 0.5% [Delphi attempt 2 forecasts 1e23 to within 0.2%].
The first Delphi recipe (µP-style width transfer + $\sqrt{B}$ LR-with-batch scaling + fixed step count + weight decay 0.1) missed the 1e22 forecast by 2.5% and diverged at 1e23 despite fitting the seven IsoFLOP buckets cleanly [Delphi attempt 1].
A $\left(\frac{T_0}{T}\right)^{0.3}$ token-horizon correction on the projection learning rate (in addition to $\sqrt{B/B_0}$ ) closes the extrapolation gap; scalar/embedding LRs get a joint $\sqrt{BT_0/(B_0T)}$ correction and $\varepsilon$ scales oppositely [Full Recipe Comparison table].
AdamH — Adam followed by a Frobenius-norm-sphere retraction of every projection weight — removes weight decay from the recipe (there is nothing to regularize away) and empirically transfers hyperparameters across widths more cleanly than Adam [AdamH and a token-horizon LR recipe, width-scaling figure].
Downstream benchmark scores are forecast via a two-step procedure: fit a Delphi scaling law to a soft benchmark metric (log-prob of the correct MMLU option; bits-per-byte of the HumanEval reference solution; soft GSM8K), then compose with a sigmoid observational projection fit on external open-weights models (Llama / Qwen / OLMo) to recover the discrete score [Predictions are accurate for downstream evals as well].
Seed-to-seed spread across three 1e21 and three 1e22 seed replicates is ~0.1%, roughly 10× tighter than the bootstrap 95% CI of the scaling-law forecast at the same budgets (±0.5% at 1e21, ±4% at 1e23) [Seed spread is 10× tighter than the forecast CI].
The compute-optimal-vs-overtrained penalty is quantified: 10× past the compute-optimal token ratio lands at a loss a compute-optimal run would only reach with ~6× the compute, extrapolated from the same seven IsoFLOP buckets [Quantifying the cost of leaving compute-optimal].

Method

The unit of experiment is a scaling recipe: a functional form that takes (compute budget $C$ , sequence length $\ell$ , model width $H$ ) plus reference-model tunings $(B_0, T_0, \eta_0, \eta_{0,\mathrm{Adam}}, \varepsilon_0)$ and returns a full training configuration — steps, depth, heads, batch size, learning rate, momentum, $\varepsilon$ , gradient clip, weight decay, init std. Delphi’s architecture family is Qwen 3, tokenizer is Llama 3, data mixture is Nemotron-CC + StarCoderData + ProofPile 2, LR schedule is WSD with 10% warmup and zero floor, precision is f32 params / bf16 compute, parallelism is FSDP — all held fixed across the suite so that a misbehaving run points back at the recipe.

Recipes are stress-tested with IsoFLOP sweeps: for each of seven compute budgets from 3e18 to 3e20, sweep model size while adjusting token count to hold total FLOPs fixed, fit a per-budget parabola on validation loss, extract compute-optimal points, and fit an asymptotic power law through the minima. That power law is what gets extrapolated to the 1e21 / 1e22 / 1e23 held-out runs. The recipe transition from attempt 1 to attempt 2 kept the same seven IsoFLOP buckets and the same fit protocol; only the recipe changed. AdamH’s update rule takes a step with Frobenius norm matching $\eta \|W\|_F$ regardless of the update magnitude, then projects back onto the sphere of radius $\|W\|_F$ , so weight norm is preserved exactly and weight decay has no work to do. Reference-model hyperparameters were retuned with Vizier (Google’s GP-based optimizer) instead of grid search. Downstream forecasting uses soft metrics (correct-answer log-prob for multiple-choice, reference-completion bits-per-byte for generative) fit with the same scaling-law family as pretraining loss, then a sigmoid observational projection fit on public models maps soft-metric to hard-metric.

Results

The main headline: at held-out compute budgets 1e21 FLOPs → +0.5%, 1e22 FLOPs → +0.2% (was +2.5% under Delphi attempt 1), 1e23 FLOPs → +0.2% (diverged under Delphi attempt 1) [Delphi attempt 2 forecasts]. At every overlapping IsoFLOP point, Delphi attempt 2 sits 0.01–0.15 loss below attempt 1; the parabola shapes are similar, so the fix is a recipe that transfers better across scale, not a new compute-optimal allocation. Downstream: soft-metric fits are noisier than pretraining loss but hold across MMLU / HumanEval / GSM8K at all three held-out budgets, and the observational projection recovers hard scores at 1e23 of ~60% MMLU, 19% HumanEval, 27% GSM8K. Seed spread across three replicates each at 1e21 and 1e22 is ~0.1%, staying inside a bootstrap 95% CI that widens from ±0.5% at 1e21 to ±4% at 1e23 as extrapolation length grows. The overtraining calculator (extrapolating the per-bucket ○−× gap between overtrained and compute-optimal points) puts 10× overtraining at ~6× the compute-equivalent loss, though the authors flag this side quest is not held-out validated the way the compute-optimal fit is.

Why it’s interesting

Delphi is the cleanest recent instance of the Fitting paradigm this wiki has catalogued in Hyperparameter scaling laws: predict optimal hyperparameters from cheap small-scale IsoFLOP runs, publish the recipe as a functional form, and validate the fit with held-out runs at multiple scales. The token-horizon LR correction $(T_0/T)^{0.3}$ is a concrete instance of the Complete(d)P direction — necessary because $\sqrt{B}$ -only scaling breaks when the batch-size-vs-step-count tradeoff drifts far from the reference. The AdamH intervention is a sibling to the Hyperparameter scaling laws “SSO / Muon-is-half-aligned” debate: AdamH constrains projection weights to a Frobenius-norm sphere (via retraction of each step), which is a different geometric constraint than SSO’s spectral sphere but shares the goal of removing degrees of freedom from the hyperparameter search. Sid pointed to this post specifically for the downstream-benchmark plot (soft-metric fit + sigmoid observational projection): the same forecast intermediate checkpoints on downstream evals to check they improve monotonically with pretraining pattern he is trying to reproduce for VLA sim-eval on RoboLab — Task-Based Evaluation Benchmark for Robot Manipulation Policies on Isaac Lab (NVLabs), which suggests the two-step soft-metric → hard-metric methodology may transfer to robot-policy sim-eval where success rates are similarly discrete and noisy at small scale.