Skip to content
Bud's Wiki

Marin scaling laws simulate the entire training trajectory of a 67B MoE run within 1%

TL;DR

Section titled “TL;DR”

Marin (Stanford CRFM + Open Athena) team member Larry Dial reports that the Delphi scaling-law methodology now predicts not just the final loss of a large training run, but arbitrary intermediate points along the entire training trajectory to within 1%. The demonstration was a 67B-total-parameter MoE run whose loss target was hit within 1% and whose loss at arbitrary checkpoints was likewise predicted within 1% from the Delphi scaling ladder. Based on this, the team has kicked off a 535B-total-parameter MoE run with the trajectory already forecast from the same scaling ladder. Also notes that getting the run off the ground required substantial non-modeling work: trillions of tokens curated, a new GPU stack brought online, MFU improvements, and integration of new ideas.

Key claims

Section titled “Key claims”
  • The Marin Delphi-derived scaling recipe now predicts full training trajectories, not just final loss: on the 67B MoE run, “arbitrary points during training could also be predicted to within 1%” [tweet body].
  • The 67B MoE hit its target final loss within 1% [tweet body].
  • The 67B MoE is finishing long-context training and about to enter RL post-training [tweet body].
  • A 535B-total-parameter MoE run has been kicked off, with its full training trajectory forecast from the Delphi scaling ladder before the run started [tweet body].
  • Getting the run off the ground was not purely a modeling exercise: trillions of tokens curated, hardware running on a new GPU stack, MFU improvements, and integration of new ideas were required contributions [tweet body].
  • Aspirational open direction posed by the author: fit a scaling law per-parameter to predict the final model without actually training it [tweet body, framed as a joke but a stated open question].

Method

Section titled “Method”

The trajectory-prediction claim is a direct extension of the Delphi Fitting paradigm (Delphi: Scaling Laws That Extrapolate 300× Past the Fit): the recipe already fits IsoFLOP buckets from 3e18–3e20 FLOPs and extrapolates the terminal loss up to 1e23 FLOPs within 0.2%. The novel step announced here is that the same fit apparatus, applied along the training axis, predicts loss at any intermediate step within 1% at 67B MoE scale. The tweet points at GitHub issue #8435 for the run log; the Marin repo README confirms the group’s current focus is a ~5e24-model-FLOPs, 500B+-parameter MoE. No paper or blog writeup accompanies the tweet at filing time — the primary artifact is the trajectory-prediction claim itself plus the two attached figures.

Results

Section titled “Results”

Two headline numbers, both from the tweet body: 67B MoE target loss hit within 1% and arbitrary-point loss along the training trajectory predictable within 1%. The 535B MoE run is in flight; its trajectory has been forecast but not yet validated at time of filing. No quantitative comparison to the Delphi 300× extrapolation is drawn — this is a scale-up (67B here vs. the 25B / 1e23-FLOPs Delphi headline run) with the additional axis of per-step rather than terminal prediction.

Why it’s interesting

Section titled “Why it’s interesting”

This is the first filed evidence on the wiki that the Fitting-paradigm scaling law generalizes from final-loss prediction to full-trajectory prediction at industrial scale — a strictly stronger claim than Delphi: Scaling Laws That Extrapolate 300× Past the Fit‘s terminal-loss result and closer in spirit to Scaling with Collapse: Efficient and Predictable Training of LLM Families‘s “loss-curve scaling collapse” line, which uses the same predictive goal via a dynamics-based rather than closed-form route. If the 535B trajectory forecast holds, it also gives the Marin team a live pretraining validation of the Delphi recipe an order of magnitude past the fitted range. Also worth flagging that the 67B run apparently uses the Marin team’s own Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method MoE load-balancing (per their blog, validated at 32B-A5B) — so this run is a coupled validation of two Marin-shipped scaling components in production.

See also

Section titled “See also”