Marin Selected Experiment Reports

TL;DR

A semi-automatically generated, hand-curated index of pretraining experiments from the Marin project (Stanford CRFM’s open LLM development effort behind the Marin 8B / 32B models and the Levanter trainer). The page links each experiment to its GitHub issue, WandB report, and (where applicable) data-browser views, alongside one-line conclusions. Topics span optimizer/LR schedule choice, tokenizer comparison, MoE vs dense (OLMoE replication), Int8 / MuP / remat ablations, data-source ablations (Wikipedia/Arxiv/StackExchange markdownification), SFT recipes, and scaling laws. It’s notable less for any single result than as a living artifact of how an open foundation-model effort exposes its experimental process — including failed and negative results — to outside readers.

Key claims

Cosine LR schedule with a high peak LR is the best tested option; WSD “isn’t terrible” but does not match cosine in the Marin sweep [§Modeling, “Figuring out learning rate schedule!” / Issue #764].
Llama3’s tokenizer was selected as the best across the tested options for the Marin pretraining mix [§Modeling, “Pick Tokenizer type” / Issue #524].
Default z-loss is adopted because the comparison run “seems not harmful” — the conservative pick rather than a strict win [§Modeling, “Default z-loss?” / Issue #935].
Z-loss specifically mitigates exploding logits observed in deep cooldown of the Marin 8B “monumental-jellyfish” / “Spoonbill” runs [§Cooldowns, Issue #916].
OLMoE replication finds MoE outperforms a dense model of similar total size in both training and evaluation loss despite lower MFU [§Modeling, “Mixture of Experts” / Issue #1183].
Int8 training in Levanter is “much faster on the right hardware” but may give worse time-to-loss except early in training [§Training and Performance, Issue #620].
MuP was tested as a scaling-law tool and judged not worth it compared to Marin’s existing heuristic version of hyperparameter scaling [§Training and Performance, Issue #621].
Rematerialization-strategy tuning enables the 70B run to fit on fewer TPU slices but incurs a “substantial performance hit” [§Training and Performance, Issue #906].
Cooldown ablations on markdownified Wikipedia / Arxiv / StackExchange “no major improvement compared to control” — i.e. the high-quality format substitution did not help over the chosen baseline [§Data Experiments, Issues #845, #846, #847].
Mixture-of-formats training on Wikipedia + Arxiv showed “no major difference observed” — switching evaluation framework was the actionable outcome [§Data Experiments, Issue #818].
“High quality many epochs vs. low quality few epochs” concludes “there’s no data like more data” [§Data Experiments, Issue #636].
Some amount of HTML format preservation is helpful for Paloma loss when comparing web-extraction methods [§Data Experiments, “Compare HTML -> text methods” / Issue #246].
A Stack Exchange quality classifier produced better loss than Reddit-ELI5 / OpenHermes classifiers, but the entry is annotated as a loose end flagged for further investigation [§Data Filtering, Issue #596].
AdamW hyperparameter sweep concluded the (near-)optimal HP set “remains surprisingly stable across three settings” [§Scaling Laws, “Optimizer Scaling Law Part 1: AdamW” / Issue #725].

Method

The page is a flat hierarchy of experiment groups (Marin 8B Base, Marin 32B, Big Runs, Modeling, Training and Performance, Data Experiments, Supervised Fine Tuning, Scaling Laws, Baselines and Reproductions, Other Projects, Uncategorized). Each experiment is one bullet linking out to:

A GitHub issue on marin-community/marin capturing the experiment proposal and results discussion.
A WandB report (often on the marin-community or marin-community/marin-optimizer projects) with the loss curves.
Where applicable, a Stanford CRFM “Data Browser” view of the exact training data used.

Each bullet typically ends with a one-line “Conclusion:” — the page is explicit about which experiments produced negative or null results (e.g. the Wikipedia/Arxiv/StackExchange cooldown ablations all report “No major improvement compared to control”). The page itself notes it is “periodically updated by a script, and then curated by hand,” and reserves an “Uncategorized” section for experiments added by the script but not yet curated.

Results

The page is an index, not a results paper, so its “results” are the linked conclusions enumerated in Key claims above. Headline patterns visible across the list:

The pretraining-recipe consensus emerging from the Marin runs at 8B–70B scale: Llama3 tokenizer, cosine LR with high peak, z-loss on by default, AdamW with HPs that transfer across settings, MoE preferred over dense at parity, MuP rejected in favor of a heuristic alternative.
Multiple data-side ablations (Wikipedia / Arxiv / StackExchange cooldown, mixture-of-formats) producing null results — i.e. the high-quality markdownified data did not move the needle over the baseline mix in those tests.
Throughput vs quality trade-offs explicitly logged for both Int8 and remat strategies — adopted for capacity reasons, with the cost acknowledged in the conclusion line.

Why it’s interesting

This is the closest live-tracked counterpart at Luma’s interest scale to the On the Slow Death of Scaling thesis that scaling is no longer the only frontier and rigorous ablations matter: Marin is publishing the ablations themselves, including their null and negative results. It complements Smol Training Playbook (GPU MODE talk on SmolLM3) (HuggingFace’s SmolLM3 process-documentation playbook) — both are open-process artifacts shipped alongside open weights, but Marin’s reports page is a more granular per-experiment index rather than a single retrospective. The OLMoE-replication conclusion (MoE > dense at parity despite lower MFU) lines up with the production accounts in The MoE 101 Guide: From Theory to Production (Daria Soboleva, Cerebras) and MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production; the MuP-vs-heuristic verdict contrasts with Beyond MuP 3: Special Cases, Special Treatment (Embedding, LM Head, RMS Norm) and Spectral Condition for μP under Width-Depth Scaling, which argue MuP is workable once edge cases (embeddings, LM head, RMSNorm, width-depth scaling) are handled — useful to revisit whether Marin’s heuristic version is hitting the same operating point or a worse one.