Olmo 3: Charting a path through the model flow to lead open-source AI

TL;DR

Olmo 3 is Ai2’s open-source 7B/32B foundation-model family released with the entire “model flow” — every base, mid-trained, long-context, SFT, DPO, and RLVR checkpoint, plus all training data, tooling, and the tracing layer (OlmoTrace) that links outputs back to specific training documents. The release is structured as three forkable post-training paths from a shared base: Olmo 3-Think (reasoning), Olmo 3-Instruct (chat / tool use), and Olmo 3-RL Zero (RLVR research). On benchmarks Olmo 3-Base (32B) is positioned as the strongest fully open base model and competitive with Qwen 2.5 32B / Gemma 3 27B / Llama 3.1 70B open-weights baselines, while Olmo 3-Think (32B) is framed as the strongest fully open thinking model and within ~2 points of Qwen 3 32B / DeepSeek R1 Distill 32B on multiple reasoning benchmarks — trained on roughly 6× fewer tokens than Qwen 3 32B.

Key claims

Olmo 3 ships the full model flow as a first-class release artifact — every dataset, checkpoint, and dependency for pre-training, mid-training, long-context extension, SFT, DPO, and RLVR — rather than only the final weights [§Olmo 3 architecture and training stages; §Transparency at the core].
The base model is trained in three stages: a large-scale broad-coverage pre-train, a mid-training phase concentrated on harder math / code / reading-comprehension distributions, and a final long-context extension stage [§The Olmo 3 architecture and training stages].
Olmo 3-Base (32B) outperforms other fully open base models (Marin 32B, Apertus 70B) in the post’s evaluation suite and is competitive with Qwen 2.5 32B, Gemma 3 27B, and Llama 3.1 70B on coding, reading comprehension, math, and long-context retrieval (RULER) [§Strong performance across the board, Olmo 3-Base benchmark table].
Olmo 3-Think (32B) wins or sits within ~2 points of the best open-weight model on MATH, OMEGA, BigBenchHard, HumanEvalPlus, PopQA, and IFEval; ties Qwen 3 VL 32B Thinking on OMEGA; trained on roughly 6× fewer tokens than Qwen 3 32B [§Strong performance across the board, Olmo 3-Think benchmark table].
Pre-training corpus is Dolma 3: a ~9.3T-token pool drawn from web pages, science PDFs processed with olmOCR, code, math problem/solution pairs, and encyclopedic text; the actual mix used (Dolma 3 Mix) is 5.9T tokens with elevated code/math proportions and explicit deduplication + quality filtering + decontamination [§Expanded training data].
Mid-training uses Dolma 3 Dolmino: 100B training tokens sampled from a 2.2T-token pool of math, science, code, instruction-following, and reading-comprehension data — including reasoning traces that enable RL directly on the base model [§Expanded training data].
Long-context extension uses Dolma 3 Longmino: ~50B training tokens drawn from a 639B-token pool of long documents combined with mid-training data; the base maintains quality up to ~65K-token contexts [§Expanded training data; §Olmo 3-Base].
Post-training data is bundled as Dolci with separate mixes per stage: SFT (reasoning + tool-use + chat), DPO (contrastive preferences), and RLVR (hard math / code / instruction-following / chat prompts) [§Expanded training data].
Olmo 3.1 32B Think (a continuation of the original RL run for 21 days × 224 GPUs with extra Dolci-Think-RL epochs) adds +5 on AIME, +4 on ZebraLogic, +4 on IFEval, +20 on IFBench over the original 32B Think release [§Olmo 3.1 update preface].
Engineering: SFT throughput improved 8× by moving from the flexible Open Instruct codebase to the efficiency-oriented Olmo Core pretraining codebase; RL throughput improved 4× via in-flight weight updates, continuous batching, and threading improvements, taking MBU from 12.9% (Olmo 2 baseline) to 43.21% (Olmo 3 with in-flight updates) [§Efficient training stack, MBU table].
Pre-training cluster: up to 1,024 H100 GPUs at 7.7K tokens/device/sec for Olmo 3-Base 7B; mid-training on 128 H100s; post-training on 256 H100s [§Efficient training stack].
OlmoTrace links generated tokens back to specific training documents in real time, exposing a per-output dependency on the released training data [§Transparency at the core].

Method

Olmo 3 is a decoder-only transformer family at 7B and 32B parameters. The post does not break out architectural details; the meat is the flow: a three-stage pretraining pipeline (broad pretrain on Dolma 3 Mix → mid-training on Dolma 3 Dolmino → long-context extension on Dolma 3 Longmino) followed by three forkable post-training paths from the shared base, each running the same SFT → DPO → RLVR recipe over different Dolci mix subsets. Olmo 3-Think uses reasoning-tuned versions of all three stages; Olmo 3-Instruct uses chat/tool-use-tuned versions targeting shorter outputs and lower inference cost; Olmo 3-RL Zero applies RLVR directly to the base (skipping SFT/DPO) across four domain-focused checkpoints (math, code, instruction-following, chat) to enable clean RL-algorithm benchmarking.

Each intermediate stage — base, mid-trained, long-context, post-trained — is released as a checkpoint, with the explicit goal that downstream teams fork the flow at whatever stage suits their data and compute (e.g. swap in domain-specific data during mid-training, attach a custom RL objective at the RLVR stage). Training data at every stage is released without license restrictions, and the data-processing tooling (decontamination, tokenization, deduplication) is open-sourced as part of the same release.

Results

Olmo 3-Base 32B vs other fully open base models: wins on HumanEval (66.5 vs Marin 52.3, Apertus 32.5), DROP Gen2MC MC (87.2 vs Marin 71.0), SQuAD Gen2MC MC (98.2 vs Marin 97.6), DROP (81.0 vs Marin 76.5), SQuAD (94.5 vs Marin 94.4), CoQA (74.1 vs Marin 70.7), and is within 2 points of best on GSM8k, SciQ MC, CoQA Gen2MC MC, Winogrande RC, and Basic Skills [§benchmark table].
Olmo 3-Base 32B vs open-weight baselines: best of all listed models on HumanEval (66.5 vs Qwen 2.5 65.6, Gemma 3 27B 62.1, Llama 3.1 70B 57.4), DROP, SQuAD, CoQA [§benchmark table].
Olmo 3-Think (32B) vs Qwen 3 32B / Qwen 3 VL 32B Thinking / Gemma 3 27B Instruct / DeepSeek R1 Distill 32B: best on IFEval (89.0); within 2 of best on MATH (96.1), OMEGA (50.8, tied), BigBenchHard (89.8), HumanEvalPlus (91.4), PopQA (31.9) [§Olmo 3-Think benchmark table].
Olmo 3-Instruct (7B) vs Qwen 3 8B (no reasoning) / Qwen 3 VL 8B Instruct / Apertus 8B / Granite 3.3 8B: best on Safety (87.3) and within 2 of best on IFBench (32.3), LitQA2 (38.2), with a large AIME 2024 lead vs the no-reasoning Qwen 3 8B (44.3 vs 26.2) [§Olmo 3-Instruct benchmark table].
Engineering throughput: Olmo 2 baseline post-training does 881 tokens/sec at 0.30 MFU / 12.9 MBU; Olmo 3 with continuous batching + better threading + in-flight updates reaches 2,949 tokens/sec at 1.01 MFU / 43.21 MBU — a 3.3× throughput / 3.4× MBU improvement on the same axis [§Efficient training stack table].

Why it’s interesting

The release is the cleanest worked example so far of the model-flow-as-artifact pattern the Open foundation-model releases page has been tracking. Action100M (Action100M: A Large-scale Video Action Dataset) ships a dataset-as-foundation; the Smol Training Playbook (Smol Training Playbook (GPU MODE talk on SmolLM3)) ships process documentation; Olmo 3 ships every intermediate checkpoint plus all data plus the tracing layer — i.e. the entire reproducible trajectory, not just the endpoint. The Olmo Think / Instruct / RL Zero paths are also a concrete realization of the “swap in your own data at mid-training or RL” hook that the SmolLM3 playbook argues is the dominant lever for ROI.

The pre-train → mid-train → long-context → SFT → DPO → RLVR pipeline is exactly the stage-decomposition that On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models formalizes and ablates at 100M scale. That paper’s headline result — that mid-training on edge-of-competence data is more compute-efficient than pure RL under a fixed budget — would predict that Olmo 3’s Dolma 3 Dolmino mid-training stage (math/code/reasoning at the hardness boundary) is doing more of the headline reasoning lift than the subsequent RLVR pass. Whether the Olmo team’s full technical report confirms or refutes this on a 32B-scale system is a load-bearing question for the Reasoning RL page, whose current TL;DR treats RL as the primary lever.