Synthetic Data Powering Pretraining (UC Berkeley EE 290/194-11 Lecture 11a)

TL;DR

A 50-slide guest lecture by Eric Tramel (NVIDIA NeMo Data Designer) for UC Berkeley’s Scalable AI course, framing synthetic data as the structurally-required answer to the LLM pretraining data wall — especially under MoE economics, which make the wall worse, not better. The deck’s central argument is that the “elaborate” Phi-style knowledge-construction approach plateaus in the hundreds of billions of tokens while the “simple” approach — process the whole web with cheap (3B) generators that rephrase existing content into diverse formats — scales to trillions and beats Phi-style at fixed compute. Adds two further claims load-bearing for the rest of the wiki: (i) seeding capabilities (code, reasoning) by front-loading them into pretraining compounds through SFT and RL (“pretraining determines the ceiling”), and (ii) at next-generation inference economics (B200 NVFP4, 2026) synthetic-token cost falls ~10× per GPU generation, making yet-untried data methodologies viable. Not a primary research artifact — a synthesis lecture — but a useful single source for the current consensus recipe and the citations behind it.

Key claims

Modern frontier models are now exclusively MoEs for inference economics, and MoE sparsity accelerates the data wall: MoE training tokens scale with total parameters (not active), each expert sees ~1/E of tokens under balanced routing, and practitioner experience puts the safe-repetition ceiling at ~2 epochs for MoE vs ~4 for dense [Slides 5–6, 9].
The industry data line is moving above Chinchilla-optimal: an empirical fit across released models gives D ≈ 614 · N_active (vs Chinchilla’s D = 20N) when active parameters are used, or D ≈ 36 · N_total when total MoE parameters are used [Slides 7–8].
The structural driver of overtraining is inference-cost amortization (Sardana et al. 2024 — trained once, served billions of times), not training-side optima; the Gadre et al. 2024 overtraining laws hold up to 640× Chinchilla with 0.7% relative error [Slide 9].
Model collapse (Shumailov et al. 2024, Nature) applies to recursive multi-generation training where each generation replaces the previous data — not to the practitioner setup of generating once and mixing with real; Gerstgrasser et al. 2024 show mixing with real data prevents collapse [Slides 14–15].
Knowledge-construction synthesis (Phi-1 / Phi-2 / Phi-4 / Cosmopedia) maxes out in the low hundreds of billions of tokens and the Microsoft line itself returned to web mixing in later Phi versions; the failure mode is distribution narrowing of “tacitly expected” capabilities [Slides 16–17].
Rephrasing — transform-don’t-replace — is the recipe that does scale: Maini, Seto et al. 2024 report ~3× training speedup (85B rephrased ≈ 300B real-only) from a Mistral-7B-rephrased C4; Nemotron-CC = 6.3T tokens with 1.9T synthetic; BeyondWeb and Qwen3 are at “trillions” of synthetic tokens [Slides 18–19].
Pure-synthetic data does not outperform Common Crawl alone — Kang et al. 2025 sweep 1,000+ LLM variants and report a ~1/3 rephrased + 2/3 web mixture as the sweet spot, with 5–10× convergence acceleration; synthetic is a multiplier on real data, not a replacement [Slide 20].
Generator size barely matters because the generator’s job is reformatting, not reasoning: Maini, Dorna et al. 2025 find a 3B generator produces data statistically indistinguishable from 8B at 2.7× lower cost; the lowest-benchmarks model (OLMo-2-7B) of four families produced the highest-quality synthetic data; four model families landed within 1pp of each other [Slide 21].
Concrete training-efficiency numbers (BeyondWeb @8B vs RedPajama at the 180B accuracy mark): generation FLOPs are ~13–33% of training FLOPs, total compute drops to 14–17% of the RedPajama baseline at parity [Slide 22, Table].
Generation cost is small and falling fast: Nemotron-CC (12B H100 FP8) costs $46–$ 87/B tokens, BeyondWeb (3B H100 FP8) $14–$ 35/B, projected 3B on B200 NVFP4 (2026) $0.75–$ 3.70/B, and next-gen (2028) $0.08–$ 0.40/B — roughly ~10× cheaper per GPU generation, ~100× by 2028 [Slides 23–24].
Diversity beats volume on the same content: Kimi K2 reports 10 rephrasings × 1 epoch ≥ 1 rephrasing × 10 epochs; Maini, Dorna et al. 2025 report single-strategy learning curves flatten during training while multi-strategy curves stay positive 50× beyond Chinchilla-optimal [Slides 26–27].
Code synthesis has a structural advantage: synthetic code can be verified by execution / linting; reported wins include Phi-4’s “instruction reversal” (code → problem), Nemotron-Code-v2 (340B tokens, 5 synthesis techniques), Qwen2.5-Coder Text-Code Grounding (+5.2pp HumanEval+MBPP), and SwallowCode (+17pp on HumanEval, with the headline “rewriting beats filtering”) [Slides 31–32].
Front-loading reasoning data into pretraining compounds downstream: Akter et al. 2025 train 8B / 1T tokens with 20% reasoning data (17% code, 56% math, 27% science) and report code pretraining +9pp, then SFT code 7.09 → 16.75% and RL LiveCodeBench 13.16 → 32.43% — gains amplify across post-training stages [Slide 33].
Front-loading is a curriculum design principle, captured by the lecture in the slogan “RL amplifies patterns already in pretraining ⇒ pretraining determines the ceiling” (Zhao et al. 2025, “Echo Chamber”) [Slide 35].
The deck frames the future as Memory → Logic → Simulation (Langlais 2026): rephrasing for knowledge retention is mature, reasoning primitives (Front-Loading, RLP) are emerging, and synthetic environments for agentic CPT are the frontier. Karpathy 2025 is invoked for “cognitive core” — shift compute from training dense fact-machines to lighter, adaptive cores [Slides 35–36].

Method

This is a lecture deck, not an experimental paper. The “method” is a literature synthesis structured in five sections: (1) the data wall and the scaling imperative under MoE economics; (2) why model collapse does not apply to practitioner setups; (3) what works and why in model-amplified data, organized around the elaborate-vs-simple axis and culminating in the Maini / Su / Maini-Dorna / Kang line on rephrasing; (4) capability seeding via synthetic code (verifiable) and reasoning data (front-loaded into pretraining); (5) a forward-looking framework (Memory / Logic / Simulation) and an inference-economics argument for why next-gen GPU generations unlock yet-untried data recipes.

The two cost models in the deck are worth recording: total compute C_total = 6 · P_train · D_train + 2 · P_gen · D_gen, with generation typically ~15% of training (slide 23), and a synthetic-token cost curve falling roughly 10× per GPU generation through H100 (FP8) → B200 (NVFP4 2026) → next-gen (NVFP4 2028) (slide 24).

Results

The deck does not present new experiments. Headline reported numbers (all attributed to cited papers):

Maini, Seto et al. 2024 (WRAP / Rephrased C4): ~3× training speedup; 85B rephrased ≈ 300B real-only.
Su et al. 2024 (Nemotron-CC): 6.3T-token refined dataset with 1.9T synthetic tokens.
Kang et al. 2025: 1,000+ LLM variants; ~1/3 rephrased + 2/3 web → 5–10× convergence speedup; pure synthetic does not beat Common Crawl alone.
Maini, Dorna et al. 2025 (BeyondWeb): 3B generators produce data statistically indistinguishable from 8B at 2.7× lower cost; multi-strategy curves stay positive 50× beyond Chinchilla-optimal; vs. RedPajama @180B accuracy, total compute drops to 14–17%.
Kimi K2 (2025): 10 rephrasings × 1 epoch ≥ 1 rephrasing × 10 epochs.
Akter et al. 2025 (Front-Loading Reasoning): 20% reasoning data in pretraining → +9pp code on the pretraining checkpoint; gains compound to SFT 7.09 → 16.75% and RL LiveCodeBench 13.16 → 32.43%.
Synthetic-token cost projection: H100/FP8 $14–$ 87/B → B200/NVFP4 (2026) $0.75–$ 3.70/B → next-gen (2028) $0.08–$ 0.40/B.

The deck does not report failure modes or negative results beyond the Phi-line plateau and pure-synthetic ≤ Common Crawl finding above.

Why it’s interesting

For Luma, this is the cleanest single-source summary of the open-source consensus on how to push past the data wall in 2026: small (3B) cheap generators rephrasing the entire web into diverse formats, mixed at ~1/3 with real data, with reasoning and code capabilities deliberately seeded into pretraining rather than added post-hoc. The MoE-makes-the-wall-worse framing pairs naturally with Scalable Training of Mixture-of-Experts Models with Megatron Core (the NVIDIA systems substrate for trillion-parameter MoE training) and Mixture of Experts (MoE), Visually Explained (conceptual primer) — together they’re a “MoE: economics → algorithms → systems → data” stack from a single vendor lineage. The Akter et al. front-loading result — pretraining-side reasoning compounds through SFT and RL — is the most direct empirical support yet for the “pretraining determines the ceiling” thesis in Reasoning RL‘s Echo Chamber citation, and it argues against the common framing that reasoning is a post-training capability.

The deck also complements Replaying pre-training data improves fine-tuning and Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training on the data-curation axis: those papers optimize real-data mixtures (CLIMB) and replay schedules (Kotha & Liang) at fixed corpora, while this lecture argues the binding constraint at trillion-token scale is whether you have enough novel tokens at all — making synthetic generation a precondition for the mixture-optimization machinery to keep paying. For Luma’s diffusion training, Terrance’s pointer note is the natural read-across: noise injection already acts like data augmentation, but rephrasing text prompts into diverse formats is an underused lever, and front-loading any capability the team cares about (e.g. reasoning traces, structured-control conditioning) into pretraining is the actionable curriculum recommendation.