Distillation Scaling Laws

TL;DR

Apple’s 6-author scaling-law study of knowledge distillation for LLM pretraining. Fits a closed-form law predicting student cross-entropy as a function of student parameters $N_S$ , distillation tokens $D_S$ , and teacher cross-entropy $L_T$ (with $L_T$ itself a stand-in for teacher $N_T, D_T$ via Chinchilla). The law produces compute-optimal recipes for two regimes: (1) when a teacher already exists, (2) when the teacher must also be trained. Headline prescription: if many students will be distilled from the same teacher, or a teacher is already on hand, distillation beats supervised pretraining up to a student-size- dependent compute threshold; if only one student is needed and the teacher must also be trained, just do supervised pretraining.

Key claims

A distillation scaling law of the form $L_S = L_T + (A/N_S^\alpha + B/D_S^\beta - L_T)^\gamma$ (Apple’s preferred functional form) predicts student loss across the joint $(N_S, D_S, L_T)$ surface; large-scale fit covers students 143M–12.6B and teachers 198M–7.75B [§3, Eqs. 3.1–3.3].
Distillation outperforms supervised pretraining for the student up to a compute level that scales predictably with student size — i.e. there is a finite “distillation horizon” beyond which a same-budget supervised run wins [§4, Fig. 1].
When the teacher already exists (or one teacher will be amortized over many students), the compute-optimal recipe allocates more distillation tokens than a Chinchilla supervised run would for the same student size [§5.1].
When the teacher must be trained from scratch and only one student is needed, supervised pretraining of the student is preferable to “train teacher then distill” at iso-compute — distillation’s overhead is not amortized [§5.2].
A stronger teacher is not monotonically better for the student: above a teacher capacity that depends on student size and distillation budget, the student’s loss worsens (the “capacity gap” / “weak-teacher anomaly”), and the scaling law predicts the crossover [§4.2, Figs. 5–6].
The law isolates a teacher-size axis distinct from the student’s $N_S, D_S$ — same-loss teachers trained at different $(N_T, D_T)$ trade-offs distill equivalently, so $L_T$ alone (not the teacher’s parameter count) is the sufficient statistic for the teacher’s contribution [§3.2].
Empirically verified over 600+ distillation runs; published as ICML 2025 (PMLR v267:5977–6045) [Abstract, §1].

Method

The authors fit a parametric loss surface $L_S(N_S, D_S, L_T)$ on a large grid of teacher-student pairs (cross-product of 6 teacher sizes × 9 student sizes × multiple token budgets), using a Chinchilla-style power-law form augmented with a teacher term. The functional form collapses to standard Chinchilla as $L_T \to 0$ , and recovers a “pure distillation floor” as $D_S \to \infty$ . Compute is accounted for separately on the teacher side (training + inference to produce soft labels for $D_S$ tokens) and student side (forward + backward on $D_S$ tokens with a soft-label CE loss). Two operating regimes are derived analytically by minimizing $L_S$ subject to a compute constraint: (1) teacher cost fully amortized → student-only budget; (2) single-student budget including teacher training. The fit is validated by extrapolating to held-out $(N_S, D_S, L_T)$ points.

Results

Predicted vs. held-out student loss agrees within typical Chinchilla-fit error (sub-percent on log loss) across the 143M–12.6B student range [§3.3, Fig. 4].
Compute-optimal recipes for “teacher exists” prescribe student token budgets that grow super-linearly in $N_S$ relative to Chinchilla supervised optimal [§5.1, Table 1].
The weak-teacher / capacity-gap phenomenon is reproduced: for a 1.5B student trained on ≤100B distillation tokens, the optimal teacher size is in the 4–8B range, not the largest available teacher [Fig. 6].
ICML 2025 acceptance at PMLR v267, oral / expo presentation; 15 MB v1 (Feb 2025), v2 (Jul 2025).

Why it’s interesting

Slots straight into the wiki’s scaling-laws cluster — Hyperparameter scaling laws currently covers LR, MoE-FLOPs-ratio, batch-size-via-SDE, and loop-count, but the teacher-student axis was missing. The headline result that “if only one student needs distilling and teacher also needs training, just do supervised pretraining” sharpens the production trade-off that Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling hits from the other side (weaker-cheaper sampling models beat stronger-expensive ones for reasoning-data synthesis at compute parity) — both papers say “don’t pay for unnecessary teacher capacity,” but at different parts of the pipeline. Also relevant to Beyond MuP 3: Special Cases, Special Treatment (Embedding, LM Head, RMS Norm)‘s broader “predictable extrapolation from cheap runs” agenda: distillation is now governed by a Chinchilla-shaped law rather than folklore.