Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

TL;DR

A systematic empirical study of on-policy distillation (OPD) — the recipe Qwen3 / MiMo / GLM-5 / Thinking Machines all use to bolt dense teacher-token rewards onto student rollouts. The paper identifies two necessary conditions for OPD to actually help (compatible thinking patterns + genuinely new teacher knowledge), instruments the failure modes at token-level (overlap ratio, overlap-token advantage, entropy gap), and shows a stronger teacher can completely fail to teach a weaker student while a same-family RL-trained teacher succeeds. Proposes two recovery recipes — off-policy SFT cold start and teacher-aligned prompt selection — and ends with a warning that the “free lunch” dense per-token reward degrades systematically with trajectory depth, raising doubt that current OPD scales to long-horizon agentic reasoning.

Key claims

OPD requires thinking-pattern consistency: a Qwen3-4B-Base-GRPO teacher (matched base) outperforms Qwen3-4B-Non-thinking as a teacher for Qwen3-1.7B-Base, even though benchmark scores are comparable; the GRPO teacher has higher initial top-k overlap ratio with the student and the gap that opens early in training cannot be closed later [§3.1, Fig. 2].
OPD requires genuinely new teacher knowledge: same-pipeline 4B teachers yield limited gains, while RL-post-trained teachers from the same base produce substantially stronger transfer on both DeepSeek-R1-Distill and Qwen3 families [§3.2, Fig. 4].
Reverse-distillation experiment is the cleanest evidence: distilling JustRL-1.5B back into its pre-RL checkpoint (R1-Distill-1.5B) removes all RL gains, and using a 7B same-family teacher (R1-Distill-7B) produces an indistinguishable trajectory — implying scale alone does not give a teacher new transferable knowledge in OPD’s reverse-KL geometry [§3.3, Fig. 5].
Successful OPD has a consistent token-level signature: overlap ratio rises from ~72% to ~91%, entropy gap narrows, and the shared top-k set concentrates 97–99% of the combined probability mass [§4.1, Fig. 6].
Restricting the distillation loss to the overlap of student/teacher top-k matches full student-top-k performance; restricting to the symmetric difference is consistently weaker — i.e. the overlap region is the operative locus of OPD’s gradient, not just a correlate [§4.2, Fig. 7].
“Overlap optimization is self-reinforcing”: reverse-KL updates concentrate mass on shared high-probability tokens, pushing non-overlap tokens out of the student’s top-k and growing the overlap region as a virtuous cycle [§4.2].
Recovery recipe A — off-policy cold start: a warmup SFT phase on teacher-generated rollouts (200K from Qwen3-4B on OpenThoughts3-math) before OPD raises initial overlap ratio, shrinks the entropy gap, stabilizes training, and improves the final OPD ceiling — not just early dynamics [§5.1, Fig. 8].
Recovery recipe B — teacher-aligned prompt selection: matching both prompt template (JustRL’s \boxed{} format) and prompt content (using DAPO-Math-17K, the teacher’s own RL training set) sharpens alignment on high-probability tokens, though pure teacher-prompt content drops student entropy and benefits from mixing with OOD prompts [§5.2].
The dense token reward is not actually a free lunch: reward quality degrades systematically with trajectory depth, and OPD instability originates at later tokens before propagating backward — a structural problem for long-horizon/agentic distillation [§6].
Even failing teachers provide global reward correlated with rollout correctness — the failure is not signal-quality but local optimization geometry (a larger teacher induces a locally-flat reward landscape around the student policy) [§6].

Method

OPD computes the per-step reverse KL between teacher and student on prefixes the student sampled, decomposed token-wise as $-\sum_t \mathrm{KL}(\pi_s(\cdot|s_t) \,\|\, \pi_T(\cdot|s_t))$ . Three supervision granularities exist in the wild: sampled-token OPD (Monte Carlo estimator, the lightweight default used by Qwen3 / MiMo / Thinking Machines), full-vocabulary OPD (exact, expensive), and top-k OPD (subset approximation). The paper instruments three diagnostic metrics throughout training — overlap ratio (|S∩T|/k of student/teacher top-k sets), overlap-token advantage (within-intersection log-probability gap), and entropy gap (H(π_s) − H(π_T)) — and uses them to separate succeeding from failing runs.

Experiments cover four student/teacher pairs spanning Qwen3-1.7B(-Base) and DeepSeek-R1-Distill-1.5B/7B with both same-pipeline and RL-post-trained teachers, all trained with verl (v0.7.0) on DAPO-Math-17K or OpenThoughts3 math subsets. The decomposition ablation (overlap-only vs symmetric-difference-only vs full top-k) and the two recovery recipes (off-policy SFT cold start; teacher-aligned prompts) are evaluated under the same evaluation harness (AIME24/25 + AMC23, avg@16 at T=0.7, top-p=0.95). Code, SFT checkpoint, and the Qwen3-4B-Base-GRPO teacher are released.

Results

Cold-start recipe: starting OPD from Qwen3-1.7B-SFT (cold-start checkpoint) yields consistently better validation than starting from Qwen3-1.7B-Base, and the gap persists through training — the cold start raises the final ceiling, not just the warmup phase [§5.1, Fig. 8].
Overlap-only top-k OPD matches full student top-k OPD across AIME24, AIME25, AMC23; non-overlap-only OPD is consistently weaker on all three [§4.2, Fig. 7].
Reverse distillation: JustRL-1.5B distilled into R1-Distill-1.5B or R1-Distill-7B regresses to (almost exactly) R1-Distill-1.5B’s pre-RL performance level — a 7B teacher gives the same trajectory as a 1.5B teacher despite outscoring the 1.5B student on benchmarks [§3.3, Fig. 5].
Overlap-ratio dynamics: successful runs climb steadily from ~72% to >91%; failing runs stagnate near initial value with persistent entropy mismatch [§4.1, Fig. 6 bottom].
Probability mass concentration: the shared top-k set holds 97–99% of combined teacher+student probability mass throughout successful runs (Appendix B.1), so overlap-ratio gains reflect alignment on the probabilistically dominant region, not a set-level coincidence.

Why it’s interesting

This is the first filed paper to open up the black box of on-policy distillation — the recipe is now standard in flagship post-training pipelines but treated as a folk technique. Filed papers in Reasoning RL (Nemotron-Cascade 2’s MOPD, MiroThinker’s SFT→DPO→GRPO, HuDA, Training-Free GRPO) all assume OPD or a GRPO-family RL works; this paper supplies the failure modes, diagnostic metrics, and rescue recipes the others have been quietly battle-testing. The two-condition framing (compatible thinking patterns + new knowledge) gives Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation a precise vocabulary for why their “Multi-Domain On-Policy Distillation” works — they’re distilling from the strongest intermediate teacher per domain, which by construction has new knowledge beyond the student’s current capabilities, exactly the condition this paper identifies.

The reverse-distillation result also sharpens Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling‘s “smaller-weaker-yet-better” finding: a same-family weaker teacher matches a 4.7× larger same-family teacher in OPD because neither has new knowledge beyond the student. That puts compute-optimal teacher choice and OPD-condition checks on the same axis — pick a teacher that introduces capability the student hasn’t seen, regardless of benchmark score gap.

The §6 warning about long-horizon scaling is the part Luma should track most carefully: if dense per-token reward quality degrades with trajectory depth, OPD’s role in agentic / long-CoT post-training is more constrained than the recent Qwen3 / Nemotron-Cascade / MiroThinker pipelines imply, and the field may need either trajectory-depth-aware reward weighting or a return to outcome-reward RL for the long-horizon regime.