Skip to content
Bud's Wiki

On-policy distillation isn't a free lunch — four failure modes of OPD/OPSD in current post-training recipes (Bhavin Jawade)

TL;DR

Section titled “TL;DR”

Bhavin Jawade’s own synthesis across four recent papers on why on-policy distillation (OPD) — the dense-reward-on-student-rollouts recipe now standard in DeepSeek v4, MiMo, Nemotron-Cascade-2 (MOPD), and GLM 5.x (on-policy cross-stage self-distillation) — has structural failure modes that no per-token reweighting can fix. Four modes named: (1) early mistakes are uncorrectable on the student’s own rollout (TRD); (2) stronger teachers can be worse when their token distribution has low overlap with the student (Rethinking OPD); (3) privileged-information-conditioned OPSD collapses when the privileged info is instance-specific (Many Faces of OPD); (4) thinking collapse — dense answer-conditioned supervision suppresses the student’s own deliberation tokens at high-entropy decision forks (Diagnosing Thinking Collapse). Together the four papers sketch a diagnostic vocabulary the field is only beginning to converge on.

Key claims

Section titled “Key claims”
  • Early mistakes in student rollouts are structurally uncorrectable under per-token KL: the gradient obtained from token-level KL on the student’s own rollout agrees with the ideal corrective gradient at exactly one point — the token where the student first diverged — and disagrees everywhere after; reweighting or clipping only rescales magnitudes of terms that point the wrong direction (TRD, cited from thread §1).
  • TRD’s proposed fix is to distill along a teacher-refined trajectory rather than the raw student rollout, restoring a target the student can actually follow (TRD, cited from thread §1).
  • A stronger teacher can teach nothing to a student if their next-token distributions barely overlap at states the student visits; benchmark score gap is a poor predictor of transferable OPD signal (Rethinking OPD, cited from thread §2).
  • The empirical signature of successful OPD is early top-k thinking-pattern overlap: in runs that work, shared top-k tokens carry 97–99% of the probability mass and the overlap ratio climbs steadily during training; low-overlap runs never recover (Rethinking OPD, cited from thread §2).
  • Practical rule: pick teachers by distributional closeness to the student, not by leaderboard rank (Rethinking OPD, cited from thread §2).
  • OPSD with instance-specific privileged information (e.g. gold answers) systematically fails: the student cannot recover per-instance teacher reasoning and instead learns a single averaged answer-free policy that is too generic to solve any particular problem (Many Faces of OPD, cited from thread §3).
  • The signature of that OPSD failure is initial gains followed by collapse: rollouts grow long, fill with hedging tokens, and accuracy craters toward zero; OPSD only works when the privileged information is a shared rule across all instances (e.g. a system prompt or alignment preference) (Many Faces of OPD, cited from thread §3).
  • Thinking collapse — the answer-conditioned teacher has no reason to hesitate/backtrack, so its per-token targets quietly suppress the student’s deliberation tokens (wait, maybe, alternatively); frequency of these tokens erodes over training and native reasoning behavior measurably degrades (Diagnosing Thinking Collapse, cited from thread §4).
  • The damage from thinking collapse concentrates locally at high-entropy decision forks where the student’s top-1 is an exploratory marker and the teacher’s top-1 is not, producing a strong wrong-direction gradient precisely at the branch points that carry reasoning (Diagnosing Thinking Collapse, cited from thread §4).

Method

Section titled “Method”

The tweet is a written synthesis, not a paper. Bhavin threads four independent papers — TRD (“Trajectory-Refined Distillation”), Rethinking OPD (Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe, filed), a “Many Faces of OPD” study on privileged-information-conditioned OPSD, and a “Diagnosing and Mitigating Thinking Collapse” study — into a single failure-mode taxonomy for the OPD/OPSD family of post-training methods that DeepSeek v4, MiMo, Nemotron-Cascade-2, and GLM 5.x now use as a load-bearing recipe stage.

Each of the four modes is described mechanistically (gradient direction, overlap ratio, hedging-token frequency, high-entropy fork damage) rather than as an abstract failure — the thread reads as a diagnostic checklist a practitioner would apply when an OPD run behaves badly.

Results

Section titled “Results”

Not applicable — no experiments in the tweet itself. Numbers cited from the referenced papers include the overlap ratio 97–99% probability mass concentration threshold (Rethinking OPD, already on file at Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe), and the qualitative collapse signature (long rollouts, hedging fill, accuracy → 0) from the OPSD-privileged-info paper.

Why it’s interesting

Section titled “Why it’s interesting”

Sharpens the OPD/OPSD failure-mode taxonomy that Reasoning RL has been accumulating: Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe contributed conditions 1–2 (thinking-pattern overlap + genuinely new teacher knowledge) plus the two-recovery-recipe playbook and the trajectory-depth degradation warning; this thread names two additional failure modes not covered by that paper — TRD’s early-divergence-is-uncorrectable proof, and thinking collapse’s local suppression of deliberation tokens at high-entropy forks — and points at two papers Luma should track alongside Rethinking OPD.

Directly relevant to On-Policy Distillation‘s framing of OPD as a compute-cheap RL substitute: if early-mistake gradients are structurally wrong and answer-conditioned teachers suppress deliberation, then the Thinking Machines recipe’s replication of Qwen3’s 74.4% AIME’24 at ~10× lower compute than RL may be more fragile than the “one-line modification of an RL training loop” framing suggests. The thread also adds specificity to the trajectory-depth-degradation warning in Rethinking OPD §6 — thinking collapse is the mechanism by which dense per-token reward degrades on long CoT, not just an empirical observation.

See also

Section titled “See also”