MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training

ListenOne Student, Many Teachers: MOPD and Capability Fusion11:38

Show notes

MOPD is Xiaomi MiMo's recipe for combining several single-domain RL experts into one model — by distilling all of them into a shared student on the student's own rollouts, routing each prompt to its matching teacher. We walk through why fusing in policy space beats weight-merging or mixed RL, why a bigger teacher can actually break training, and what the parallel per-domain workflow means for anyone building multi-capability models. It's the load-bearing post-training stage behind the 309-billion-parameter MiMo-V2-Flash. **Sources** - MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

MOPD is Xiaomi MiMo’s multi-teacher on-policy distillation recipe: run per-domain RL independently to produce a set of specialised domain teachers, then distill all of them into a single student on the student’s own rollouts, with each prompt routed to the matching domain teacher. It fuses capabilities in policy space (per-prompt routing to a live teacher) rather than in weight space (Param-Merge) or a static dataset (Off-Policy Finetune), avoids exposure bias by construction, and gives dense token-level supervision. On Qwen3-30B-A3B it leads Mix-RL, Cascade RL, Off-Policy Finetune, and Param-Merge on a three-domain (Math / IF / SWE) capability-integration benchmark, and it is the load-bearing post-training stage for the 309B MiMo-V2-Flash. The paper also introduces two dispatch-side losses (clipped policy-gradient reverse-KL and a bias-corrected top-k variant), argues same-origin teachers are structurally necessary for stability, and shows multi-round iterative refinement continues to close headroom.

Key claims

Existing capability-integration paradigms each trade off one of {dense signal, on-policy, parallelisable}; MOPD is the only method that has all three [§1, Table 1].
On Qwen3-30B-A3B (three domains: Math / IF / SWE), MOPD reaches a normalised score of 0.9373 vs. next-best Mix-RL at 0.8818, and its per-domain normalised range is only 0.09 — the most uniform capability recovery of any method tested [§4.2, Table 2].
MOPD is markedly more sample-efficient than Mix-RL: MOPD reaches the teacher-level plateau on IF within 10K IF samples and on SWE within 20K SWE samples, while Mix-RL needs its full 30-60K sample budget per domain [§4.2, Fig. 1].
Cascade RL suffers cross-domain interference: the first-trained IF domain closes 87% of headroom but Math (second stage) closes only 34%, and Math accuracy actively degrades during the subsequent SWE stage [§4.2].
Param-Merge is unreliable: linear averaging scores 0.328 normalised, Task Arithmetic recovers to 0.857 but per-domain profile varies widely (exceeds teacher on IF, closes only 66% on Math) [§4.2, Table 2].
Deployed on the 309B/15B-active MiMo-V2-Flash across five domains (Math / Code / IF / SWE / Tool Use), MOPD matches or exceeds the corresponding teacher on 5 of 7 benchmarks; the two regressions (IFBench, SWE-Bench Verified) are modest [§4.3, Table 3].
MOPD-PG (policy-gradient reverse-KL with two-sided-clipped advantage $\log \pi_T - \log \pi_s$ ) and MOPD-Top-k (top-k reverse KL with a bias-correction term for truncation) perform comparably under same-origin teachers, with matching training-loss and entropy trajectories [§4.4.1, Table 5].
Same-origin teachers are structurally required: replacing the same-base Math RL teacher (Qwen3-30B-A3B) with a stronger but distributionally different Qwen3-235B-A22B causes Math accuracy to degrade under PG loss and catastrophically diverge around step 18 under top-k loss; initial per-token KL is ~2× larger in the mismatched case [§4.4.2].
Multi-round MOPD works: initialising round-2 per-domain RL from the round-1 student yields stronger teachers (normalised score 1.030 vs 0.937 after round 1) and a round-2 student that continues to close headroom (0.986), confirming that a single MOPD pass leaves recoverable per-domain capability behind [§4.4.3, Table 4].
Teacher prefill is deployed as a stand-alone service outside the RL trainer and issued asynchronously per rollout; because prefill overlaps with other-sequence sampling, teacher cost is essentially hidden behind student sampling time [§3.3].
The top-k loss adds a bias-correction term beyond the standard reverse KL because naive top-k-truncated reverse KL is not minimised at $\pi_s = \pi_T$ on the top-k tokens without it [§3.2.2].

Method

MOPD runs three sequential stages. Stage 1 (SFT) produces a broad-coverage checkpoint that seeds both the domain teachers and the final student. Stage 2 (per-domain RL) trains each domain expert independently from the Stage-1 checkpoint using whatever RL recipe suits the domain (verifiable-answer RL for Math, sandboxed agent RL for SWE, rubric-based RL for IF); these runs are fully parallelisable and can each iterate their own hyperparameters. Stage 3 (MOPD) initialises a student from the same Stage-1 checkpoint, holds the Stage-2 teachers frozen, and at each step: (1) samples a prompt batch; (2) student rollouts trajectories and logs per-token distributions; (3) each trajectory is dispatched by domain tag to its teacher, which prefills the trajectory to obtain teacher per-token distributions; (4) student is updated by minimising per-token reverse KL to the dispatched teacher along the trajectory.

Two loss instantiations are given. Policy-gradient form: apply the standard policy-gradient identity to reverse KL, giving a per-token advantage $A_t = \log \pi_T(y_t|\cdot) - \log \pi_s(y_t|\cdot)$ ; two-side-clip $A_t \in [-c, c]$ for stability. Drops into existing PPO/GRPO stacks with only the advantage-computation swapped. Top-k form: distill over the teacher’s top-k tokens with an added bias-correction term so the loss is minimised at student = teacher on the top-k tokens (naive top-k reverse KL lacks this fixed point). Top-k keeps the teacher prefill payload small enough to be treated like a reward signal on the wire.

Infrastructure-wise, teacher prefill is deployed as a stand-alone service outside the RL trainer, and prefill requests are issued asynchronously per finished student rollout — the same architectural pattern as reward-model computation in RLHF pipelines — so teacher cost overlaps with other sequences’ sampling and does not appear in wall-clock cost.

Results

Qwen3-30B-A3B, three domains: MOPD scores 0.9373 normalised, vs Mix-RL 0.8818, Off-Policy Finetune 0.8241, Cascade RL 0.7752, Param-Merge (Task Arith) 0.8574 [Table 2]. Per-benchmark: AIME25 51.46, AIME26 65.31 (Math); IFBench 77.89, IFEval 93.84 (IF); SWE-bench Verified 50.40 vs teacher 51.20 (SWE).
MiMo-V2-Flash (309B/15B active): MOPD ≥ teacher on AIME25 (94.1 vs 93.9), HMMT25 (84.4 vs 82.6), LCB (83.2 vs 82.6), τ-Bench (80.3 vs 79.6), τ-Telecom (95.3 vs 95.0); slight regressions on IFBench (66.7 vs 68.9) and SWE-Bench Verified (73.4 vs 74.2) [Table 3].
Loss-form ablation on Qwen3-30B-A3B: PG and top-k reach normalised 0.9373 vs 0.9182 (near-identical); training curves overlap and entropy stays stable around 0.6 [§4.4.1].
Same-origin ablation: swapping in Qwen3-235B-A22B as Math teacher (larger, stronger) drops MOPD’s Math score under both loss forms; PG converges to a low-entropy narrow policy, top-k diverges catastrophically at step ~18 [§4.4.2].
Multi-round: round-2 RL teacher hits normalised 1.030 (above the round-1 teacher baseline of 1.0), and round-2 MOPD student reaches 0.986 [§4.4.3, Table 4].

Why it’s interesting

MOPD is now the load-bearing post-training stage in Fuli Luo on MiMo-V2-Flash engineering — Hybrid SWA, MTP, MOPD post-training‘s MiMo-V2-Flash writeup and shows up as the recipe underneath the Nemotron-3 Ultra “Multi-Domain On-Policy Distillation” stage in NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 and Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation — this is the actual paper. Filing it closes a hole in the wiki: Reasoning RL has been accumulating MOPD-branded datapoints for months without a canonical source. The two structural claims a Luma reader should carry forward are the same-origin teacher requirement and the modular parallel-development workflow (§5): the empirical stability failure with Qwen3-235B-A22B as Math teacher gives a mechanistic reason why bigger-teacher-is-better SFT distillation intuitions don’t carry over to OPD, and the workflow decoupling explains why every recent 200B+ open model’s post-training report describes fully parallel per-domain teams.

Complements Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe‘s failure-mode taxonomy from the other side: Rethinking OPD identifies that a stronger teacher with poor distributional overlap teaches the student nothing (condition 2, the thinking-pattern overlap condition); MOPD’s §4.4.2 ablation is a direct empirical confirmation on the multi-domain integration axis — replacing a same-origin 30B Math teacher with a 235B one collapses training, and the paper explicitly attributes it to initial per-token KL being ~2× larger. Chinmay’s Slack-linked tweet [(Reasoning RL recent contribution)] highlights the same two takeaways (“top-k performs similarly to PG on sampled token, and same-origin teachers matter a lot”). Also directly relevant to On-policy distillation isn't a free lunch — four failure modes of OPD/OPSD in current post-training recipes (Bhavin Jawade)‘s four-failure-mode synthesis: MOPD’s same-origin constraint is Jawade’s mode 2 stated positively.