SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training

TL;DR

A Qwen-team systematic study of how to compress a pretrained MoE model at pretraining scale via structured pruning + knowledge distillation, applied to Qwen3-Next-80A3B → 23A2B. Three findings: (1) across depth / width / expert axes, pruning a pretrained MoE beats training the target architecture from scratch at the same token budget; (2) different one-shot expert-compression methods converge to near-identical performance after enough continued pretraining, so the choice barely matters in the limit — but a simple partial-preservation expert-merging strategy outperforms on most benchmarks before that limit; (3) combining KD with the standard LM loss beats KD alone, especially on knowledge tasks, and progressive (gradual) pruning beats one-shot at matched total compute. Introduces multi-token-prediction (MTP) distillation as an additional consistent gain.

Key claims

Pruning a pretrained MoE as initialization outperforms training the target architecture from scratch under matched training-token budget, across all three compression axes (depth, width, expert count) [§3].
Different one-shot expert-compression methods (drop, merge variants) converge to similar final performance after large-scale continual pretraining — the choice is much less consequential than the field treats it as [§4].
A simple partial-preservation expert-merging strategy (keep some experts intact, merge others) improves downstream performance on most benchmarks over both pure dropping and pure merging [§4].
Knowledge distillation combined with the language-modeling loss beats KD alone, with the largest gap on knowledge-intensive tasks [§5].
Multi-token-prediction (MTP) distillation — extending the KD signal to predicted future tokens — yields consistent additional gains on top of standard KD [§5].
At equal training tokens, a progressive pruning schedule (gradually shrinking the model over training) outperforms one-shot compression, consistent with “gradual architecture transitions lead to better optimization trajectories” [§6].
Final result: Qwen3-Next-80A3B compressed to 23A2B retains competitive performance against same-budget baselines [§7, Abstract].

Method

The paper studies one fixed source — Qwen3-Next-80A3B (80B total / 3B active MoE) — and one fixed target shape — 23A2B (23B total / 2B active). Compression decomposes into three orthogonal axes that the paper sweeps independently: depth (drop layers), width (shrink hidden dim / heads), and experts (reduce expert count per MoE layer). For each axis it compares (a) train-from-scratch the target shape against (b) prune-then-continue-pretrain from the pretrained 80A3B teacher.

For expert compression specifically, several one-shot methods are compared — naive drop, activation-weighted drop, simple averaging merge, and a proposed partial-preservation merge that keeps a high-utilization core intact and merges only the low-utilization tail. On the loss side, two variants of distillation are evaluated: KD-only (KL on teacher logits) vs KD + LM loss (token-level CE on ground-truth tokens, added to the KD term). The authors then propose MTP distillation, which applies the KD signal to a multi-token prediction head — distilling not only the next-token distribution but also future-token distributions from the teacher. Finally, schedule is varied: one-shot compression up-front, vs progressive schedules that pruck successively during continued pretraining.

Results

Headline: a 23A2B model derived from Qwen3-Next-80A3B retains competitive performance, suggesting roughly a 3.5× total-parameter and 1.5× active-parameter compression at modest quality cost [§7, Abstract].
Pruning-as-init vs scratch is a consistent win across all three compression axes at matched compute [§3].
Partial-preservation expert merging is the proposed expert-compression default; it beats other one-shot variants on most downstream benchmarks before they converge [§4].
KD + LM > KD-only, with the gap concentrated on knowledge-heavy evaluations [§5].
MTP distillation stacks on top of KD + LM for an additional consistent gain [§5].
Progressive schedules beat one-shot at equal training tokens [§6].

(Specific numeric deltas are in the paper’s tables — abstract reports the qualitative direction of each finding.)

Why it’s interesting

This is the engineering report behind Qwen’s smaller MoE variants — the “how do you actually ship a 23A2B that’s worth shipping when you have an 80A3B sitting on a checkpoint” recipe. It’s directly adjacent to Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling, which argues distillation choices for small reasoners are non-obvious (WC > SE teacher under matched FLOPs); SlimQwen makes a related claim on the pretraining side — that the choice of one-shot expert-compression method is less consequential than the field assumes once enough continued pretraining is poured in. It also lands in the same Qwen-team thread as Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models — both papers are “details that determine whether an MoE pretraining run actually works at scale,” with Demons-in-the-Detail on routing-loss scope and SlimQwen on compression-stage choices. For Luma, the MTP-distillation result is the most portable finding outside MoE-LLM territory: a recipe for distilling multi-step predictive signal that could plausibly transfer to autoregressive video / world-model students with frame-wise or token-wise multi-future heads.