Embarrassingly Simple Self-Distillation Improves Code Generation

TL;DR

Apple shows that a coding LLM can be post-trained on its own unfiltered, unverified samples and improve substantially: Qwen3-30B-A3B-Instruct climbs from 42.4% → 55.3% pass@1 on LiveCodeBench v6 with no teacher, verifier, executor, reward model, or RL. One temperature-shifted+truncated sample per prompt over 10K rSTARcoder problems is enough; standard SFT does the rest. The mechanism, supported by toy simulation, real-model analysis, and a Rényi-entropy decomposition of the SSD loss, is that training on temperature-shifted samples asymmetrically reshapes the model’s token distributions — compressing distractor tails at “lock” positions (syntax-forced) while preserving diversity at “fork” positions (genuinely ambiguous) — relaxing a precision–exploration conflict that any single global decoding temperature cannot resolve.

Key claims

SSD lifts Qwen3-30B-Instruct from 42.4% → 55.3% pass@1 on LiveCodeBench v6 (+12.9 pp, +30.4% relative), with consistent positive gains across Llama-3.1-8B, Qwen3-4B-Instruct/Thinking, and Qwen3-30B-Thinking (Instruct variants benefit more) [§3.2, Table 2].
Gains concentrate on hard problems: for Qwen3-30B-Instruct on LCB v6, pass@1 improves +6.5 pp (easy), +14.2 pp (medium), +15.3 pp (hard); pass@5 improves +6.6 / +19.6 / +23.0 pp [§3.2, Table 2].
SSD does not collapse diversity — pass@5 gains exceed pass@1 gains across the Qwen family (e.g. +18.1 pp pass@5 vs +12.9 pp pass@1 for Qwen3-30B-Instruct on LCB v6) [§3.2].
A single sample per prompt over 10K competitive-programming problems suffices; no execution, no test cases, no correctness filtering, only minimal syntactic filtering (drop empty/single-line responses) [§3.1].
Global temperature/truncation tuning on the base model cannot replicate SSD: base-model temperature sweeps span only 1.5–3.0 pp on pass@1, and SSD beats the best-tuned base by +11.8 pp (Qwen3-30B-Instruct) and by +13.3 pp on hard pass@1 / +19.4 pp on hard pass@5 [§3.3, Fig. 2].
The induced SSD loss decomposes into three terms — support compression (drives mass out of the discarded tail), within-support reshaping (a Rényi entropy of order $1/T$ over the retained set), and head alignment with the tempered base — explaining lock-vs-fork asymmetric behavior [§4.3, Eq. 4, App. B.1–B.3].
Training-time and evaluation-time temperatures compose: under no truncation, performance is governed by an effective temperature $T_{\text{eff}} = T_{\text{train}} \cdot T_{\text{eval}}$ with a quadratic peak near a fixed value (~1.4 for Qwen3-30B-Instruct), confirmed by a diagonal-band pattern in the hyperparameter grid [§3.4, §B.3, Fig. 3a].
Stress test: even when sampling at $T_{\text{train}}=2.0$ with truncation disabled produces near-gibberish training data (~17% have no extractable code, many devolve into multilingual gibberish), SSD still improves Qwen3-30B-Instruct to 48.1% pass@1 / 64.0% pass@5 (+5.7 / +10.5 pp), with the gain depending critically on evaluation-time truncation to clean up tails the training stage left behind [§4.4, Fig. 7, Fig. 15b].
The SSD-trained model degrades only mildly out of domain (math reasoning, general code, code understanding) for the 30B models, despite training only on competitive-programming prompts [§3.2, §C.3].

Method

For a frozen base LLM $\pi_\theta$ and a set of prompts $\mathcal{P}$ (10K de-duplicated rSTARcoder problems), sample one candidate solution per prompt at $(T_{\text{train}}, \tau_{\text{train}})$ — a chosen temperature with top- $p$ /top- $k$ truncation (Table 4 lists per-model defaults). Apply only syntactic filtering (drop empty/single-line responses); no execution, no test cases, no correctness check. SFT the model on these raw outputs with standard cross-entropy for 2,500 iterations (instruct models) or 300 iterations (thinking models), AdamW, cosine LR (peak $10^{-5}$ ), global batch 32, sequence length 65,536, on 8×B200 GPUs (EP8 for MoE). Decode at evaluation time with $(T_{\text{eval}}, \tau_{\text{eval}})$ .

The theoretical claim driving the design (§4.3, Eq. 4) is that fitting samples from $\pi_\theta^{T_{\text{train}}, \tau_{\text{train}}}$ — rather than the base distribution — induces a three-term loss whose first term compresses support to the set $S$ surviving training-time truncation. At a lock (sharp peak + long distractor tail), $|S|$ is small so compression dominates and the head becomes harder to dislodge; at a fork (mass spread across plausible continuations), $|S|$ is large so the within-support Rényi reshaping term takes over, flattening the head without reopening the discarded tail. The result is asymmetric: locks get sharper, forks stay diverse — which is exactly what no single global decoding temperature can achieve simultaneously.

Results

Headline: Qwen3-30B-A3B-Instruct: LCB v6 pass@1 42.4 → 55.3 (+12.9 pp); pass@5 even larger; hard-problem pass@5 31.1 → 54.1 (+23.0 pp) [§3.2, Table 2]. LCB v5 (374-problem secondary): 45.8 → 54.3 pass@1 (+8.5), with +12.0 pp on pass@5 [§3.2]. All five tested base models (Llama-3.1-8B-Instruct, Qwen3-4B-Instruct/Thinking, Qwen3-30B-Instruct/Thinking) improve under SSD; instruct variants benefit more than thinking variants (+12.9 pp vs +2.1 pp at 30B), consistent with thinking models already having a tighter operating point.

Best-tuned-base-model gap: SSD beats the best decoding configuration of the base model by +11.8 / +5.8 / +2.2 / +1.1 pp across the Qwen family on LCB v6, with the gap widening on hard problems and at pass@5 [§3.3]. Hyperparameter grid: in the no-truncation slice, configurations collapse onto a single curve in $T_{\text{eff}} = T_{\text{train}} \cdot T_{\text{eval}}$ with a quadratic peak; with non-trivial training-time truncation, the best Qwen3-30B-Instruct setting ( $T_{\text{train}}=1.0$ , $\tau_{\text{train}}=$ top- $p$ truncation, $T_{\text{eval}}=0.7$ ) reaches 49.7% pass@1 [§3.4, Fig. 3a]. Bad-data stress test: $T_{\text{train}}=2.0$ , no truncation, ~17% no-code outputs → 48.1% / 64.0% pass@1/pass@5 [§4.4].

Why it’s interesting

This is the cleanest existing isolation of a claim the Synthetic Training Data page has been edging toward: the verification gate is doing less work than it appears to. Across that page’s filed papers — InfTool’s gated rewards, Video-Thinker’s hindsight curation, Action100M’s Self-Refine — the curation step has been framed as the load-bearing piece. SSD removes it entirely (no execution, no correctness check, no reward model) and still gets +30% relative on LiveCodeBench. The mechanism the paper offers — temperature-shifted samples implicitly reshape the model’s distribution in a context-dependent way that decoding alone cannot — reframes self-distillation as a decoding–training co-design problem rather than a data-quality problem.

It also forms a sharp contrast pair with Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling, which is the other major filed datapoint on “self-improvement from unverified-ish data.” Smaller-Weaker-Yet-Better filters by final-answer correctness and argues that coverage × diversity at compute parity beats teacher quality; SSD removes the filter entirely and argues that the training-time temperature/truncation alone is the lever — i.e. it answers Smaller-Weaker-Yet-Better’s open question (“how much of the gain is the gate?”) with “in code, not as much as you’d think, and the temperature is doing the heavy lifting.” For Reasoning RL it is a strong negative datapoint: at least on LCB-style competitive coding, a simple SFT-on-self-samples recipe captures most of what RL/verifier-based pipelines deliver, with none of the operational complexity. Whether this holds on agentic / multi-turn / tool-using tasks is the obvious follow-up.