Self-Distilled Agentic Reinforcement Learning

ListenThe Teacher That Knows Less: Gating Distillation Inside RL11:48

Show notes

This episode digs into **SDAR (Self-Distilled Agentic Reinforcement Learning)**, a multi-turn agent post-training recipe that adds on-policy self-distillation to GRPO as a *gated auxiliary objective* rather than a replacement. We unpack the two failure modes it names — compounding multi-turn drift and asymmetric trust in skill-conditioned teachers — and the detached sigmoid gate that fixes both, plus the surprising finding that the "privileged" teacher is on average *less* confident than the student. We close with what the recipe implies for anyone running a GRPO stack on image or video models. **Sources** - Self-Distilled Agentic Reinforcement Learning (Lu, Yao, Han, Wang, Wu, Gu, Cai, Lu, Xiao, Zhuang, Shen)

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

SDAR (Self-Distilled Agentic Reinforcement Learning) is a multi-turn agent post-training recipe that reconciles GRPO with On-Policy Self-Distillation (OPSD) by treating OPSD as a gated auxiliary objective on top of an unmodified GRPO loss. The paper names two failure modes of naive OPSD in multi-turn agents — compounding trajectory drift and asymmetric trust in privileged skill-conditioned guidance — and fixes both with a detached sigmoid gate over the teacher-student log-probability gap, which up-weights teacher-endorsed positive-gap tokens and softly attenuates negative-gap tokens on which the privileged teacher is not confident. Reports +9.4% on ALFWorld, +7.0% on Search-QA, and +10.2% on WebShop-Acc over pure GRPO across Qwen2.5-3B/7B and Qwen3-1.7B, while avoiding the catastrophic collapse of GRPO+OPSD on the 1.7B backbone.

Key claims

Naive OPSD applied on top of GRPO in multi-turn settings suffers “compounding multi-turn instability”: once the student drifts from the teacher-supported trajectory, per-turn KL surges and task performance catastrophically degrades — GRPO+OPSD collapses to 32.0 on ALFWorld-Qwen3-1.7B vs. 46.1 for pure GRPO [§1, Fig. 2 left, Table 1].
In skill-conditioned OPSD the privileged teacher is the same policy augmented with retrieved skills, not a stronger model, so its per-token guidance is asymmetric: positive-gap tokens (teacher assigns higher probability) are endorsements to trust, but negative-gap tokens may reflect bad skill retrieval, poor skill utilization, or multi-turn drift rather than a real correction signal [§1].
Empirically negative-gap tokens exceed 50% of all tokens in preliminary studies on Qwen2.5-3B-Instruct, making the asymmetry pervasive rather than a corner case [Fig. 3].
SDAR keeps the verifier-driven GRPO policy loss strictly unchanged and adds an auxiliary loss L = L_GRPO + β·L_OPSD where β is a mixing coefficient; the OPSD term is a per-token detached-gate-weighted student log-probability surrogate rather than a token-level advantage reweighting [§2.2].
The default gating strategy is gap gating: g_t = σ(k · Δ_t) where Δ_t = log π_teacher(y_t | s_t^teacher) − log π_student(y_t | s_t^student) is the detached teacher-student log-prob gap on the student-sampled token, k is a sharpness parameter, and gradients flow only through the student log-probability [§2.3].
Gap gating beats entropy gating and soft-OR (gap + entropy) gating; entropy is an indirect signal that fires on uncertain-but-already-well-handled tokens, and soft-OR dilutes selectivity by triggering on either signal [§3.4, Fig. 7].
SDAR internalizes the privileged skills: unlike Skill-GRPO which drops ~20 pp when tested without skills at inference, SDAR uses no skills at inference and still surpasses skill-augmented Skill-GRPO* in most settings (84.4 vs 80.5 on ALFWorld-3B, 53.9 vs 28.1 on ALFWorld-1.7B) [§3.1, Table 1].
SDAR is robust to retrieval quality: even Random Retrieval of skills outperforms the pure GRPO baseline by +1.9 / +1.6 / +1.0 on ALFWorld / WebShop-Score / WebShop-Acc, and Keyword Matching beats the higher-nominal-quality UCB retrieval on WebShop [§3.3, Table 2].
Reverse-KL as the token-level matching objective outperforms forward-KL and JSD because the mode-seeking nature of reverse-KL naturally down-weights low-teacher-probability tokens, which complements the explicit gate [§3.4, Fig. 9].
Training-dynamics telemetry shows the mean teacher-student gap Δ stays negative throughout training (privileged teacher is less confident on average than the student) and steadily converges toward zero, while the gate activation ratio starts strictly below 0.5 and climbs — the gate correctly suppresses net-negative signal early and admits more of it as the student improves [§3.2, Fig. 5].

Method

SDAR is a composed loss recipe, not a replacement for GRPO. For each prompt x the standard GRPO objective is computed as usual over a group of student rollouts, with sequence-level advantages from environment reward and importance sampling ratios against a reference policy π_ref. A separate OPSD auxiliary loss is then added: for each valid student-sampled token y_t at position t, the paper forms two contexts — the student context s_t^student (the rolled-out trajectory so far) and the teacher context s_t^teacher (the same trajectory prepended with retrieved skills from a SkillBank via UCB, keyword-matching, full retrieval, or random-baseline strategies). The detached teacher-student log-prob gap Δ_t is computed on the sampled token as a single-sample estimate of the reverse-KL importance signal — no full-vocabulary sum required.

Three gating strategies are proposed but only gap gating g_t = σ(k · Δ_t) is used by default. The gate is stopped-gradient so it acts as a scalar weight, and the token-level distillation loss becomes a masked average of −g_t · log π_student(y_t | s_t^student). Positive-gap tokens (teacher more confident) receive strong distillation pressure; negative-gap tokens (teacher less confident) are softly attenuated toward zero rather than actively pushed against. The overall loss L_GRPO + β·L_OPSD is optimized end-to-end, with β = 0.05 and k = 5 selected as optimal in ablations. The teacher branch is only used during training — at inference time the model runs without any retrieved skills.

Training uses 8× H800 GPUs for 150 steps across ALFWorld, Search-QA (NQ + HotpotQA training set, E5 retriever à la Search-R1), and WebShop, with the SkillBank imported from SkillRL. Batch sizes are 16 tasks × 8 rollouts per prompt (ALFWorld/WebShop) or 128 tasks (Search-QA).

Results

On Qwen2.5-3B-Instruct: SDAR reaches ALFWorld 84.4 (vs. GRPO 75.0, +9.4 pp), Search-QA avg 43.4 (vs. 36.4, +7.0 pp), WebShop-Acc 68.0 (vs. 63.3, +4.7 pp). On Qwen2.5-7B: ALFWorld 85.9 (vs. 81.2), Search-QA 49.0 (vs. 42.0), WebShop-Acc 82.8 (vs. 72.6, +10.2 pp). On Qwen3-1.7B-Instruct: ALFWorld 53.9 (vs. 46.1) — while GRPO+OPSD collapses to 32.0 and Skill-GRPO to 21.1 on the same backbone, SDAR is the only hybrid method to improve over pure GRPO.

Ablations pin down the design choices: β = 0.05 is optimal (β = 1.0 lets distillation overwhelm the RL signal, β = 0.01 exerts no useful pressure); k = 5 sigmoid sharpness balances between no-gate uniform distillation (small k) and strict binarization (large k); reverse-KL beats forward-KL and JSD by measurable margins. The training-dynamics plot (Fig. 5) is the clearest positive evidence: the mean teacher-student gap is net negative throughout training, which means uniform OPSD would systematically degrade the policy — and yet the gate identifies the specific token subset where the teacher provides beneficial signal, driving Δ toward zero without ever letting the negative-mean signal dominate.

Robustness across retrieval qualities (Table 2): all four retrieval strategies improve over the GRPO baseline; UCB / KM / Full retrieval give roughly +5–10 pp on ALFWorld and WebShop, and even zero-task-aware Random Retrieval still gives +1.0–1.9 pp — supporting the claim that gains come from gated distillation, not retrieval fidelity.

Why it’s interesting

SDAR is the fourth filed paper — after Rethinking OPD (Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe), CriPO (Enhancing Rubric-based RL via Self-Distillation), and the Jawade thread (On-policy distillation isn't a free lunch — four failure modes of OPD/OPSD in current post-training recipes (Bhavin Jawade)) — that names a specific structural failure mode of the on-policy-distillation-inside-RL recipe and proposes a targeted per-token intervention. It fills a gap the others leave: Rethinking OPD and Jawade describe when OPSD fails but don’t offer a hybrid recipe that keeps the GRPO loss unbiased; CriPO addresses rubric-based single-turn reasoning; SDAR is explicitly the multi-turn-agent case where OPSD’s per-turn KL surges and the privileged context is skill retrieval rather than gold answers. The observation that skill-conditioned teachers have net-negative log-prob gap (Fig. 5) is a direct empirical rebuttal to the “stronger teacher via privileged context” framing and reinforces the Jawade thread’s mode-2 warning (On-policy distillation isn't a free lunch — four failure modes of OPD/OPSD in current post-training recipes (Bhavin Jawade)) that teacher-student distributional overlap — not nominal teacher quality — is what matters.

Two connections to the concept-page state matter. First, the recipe complements the “context-space” and “parameter-space” poles of Tool-Use Agents: SDAR is parameter-space (updates weights via GRPO) but uses a retrieved-skill library at training time as a privileged distillation channel, and importantly the skills are then internalized into the weights and not used at inference — an intermediate design point between Training-Free GRPO’s frozen-base-plus-library and MiroThinker’s pure trajectory-SFT. Second, the gate-as-detached-scalar-weight design is architecturally similar to CriPO’s “flip advantage on tokens where teacher confidently disagrees” (Enhancing Rubric-based RL via Self-Distillation) but avoids modifying advantages — preserving GRPO’s unbiasedness — which is a cleaner engineering surface for practitioners already invested in a GRPO training stack.