AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

TL;DR

AgentOPSD is a critic-free, recursive turn-level credit assignment method for agentic RL that sits between GRPO’s trajectory-level advantage and existing token-level privileged self-distillation. It aggregates token-level teacher-student log-probability gaps into turn-level evidence, then recursively updates a Bayesian belief in log-odds space so that sparse outcome supervision converts into per-turn credit that also identifies pivotal turns (via marginal belief revision between consecutive states). No extra critic, no extra rollouts, and the method is compatible with off-the-shelf policy optimizers. On Qwen2.5-3B/7B evaluated on ALFWorld, WebShop, and Search-QA, AgentOPSD outperforms GRPO and strong self-distillation baselines, reaching 89.1% success on ALFWorld with Qwen2.5-7B.

Key claims

Trajectory-level advantage estimates from RL with verifiable rewards systematically fail to credit the pivotal decisions in long-horizon multi-turn agentic tasks, motivating turn-level credit assignment [§1].
Existing privileged self-distillation gives denser supervision at the token level but leaves open how to structure those local signals into sequential credit — AgentOPSD frames this explicitly as the design question the paper answers [§1].
Aggregating token-level teacher-student log-probability gaps to turn-level evidence, then updating a Bayesian belief state in log-odds space recursively across turns, yields a principled reweighting from sparse outcome supervision to turn-level credit signals [§Method].
Pivotal turns are identified as those with the largest marginal belief revision between consecutive states — a diagnostic that comes free from the recursive belief update rather than requiring a separate criticality classifier [§Method].
AgentOPSD is critic-free and adds no extra rollouts, drop-in compatible with standard policy optimization on top of existing GRPO stacks [§Method].
On ALFWorld with Qwen2.5-7B, AgentOPSD reaches 89.1% success, beating GRPO and strong self-distillation baselines [Abstract, §Experiments].
Ablations attribute the gains to two components: (a) turn-level aggregation of token gaps and (b) history-dependent recursive belief updates — dropping either degrades performance [§Ablation].

Method

AgentOPSD keeps the standard multi-turn agentic RL loop (rollouts scored by a verifiable outcome reward) and inserts a turn-level credit assignment stage between rollout scoring and the policy update. For each rollout, the student policy’s per-token log-probabilities are compared to a privileged self-distilled teacher (the same policy given access to privileged information not available at inference), and the token-level gaps Δ = log π_teacher − log π_student are aggregated into per-turn evidence. The recursion is Bayesian: the log-odds of a hidden “this turn was helpful” latent are updated turn-by-turn using aggregated evidence, so the belief state accumulates history rather than treating each turn independently. Turn-level credit is then read off as a function of the belief posterior, and pivotal turns are the ones with the largest belief revision between consecutive time steps. Because the recursion lives in log-odds space, it composes cleanly with the additive token-log-prob-gap signal without a separate value network — this is the “critic-free” property. The credit signal is used as a turn-level advantage inside standard policy optimization; no changes are made to the RL sampling loop, and no additional rollouts are collected.

Experiments use Qwen2.5-3B and Qwen2.5-7B on three canonical multi-turn agentic environments: ALFWorld (embodied household), WebShop (web navigation and purchase), Search-QA (multi-hop web-search question answering). Baselines are GRPO and strong self-distillation methods (paper cites but does not enumerate in the abstract).

Results

Headline: 89.1% success on ALFWorld with Qwen2.5-7B, outperforming GRPO and strong self-distillation baselines. Wins reported across the three benchmarks (ALFWorld, WebShop, Search-QA) at both 3B and 7B scales [Abstract, §Experiments].

Ablations show that both turn-level aggregation of token gaps and the recursive history-dependent belief update are necessary — dropping either reduces performance materially. This is the paper’s quantitative case that turn-level credit assignment on top of self-distillation needs both the aggregation step and the recursion, not just one [§Ablation].

Why it’s interesting

AgentOPSD is from the same group (Zhengxi Lu, Zi-Han Wang, Yongliang Shen, and colleagues at Zhejiang / Tsinghua / Meituan) that filed SDAR (Self-Distilled Agentic Reinforcement Learning), and reads as a direct successor targeting the credit-assignment layer where SDAR left off. SDAR’s contribution was a gated token-level auxiliary loss on top of unmodified GRPO — a token-level intervention. AgentOPSD moves one level up: instead of gating tokens, it aggregates token gaps into turn-level evidence and recursively propagates them through a Bayesian belief in log-odds space. The recursion adds history-dependence that neither SDAR nor CriPO (Enhancing Rubric-based RL via Self-Distillation) had. This is also the first filed paper on the concept page whose credit-assignment mechanism produces a diagnostic — pivotal-turn identification via marginal belief revision — as a side product, making it a natural pairing with the “pivotal-turn” framing that AREX (AREX: Towards a Recursively Self-Improving Agent for Deep Research) argued for at the reward-shaping layer.

Two connections to filed clusters matter. First, this is a new datapoint for the Reasoning RL thread on “how to get per-step credit from outcome-only rewards” — the sub-line that includes AREX’s decisive-evidence + directional-correction dense rewards, VR-GRPO’s global+step decomposition (UniVR: Thinking in Visual Space for Unified Visual Reasoning), and now AgentOPSD’s belief-based turn-level credit. Second, on Tool-Use Agents it complements MiroThinker’s “extend the tool-call budget” scaling axis and Training-Free GRPO’s context-space credit assignment with a third orientation: keep the RL setup, but redirect where the credit lands. Whether the Bayesian recursion is the load-bearing piece or turn-level aggregation alone would suffice is the natural follow-up experiment.