EasyPPO: Stabilizing the Critic Is Key

TL;DR

EasyPPO (Berkeley + Princeton, Sept 2026) argues that instability in LLM PPO comes primarily from the critic, not the actor, and identifies two concrete failure modes: (1) filtering truncated rollouts from both actor and critic secretly reshapes the policy objective into reward-conditioned-on-completion, so truncation rates can climb even as conditional reward improves; and (2) heterogeneous return variance across prompts lets a handful of high-variance prompts dominate finite-batch critic updates. Three simple fixes — actor-only overlong filtering, noise-normalized critic regression (weight each prompt’s critic loss by 1/std of its returns), and smaller critic mini-batches — stabilize training end-to-end. On Qwen3.5-9B across coding (FrontierCS), math (AIME24), and agentic search (Search-R1) at rollout batch sizes 512–1024, EasyPPO beats vanilla PPO, VAPO, and HL-Gauss PPO with best-validation relative gains of 14.89% / 2.28% / 9.47%.

Key claims

Filtering truncated rollouts from both actor and critic changes the policy objective to reward conditioned on completion, which permits truncation rates to increase during training even while conditional reward improves — the fix is actor-only overlong filtering that still trains the critic on returns from truncated rollouts [§1 / Fig. 1 left].
Return noise is heterogeneous across prompts: some prompts have return distributions with much higher variance than others, and their gradient contributions dominate finite-batch critic updates — noise-normalized critic regression (weight each prompt’s critic loss by the inverse standard deviation of its sampled returns) equalizes noise contributions across prompts [§1 / Fig. 1 middle].
Empirically, without noise normalization critic gradient norms scale strongly with the prompt’s return standard deviation; after normalization the dependence is largely removed across training steps [author thread §5].
Moderately smaller critic mini-batches confine outlier influence to fewer rollouts through gradient clipping, providing a third orthogonal stabilizer [§1 / Fig. 1 right].
Combined, EasyPPO is the only method among vanilla PPO, PPO + actor-only filtering, HL-Gauss PPO, and VAPO that remains stable across all three tested settings (coding / math / agentic search) at rollout batches of 512–1024 — several baselines exhibit late-stage collapse [Abstract, §1].
Reported gains at best validation over vanilla PPO: +14.89% on FrontierCS (continuous reward, coding), +2.28% on AIME24 (binary reward, math), +9.47% on Search-R1 (multi-turn agentic search) [Abstract].

Method

EasyPPO is three small, orthogonal modifications to vanilla PPO in verl. Actor-only overlong filtering drops truncated rollouts from the actor update only, keeping them in the critic update so returns from both completed and truncated trajectories train the value function — this prevents the policy from optimizing a shifted objective (reward | completion) that lets truncation rates drift upward. Noise-normalized critic regression rescales each prompt’s critic loss by the inverse standard deviation of its sampled returns; because the return-noise term contributes a constant amount to the critic’s gradient magnitude across prompts, no small subset of high-variance prompts can dominate a finite-batch update. Smaller critic mini-batches (split each rollout batch into four critic mini-batches, recompute + clip at each optimizer step) shrinks the number of rollouts a single outlier can influence during gradient clipping. All three sit on top of standard PPO in verl; no changes to the actor loss, no auxiliary networks, no extra rollouts.

Results

Evaluations use Qwen3.5-9B (with SFT on FrontierSmith trajectories for FrontierCS; base model for AIME and Search-R1), 30 critic warmup steps, strictly on-policy training in verl, rollout batch sizes 512–1024. Across all three settings, EasyPPO is the only method (of vanilla PPO, PPO + actor-only filtering, HL-Gauss PPO, VAPO) that remains stable throughout the full training horizon; several baselines show late-stage collapse. Best-validation relative gains over vanilla PPO: +14.89% on FrontierCS, +2.28% on AIME24, +9.47% on Search-R1 [Abstract, §5.1]. The critic-gradient-norm-vs-prompt-return-std plot shows the pre-normalization pathology (strong positive dependence) and post-normalization near-flatness across training [Author thread §5].

Why it’s interesting

EasyPPO adds a critic-side entry to a filed conversation that has been dominated by actor-side interventions. Compare to Score Centering Stabilizes Off-policy Reinforcement Learning (Score Centering, Together AI) which cancels off-policy trainer-score drift additively on the actor gradient, Defeating the Training-Inference Mismatch via FP16 which attacks TIM at the precision layer, and the flow-matching PPO family (VINE: Taming Generative Control Policies for Reinforcement Learning, π*0.6: a VLA That Learns From Experience (RECAP) RECAP, Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning QGF) which reshapes the objective around expressive samplers — EasyPPO instead argues that the value network itself is where the instability lives when the actor uses standard PPO on chat-scale LLMs. The heterogeneous-return-variance diagnosis also connects to FlashREINFORCE: Critic-Free Single-Rollout Asynchronous RL for Agentic Language Models‘s finding that sample-mean-vs-token-mean loss aggregation matters at length: both papers identify that finite-batch aggregation choices, not the RL objective, are the load-bearing knob for LLM-RL stability. It’s also a direct pushback on the GRPO-as-default framing on Reasoning RL — the paper explicitly keeps the critic and shows that critic-based PPO can outperform GRPO-family methods when the critic is stabilized properly.