Score Centering Stabilizes Off-policy Reinforcement Learning

TL;DR

Score Centering is an additive, drop-in correction term for LLM RL that cancels the drift responsible for the training-inference mismatch (TIM) — the persistent bias between the rollout engine and the training engine that accumulates every step and destabilizes off-policy runs (quantized KV cache, quantized rollouts, async weight updates). The authors (Together AI) decompose the vanilla policy-gradient update by covariance into a reward-correlated “signal” term and a reward-independent “drift” term; on-policy the drift term has zero expectation, off-policy it does not, and its accumulation is what breaks stability. Subtracting a running estimate of the expected trainer score under the sampler cancels the drift. From 0.6B to 30B params, Score Centering alone matches or outperforms TIS/MIS importance-sampling corrections under quantization, with the gap widening as TIM grows; because it is additive, it also composes with importance sampling and beats pure-IS baselines in staleness experiments.

Key claims

The instability of RL under TIM is primarily caused by drift — a persistent, per-step bias between training and inference engines that accumulates across updates — not by variance of the importance ratio [Abstract, §1].
Standard policy gradient decomposes by covariance into a reward-correlated “signal” term and a reward-independent “drift” term; on-policy the expected trainer score is zero and drift vanishes, off-policy it is nonzero and accumulates [Fig. attached to tweet part 2 / body §2–3].
An additive correction that subtracts a running estimate of the expected trainer score cancels drift while leaving the signal term intact — no changes to the RL algorithm, no learned corrector, no extra rollouts [Abstract, §3].
On models from 0.6B to 30B parameters, Score Centering alone matches or outperforms importance-sampling-based corrections (TIS, MIS) under heavy model and KV-cache quantization, with the gap growing as TIM becomes more severe [Abstract, §4].
Because the correction is additive, Score Centering composes with importance sampling; the composition beats pure IS baselines in staleness (async weight update) experiments [Abstract, §4].

Method

The paper starts from the standard REINFORCE / policy-gradient update and applies the covariance identity E[fg] = Cov[f,g] + E[f]E[g] to the product of the trainer log-probability score and the reward. This splits the gradient into a signal term (Cov[reward, score]) and a drift term (E[reward] · E[score]). Under an on-policy sampler E[score] = 0 because the score is the derivative of a log-density normalized under itself; under an off-policy sampler (the rollout engine), the trainer score has nonzero expectation and its product with the mean reward produces a directed bias that accumulates every step. Score Centering estimates this expected score under the sampler and subtracts it from every rollout’s trainer score before forming the gradient — an additive, one-line intervention that is orthogonal to the RL objective, the reward, and the model architecture. It composes with importance sampling by applying to the centered score inside the IS ratio, so IS still corrects for the variance-inducing distribution shift while Score Centering removes the bias term IS does not touch. Experiments cover heavy weight + KV-cache quantization in rollout and async update staleness; benchmarks span 0.6B → 30B models and stack Score Centering on top of TIS/MIS baselines.

Results

Score Centering alone matches or beats TIS/MIS under quantized-rollout regimes across 0.6B, 1.5B, 7B, and 30B model scales, with the advantage over IS baselines growing as the quantization-induced mismatch becomes more severe [Abstract, §4].
Composing Score Centering with importance sampling outperforms pure-IS baselines under async weight update (staleness) — the two corrections handle different failure modes and stack [Abstract, §4].
No explicit throughput number reported; because Score Centering is a per-step subtraction of a running estimate, it inherits the compute profile of the base RL update.

Why it’s interesting

This paper stakes out a third position on the training-inference mismatch axis that the wiki has been tracking on the reasoning-RL and training-stability pages. Defeating the Training-Inference Mismatch via FP16 locates TIM in numerical precision — swap BF16 for FP16 and the ~24× log-prob-ratio gap collapses; The 4-bitter Lesson: Balancing Stability and Performance in NVFP4 RL navigates NVFP4 with dequantized backward + selective precision to preserve stability. Both treat TIM as a precision problem to remove at the source. Score Centering, in contrast, accepts that TIM will exist (quantized inference, staleness) and derives an algorithmic correction that cancels its accumulating bias without changing the numerics stack — closer in spirit to TIS/MIS but attacking bias where those attack variance, and provably orthogonal to them. The framing (covariance decomposition into signal + drift) is a rare filed instance of a policy-gradient stability proposal where the failure mode is derived rather than measured, and the fix is one term rather than a recipe. It also lands directly on the “prior fixes … become unnecessary when the underlying mismatch is removed” claim from the FP16 paper: Marek & Ryabinin’s composition experiments suggest FP16 and Score Centering may not fully substitute — the drift-vs-variance split gives two independent stability levers rather than one.