The Path Not Taken: RLVR Provably Learns Off the Principals

TL;DR

Opens the RLVR black box by tracking where in weight space the ΔW lives across training. Empirically, RLVR (GRPO, DAPO, Dr. GRPO) consistently updates off-principal, low-magnitude weights while preserving each layer’s singular-value spectrum; SFT does the opposite, targeting principal weights and distorting the spectrum. The paper’s Three-Gate Theory (KL anchor → model geometry → finite precision) mechanistically explains the bias, and the case studies show concretely that SFT-era PEFT tricks — sparse FT on principal weights, PiSSA — misfire or collapse when applied to RLVR, while the off-principal recipe tracks dense-RLVR. The headline implication for the team: RL and SFT are not interchangeable optimization regimes, and PEFT design for RL needs to be redone from scratch.

Key claims

RLVR updates concentrate on off-principal, low-magnitude weights and induce minimal spectral drift, while SFT targets principal weights and distorts the spectrum [Abstract, §1].
The bias is mechanistically a three-gate process: Gate I (KL anchor) constrains each on-policy step to small KL; Gate II (model geometry) steers the constrained step into low-curvature subspaces orthogonal to principal directions, preserving the spectrum; Gate III (finite precision) hides the resulting micro-updates in non-preferred regions so the bias appears as sparsity [Abstract, §1].
Empirically, restricting updates to principal weights (an SFT prior, following Liu et al. 2025c) gives the weakest RLVR optimization trajectory and degrades both forward-KL drift and accuracy; restricting to non-principal, low-magnitude weights closely tracks the dense RLVR run [§Case studies, sparse FT].
PiSSA collapses under RLVR. PiSSA initializes the LoRA factors along the principal singular directions of the base weight matrix — exactly the directions RLVR avoids. Under RLVR this produces no gain over standard LoRA and, when forced to update along principal directions, leads to training instability and rapid collapse [§Case studies, LoRA variants].
The off-principal bias is the load-bearing reason a recent report (Schulman & Thinking Machines Lab) finds rank-1 LoRA can match full-parameter RL — the natural learning subspace is genuinely low-effective-rank in directions orthogonal to the principal subspace, not within it [§Case studies, LoRA variants].
RLVR’s spectrum-preservation property is invariant to RL-algorithm choice (GRPO, DAPO, Dr. GRPO produce the same parameter-level signature), suggesting the bias is a property of the KL-constrained-on-policy structure rather than of any particular policy-gradient variant [§Empirical evidence].

Method

The paper proceeds in three steps. First, it constructs an update mask (which parameters actually change during RLVR training) and a principal mask (top-k locations along each layer’s principal singular directions), and shows the two are spatially anti-correlated across layers: RLVR updates avoid the principal mask. The corresponding plot for SFT shows the opposite alignment.

Second, it derives the Three-Gate Theory. The KL anchor (Gate I) constrains the per-step update norm; the model geometry (Gate II) — formalized via the curvature of the policy loss around the base weights — steers the constrained step into the low-curvature subspace, which the analysis identifies with the off-principal, low-magnitude region; finite precision (Gate III) explains why the resulting tiny updates manifest as sparse weight changes at fp/bf precisions.

Third, the paper validates the theory operationally through two PEFT case studies. Sparse fine-tuning: it constructs three masks (principal-only, non-principal-low-magnitude, dense) and runs RLVR under each, finding that the non-principal mask tracks the dense baseline while the principal mask is worst. LoRA variants: it compares vanilla LoRA, MiLoRA (minor singular components initialization), and PiSSA (principal singular components initialization) under RLVR, finding PiSSA collapses while the others survive — and that no advanced LoRA initialization beats vanilla LoRA in RLVR.

Results

Principal-only sparse FT yields the worst KL drift and final accuracy among the three masks; non-principal-low-magnitude tracks dense RLVR [§Case studies, Fig. and accompanying tables].
PiSSA under RLVR (e.g. on the math-reasoning setup studied in the paper) drops to near-zero accuracy in the follow-up evaluation by Lin et al. (~0.2% accuracy) when restricted to principal-subspace updates, while standard LoRA preserves baseline accuracy [§Case studies, LoRA variants; corroborated by arXiv:2512.23165 which cites this paper as the explanation].
PiSSA provides no gain over vanilla LoRA under RLVR, contradicting Schulman & Lab’s conjecture that advanced LoRA variants would further improve RL [§Case studies, LoRA variants].

Why it’s interesting

The wiki’s reasoning-rl cluster has accumulated nine papers documenting the GRPO-and-shaped-rewards recipe and what works on top of it, but until this paper none of them had a parameter-level account of what RL is doing to the weights. This is the first filed white-box result on that question, and it has immediate consequences for how to do PEFT on Luma RL pipelines: the Parameter-Efficient Finetuning page’s current claim list is dominated by SFT-era results — µA’s LR-transfer prescription (Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning), LoRWeB’s basis-of-LoRAs claim (Spanning the Visual Analogy Space with a Weight Basis of LoRAs), the hypernetwork-LoRA recipes — and this paper says explicitly that the SFT-era playbook does not transfer to RL.

Two specific connections worth flagging. First, the paper defends the rank-1-LoRA-matches-full-parameter-RL claim from the Thinking Machines Lab report (Interaction Models: A Scalable Approach to Human-AI Collaboration is from the same group) by giving it a mechanistic explanation: RL really is low-effective-rank, but only in the off-principal subspace. Second, the spectrum-preservation property connects to the broader “weights stay near init” literature on this wiki (Effect of the initial configuration of weights on the training and function of artificial neural networks, Controlled LLM Training on Spectral Sphere) — RLVR appears to be an extreme instance of “stay structurally close to the base”, which suggests RL post-training is closer in optimization character to the lazy-training / NTK regime than SFT is.