Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training

TL;DR

A layer-wise study of RL post-training (GRPO, GiGPO, Dr. GRPO across Qwen3 / Qwen2.5) showing that training a single transformer layer — with all others frozen — recovers most of the gains of full-parameter RL and sometimes exceeds it. The paper introduces “layer contribution” as a quantitative measure of the fraction of full-RL improvement recovered by isolating one layer. Across seven models, three RL algorithms, and multiple domains (math reasoning, code, agentic decision-making), the same structural pattern emerges: gains concentrate in a small subset — often a single — mid-stack layer, and the resulting layer ranking is stable across datasets, tasks, and algorithms. Directly complements the parameter-space finding of The Path Not Taken: RLVR Provably Learns Off the Principals with a layer-space finding: RL is not only off-principal in weight space, it is also localized in depth.

Key claims

Training a single transformer layer during RL post-training can recover most — and occasionally exceed — the gain from full-parameter RL, across seven models and three RL algorithms [Abstract].
Layer contribution is introduced as a quantifiable primitive: the fraction of full-parameter RL improvement recovered by RL-training that layer in isolation [Abstract].
The load-bearing layers concentrate in the middle of the transformer stack; layers near the input and output ends contribute substantially less [Abstract].
The layer ranking induced by contribution is strongly correlated across datasets, tasks, model families (Qwen3 vs Qwen2.5), and RL algorithms (GRPO / GiGPO / Dr. GRPO) — i.e. the pattern is a property of RL post-training on transformers, not of any particular data or optimizer [Abstract].
The uniform-parameter-update assumption implicit in current RL post-training recipes is empirically false: gains are highly concentrated, not distributed [Abstract].

Method

The paper conducts a systematic layer-wise ablation across seven checkpoints spanning two model families (Qwen3, Qwen2.5). For each (model, RL algorithm, task) triple it runs a family of restricted RL runs where all parameters except those inside a single transformer block are frozen, then measures the fraction of full-parameter RL’s task-metric improvement recovered by that isolated-layer run. This defines layer contribution. The layer sweep is repeated across three RL algorithms (GRPO, GiGPO, Dr. GRPO) and across task domains (math reasoning, code generation, agentic decision-making) to test how stable the resulting layer-contribution profile is under changes to the training signal.

The analysis then looks at the shape of the profile as a function of depth (input → mid → output stack), and computes rank correlations between profiles across (model, algo, task) triples.

Results

Across the sweep, a single mid-stack layer typically recovers “most” of full-RL improvement, and in some (model, algo, task) cells surpasses full-RL — a paper-headline observation the authors call “remarkably stable” across the seven models [Abstract].
The high-contribution layers cluster in the middle of the stack; input-adjacent and output-adjacent layers contribute substantially less [Abstract].
Layer rankings are strongly correlated across the two model families, three RL algorithms, and multiple task domains, so the profile is not a data-or-algorithm artifact [Abstract].

Why it’s interesting

This is the layer-space companion to The Path Not Taken: RLVR Provably Learns Off the Principals — that paper showed RL updates concentrate on off-principal, low-magnitude weights and preserve each layer’s singular spectrum; this paper shows those updates are also concentrated in a small subset of mid-stack layers. Read together, they give a much sharper picture of RL post-training’s real effective rank: not just low in the off-principal subspace within each layer, but also low across depth. This tightens the mechanistic explanation the Thinking Machines rank-1-matches-full-RL result was reaching for.

For Parameter-Efficient Finetuning, the paper is a direct architectural complement to LoRA-style low-rank adaptation: LoRA constrains capacity in the weight-matrix dimension; single-layer RL constrains it in the depth dimension. A joint recipe (single mid-stack layer + LoRA-style low-rank update inside that layer, or an off-principal restriction per The Path Not Taken: RLVR Provably Learns Off the Principals) is the obvious next experiment and would give a very small PEFT budget for RL. It also contrasts productively with ELT: Elastic Looped Transformers for Visual Generation and Looped Transformers more broadly: looped transformers argue some layers should be reused many times for reasoning compute; this paper argues few layers should be updated at all for RL post-training. Both point to the middle of the stack as the load-bearing region.