Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning

ListenWhen Weaker SFT Wins: Preparing Models for RL10:00

Show notes

A look at PEAR, an SFT-stage loss-reweighting recipe that treats supervised fine-tuning as *preparation* for downstream RL rather than an end in itself. The central finding: after identical GRPO training, models started from stronger SFT checkpoints can underperform those started from weaker ones — so offline accuracy is a misleading proxy for post-RL performance. PEAR uses off-policy-evaluation-style importance weights to correct the behavior-to-target distribution mismatch, reporting pass-at-eight gains up to 14.6 percent on AIME 2025. We unpack the mechanism, the ablations, and what the framing means for anyone training visual generative models in two stages. **Sources** - Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

PEAR (Policy Evaluation–inspired Algorithm for Offline Learning Reweighting) is an SFT-stage loss-reweighting recipe that treats the offline stage as preparation for downstream online RL rather than as an end in itself. The paper’s central empirical claim is that after identical GRPO training, models initialized from stronger SFT checkpoints can significantly underperform those initialized from weaker ones — offline gains are an unreliable proxy for post-RL performance. PEAR corrects the offline-to-online distribution mismatch by importance-weighting each SFT token by the likelihood ratio between the target policy and the behavior (data-generating) policy, with sequence-level, token-level (suffix-ratio), and block-level variants. On Qwen2.5/3 and DeepSeek-Distill 0.6B–8B with GRPO on math and logic games, PEAR consistently improves post-RL performance over canonical SFT and over concurrent SFT-improvement methods (iw-SFT, DFT, TALR, Proximal-SFT), with up to 14.6% pass@8 gain on AIME-2025.

Key claims

After identical RL training, models initialized from stronger SFT checkpoints can significantly underperform those initialized from weaker ones — offline gains are an unreliable proxy for post-RL performance [§1, Fig. 1].
The mechanism is a distribution mismatch between the offline behavior policy μ (which generated SFT data) and the online target policy π_θ (which generates rollouts during RL): uniform token-level SFT supervision under prefixes induced by μ over-represents continuations that are dead-ends under π_θ [§2.2].
Standard SFT-improvement objectives (probability-based reweighting from Li et al., token-adaptive TALR, KL-regularized NLL) improve offline Pass@1 but do not reliably translate into post-RL gains; ranks can reverse (e.g., TopLogP wins offline on Qwen3-1.7B-Base but worst post-RL, under-performing plain SFT) [§2.1, Fig. 1].
Off-policy evaluation (OPE) provides the correct framework: correct the behavior→target shift by reweighting logged trajectories using the likelihood ratio π_θ/μ, at sequence, token, or block granularity [§2.3, §3].
PEAR reweights the standard per-token SFT loss by ω̃_t (numerically-stabilized importance weight, computed in log-space with clipping), with three variants: sequence-level product-ratio, token-level suffix-ratio (with discount γ for variance control), and block-level (blocks of length L trade granularity for stability); recovers token-level PEAR at block size 1 [§3.2–§3.5, Alg. 1].
Optional negative-example variant: for verified failures, apply a repulsive term that pushes the model away from full negative sequences using a sequence-level weight, avoiding the instability of signed suffix-ratio products [§3.6].
Pass@8 gain up to 14.6% on AIME-2025 after identical GRPO training; consistent improvements across 6 models (Qwen3-Base 0.6B/1.7B/4B/8B, Qwen2.5-1.5B-Math, DeepSeek-Distill-Qwen-1.5B) on AIME24/25, AMC23, Olympiad, MATH500 [§4.3, Tables 1–2].
Single-step (myopic) weighting is not enough: variants that reweight only the current-token likelihood ratio (concurrent proposals iw-SFT / DFT / Proximal-SFT / A-SFT) are less effective than PEAR’s suffix-based continuation weight; the effective quantity is whether the logged continuation remains plausible under π_θ [§4.3, Fig. 4a, Table 3].
PEAR generalizes beyond NLL: applied on top of KL-based knowledge distillation, PEAR further improves post-RL performance at minimal overhead [Fig. 4b].
Transfer: PEAR-initialized models transfer to a different online RL task distribution (SFT on SynLogic, RL on Enigmata) with better post-RL performance than SFT under identical RL compute — the gain is not domain-overfit [§4.4, Table 4].
Parameter-space analysis: PEAR’s gradients have smaller average principal angle with GRPO gradients than SFT and variants (offline updates more consistent with the online RL direction), and RL after PEAR shows the smallest NSS drift between offline and online checkpoints — heavy lifting is done at the offline stage, leaving less mismatch for online RL to correct [§4.6, Figs. 7–8].

Method

The paper starts from a standard offline SFT setup: dataset D of prompt–response pairs (x, y) where y is a sequence generated by a known behavior policy μ (e.g., a stronger teacher model). Standard NLL loss weights all tokens uniformly; PEAR modifies only the weights, not the underlying per-token loss.

Sequence-level PEAR computes the trajectory-wide importance ratio ρ = ∏t π_θ(y_t | y<t, x) / μ(y_t | y_<t, x) and applies the same ρ to every token in the sequence. Simple, but high-variance for long trajectories.

Token-level PEAR (suffix-based) replaces the global ρ with a per-token suffix-ratio: at token t, weight the loss by the likelihood ratio of the remaining continuation from t onward, discounted by γ. Intuition: gradient at t up-weights actions whose continuation the target policy actually revisits. This is the paper’s default variant, denoted PEAR_{B=1}.

Block-level PEAR partitions positions into contiguous blocks of length ≤ L, computes the intra-block importance ratio r_b and the suffix-after-block s_b, and assigns each token in block b the same discounted weight w_b = r_b · s_b^γ. B=1 recovers token-level PEAR; larger B trades granularity for stability.

Numerical stabilization computes all ratios in log-space, clips per-decision ratios to [-C_ratio, C_ratio] and final weights to [-C_weight, C_weight], and stop-gradients through the weights.

Optional negative examples: for verified failures, add a repulsive term with sequence-level weight that performs gradient ascent on negative trajectories — sidesteps the signed suffix-product instability that token-level negative weighting would introduce.

Experimental setup: SFT for 1 epoch (lr 1e-5 on games, 5e-6 on math), γ = 0.9, per-decision clip [0.5, 2.0]. Downstream online RL is GRPO (lr 5e-7, batch 128, KL coeff 0.01) — the same recipe for every SFT-stage variant, isolating the SFT-stage contribution. Offline data: 100k correct trajectories on SynLogic; 100k Qwen3-8B-verified responses on SYNTHETIC-2 math. Online RL data: SynLogic for games, DAPO-17k for math. Evaluation: Pass@{1, 8, 64} on MATH-500, MINERVA, AIME-2024, AIME-2025, AMC-2023 (avg 64 samples).

Results

Post-RL Pass@1 average across math benchmarks (5 datasets): PEAR_{B=1}+GRPO vs SFT+GRPO — 33% vs 28% (Qwen2.5-1.5B-Math), 39% vs 14% (DS-Qwen-1.5B, +25 pp), 43% vs 39% (Qwen3-4B-Base), 41% vs 35% (Qwen3-8B-Base) [Table 1].
Pass@8 average: 55% vs 51%, 65% vs 32% (+33 pp), 65% vs 56%, 64% vs 59% — DS-Qwen-1.5B is the biggest gainer with +36 pp on AIME24 avg-64 and +50 pp on AIME24 pass@64 [Table 1].
On logic games (SynLogic): SFT+GRPO 9%/23%/39%/36% (Qwen3 0.6B/1.7B/4B/8B) → PEAR_{B=1}+GRPO 12%/26%/44%/41%; single-step-weighting variant hurts on 3/4 sizes (10%/18%/38%/30%) [Table 3].
Off-domain transfer: SFT on SynLogic → RL on Enigmata → evaluate on Enigmata OOD: SFT+GRPO 2.8%/13.1%/49.5%/53.3% → PEAR+GRPO 13.1%/38.3%/59.8%/61.7% (biggest gain +25.2 pp on Qwen3-1.7B) [Table 4].
Concurrent baselines: PEAR beats iw-SFT, DFT, TALR, Proximal-SFT on post-RL performance in matched conditions, though several of them win at offline Pass@1 — reinforcing the paper’s central thesis [§4.3, Fig. 1].
PEAR on KL-based KD: reweighted KD → PEAR+GRPO further improves over KD baseline with negligible overhead (reuses KD-computed probabilities) [Fig. 4b].
Negative examples: adding 50k verified negatives (with 50k positives) yields substantial additional gains on Qwen3 model families under identical RL budget [§4.5, Fig. 4a].
RQ1 (concentration): high-weight tokens are distributionally more steered away from the base policy π₀ than low-weight tokens — PEAR concentrates learning on positions the target policy would revisit [§4.6].
RQ2 (offline-online consistency): PEAR gradients have smaller principal angle with GRPO than SFT/variants; NSS drift between offline and online checkpoints is smallest for PEAR — the paper argues PEAR “does the heavy lifting” at the offline stage so RL has less mismatch to correct [Figs. 7–8].
KL constraint tradeoff: stronger KL preserves closeness to base but worsens offline-online gradient alignment — suggests KL regularization can cause offline-to-online mismatch rather than fix it [§4.6].

Why it’s interesting

This is one of two papers Sirou posted together; the direct quote in the Slack message — “models initialized from stronger SFT checkpoints can significantly underperform those initialized from weaker ones” — is this paper’s Fig. 1 finding. PEAR provides the theoretical framework (OPE / importance sampling) and the loss-reweighting recipe for the same phenomenon that Behavior Injection: Preparing Language Models for Reinforcement Learning targets with a data-augmentation recipe. Both explicitly argue that pre-RL accuracy is a misleading proxy for post-RL performance and that the goal of SFT should be reframed as “prepare a good RL initialization.”

The most Luma-relevant conceptual contribution is the offline-to-online distribution mismatch framing: the behavior policy that generated your BC/SFT data may not be the target policy that your online RL will optimize, and uniform token-level supervision can systematically push the model toward regions of state-action space that RL will never revisit. This directly reframes the “why does SFT on the train set hurt subsequent RL” question in Sirou’s Slack note — the answer is not that SFT is over-fitting in a generalization sense, but that it is aligning to μ instead of π_θ. For robot RL where the behavior policy is typically a teleop demonstrator or a scripted controller and the target policy is a stochastic manipulation policy under simulator noise, the mismatch is likely severe and PEAR-style importance-weighting (or its suffix-based continuation-weight variant) is a directly transferable primitive.

Sits alongside The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation as the second filed paper on the wiki that formally decomposes what SFT/OPD contributes to downstream RL: where Men et al. use per-token mutual information to split planning ability into general patterns vs. task-specific knowledge, PEAR uses OPE likelihood ratios to reweight tokens by future-plausibility under π_θ. Complements Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe‘s empirical OPD failure-mode taxonomy and provides the missing OPE-theoretic account of why single-step / myopic reweighting (the shape of iw-SFT / DFT / Proximal-SFT) underperforms continuation-based weighting.

Also connects to Spurious Rewards: Rethinking Training Signals in RLVR and Understanding Reasoning from Pretraining to Post-Training: if RLVR mostly elicits pretraining/SFT behaviors rather than installing new ones, then the SFT stage’s job is to make the target behaviors elicitable — PEAR operationalizes this by weighting SFT tokens toward continuations that π_θ can actually revisit.