Skip to content
Bud's Wiki

SFT-for-RL Readiness

TL;DR

Section titled “TL;DR”

A growing thread reframes the SFT stage as preparation for downstream RL rather than as an end in itself. The shared empirical thesis: stronger SFT checkpoints (measured by pass@1 / offline accuracy) can significantly underperform weaker ones after identical RL, so the SFT objective should be redesigned to grow pass@K coverage and align with what RL will amplify. The design space so far splits along three orthogonal axes — loss reweighting (Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning PEAR), data augmentation (Behavior Injection: Preparing Language Models for Reinforcement Learning BRIDGE), and sequence selection (TailSFT: Filtered Fine-Tuning Improves Post-Training Performance TailSFT).

Key claims

Section titled “Key claims”

Recent contributions

Section titled “Recent contributions”
  • [2026-11-07] TwIL-8B — 8.8B Reasoning Model Post-Trained from Granite 4.2 with SFT + GRPO/DAPO RL: TwIL-8B is a production instance of the SFT-then-RL pipeline: full-parameter SFT on reasoning traces (math/STEM/IF) before GRPO/DAPO RL on the base Granite 4.2-8B, reaching MATH-500 98.2 and LiveCodeBench 81.6
  • [2026-11-06] Beyond Validation Loss: Coverage and Reliability Dynamics in Robot Policy Pretraining: Sun, Simchowitz & Song (Stanford REAL / CMU) answer the concept page’s standing open question on robot RL transfer — the first filed robotics-side instance of the “stronger SFT can underperform post-RL” phenomenon, now on diffusion/flow-matching robot policies rather than LLMs. Diagnoses the mechanism as a coverage-reliability tradeoff visible in closed-loop pass@k over BC pretraining: checkpoints with matched pass@1 but different large-k pass@k yield different posttraining gains across filtered BC, critic action ranking, residual RL, and full-policy RL — reproducing TailSFT: Filtered Fine-Tuning Improves Post-Training Performance TailSFT’s “pass@K predicts post-RL, pass@1 does not” finding on generative robot policies. Adds a fifth axis to the three-axis design taxonomy (loss reweighting / data augmentation / sequence selection) on this page: JointAug is data augmentation in the joint (observation, action) distribution rather than at the sequence level, motivated by the diagnosis that observation conditioning (standard π(a∣o)\pi(a \mid o)) is what causes finite-data diffusion policies to overfit — a mechanism-specific answer distinct from BRIDGE’s exploratory-behavior injection (Behavior Injection: Preparing Language Models for Reinforcement Learning) and PEAR’s suffix-ratio continuation weighting (Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning). Also adds a checkpoint-selection dimension no prior entry targets: proposes an offline quantile-based criterion over held-out action-error distributions that predicts the closed-loop-best checkpoint without closed-loop evaluation, sibling to TailSFT’s lightweight pre-run diagnostic. Validation on MimicGen (sim, residual RL on JointAug beats baseline final checkpoint), RoboCasa VLA pretraining (action-only JointAug reaches matched SFT), and real Push-T (6-DoF, wrist-only; offline-selected checkpoint beats final checkpoint on real-robot success). The joint observation-action modeling path answers the open question of which of the three axes ports first to robotics with a mechanism argument: action-only data augmentation is the lighter variant and the full-joint variant is the strongest, but neither reduces to either PEAR’s loss reweighting or TailSFT’s sequence selection.
  • [2026-10-21] Learning from Fleet Experience: Collective Experience Learning across 96 Robots: CEL/FERT (Tsinghua + AgiBot + Infinigence AI) is the first filed robotics-side instance that makes the SFT-stage-as-RL-preparation framing operationally concrete at fleet scale, answering the open-question row on this page about robot RL transfer. The initialization stage is explicit two-step SFT → (M)HG-DAgger (lifts mean success across 5 real-world bimanual tasks from 36.3% to 81.0%); the downstream “RL-like” stage (FERT online BC on fleet-generated successes, failure-guided retrieval reweighting) only crosses from degrading to improving the policy above a 4-robot fleet size, with 4 robots dropping phone-packaging success below the 79.2% SFT+DAgger starting policy (50.8%) and 16 robots lifting it to 86.7%. The gap between SFT+DAgger and the self-improvement regime is empirically load-bearing — matches the cluster’s thesis that the SFT (+DAgger) checkpoint strength is specifically what determines whether downstream autonomous-rollout training amplifies or regresses performance. Distinct axis from the three LLM-side recipes already on the page (loss reweighting, data augmentation, sequence selection) in that it operates in the trajectory-selection space during an online-BC-on-successes stage via failure-conditioned similarity retrieval — a potential fourth axis for the three-axis design taxonomy, scoped to physical-policy RL-adjacent regimes.
  • [2026-09-11] Demystifying Reinforcement Learning Post-Training of Language Models: Clay et al. (UW + AI2) contribute the entropy-lens diagnostic to the SFT-for-RL-readiness cluster: comparing pretraining, SFT, and RL-post-trained checkpoints by the entropy of their output distributions reveals how each stage shapes model certainty differently. Complementary to PEAR’s OPE-theoretic reweighting (Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning), BRIDGE’s behavior-injection (Behavior Injection: Preparing Language Models for Reinforcement Learning), and TailSFT’s coverage-first sequence selection (TailSFT: Filtered Fine-Tuning Improves Post-Training Performance) — those three prescribe SFT recipes; this paper offers a shared measurement axis to diagnose why an SFT checkpoint is or isn’t RL-ready. Reinforces the cluster’s central thesis via a different framing: RL success requires sufficient base-prior probability mass on the desired behavior (an exploration condition), so SFT’s job is to move that mass into the right place — measurable via entropy.
  • [2026-09-01] TailSFT: Filtered Fine-Tuning Improves Post-Training Performance: TailSFT (Malladi, Jelassi, Foster, Ash, Krishnamurthy) contributes the sequence-selection axis to the SFT-for-RL design space, distinct from PEAR (loss reweighting) and BRIDGE (data augmentation). Filters already-fit sequences during training so gradient updates concentrate on the under-modeled tail; targets pass@K coverage growth rather than pass@1. Reports up to +17% absolute pass@16 on OLMo-3 7B math/coding and up to +4% absolute pass@1 after identical GRPO. Also introduces a lightweight pre-run diagnostic for whether TailSFT will help in a given setting — the first filed predictability tool for this recipe family.
  • [2026-08-05] Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning: PEAR (Zhang et al.) provides the OPE-theoretic account of the “stronger SFT → weaker post-RL” phenomenon: reweight SFT loss by the target/behavior likelihood ratio (sequence-level, token-level suffix-ratio, or block-level), with numerical stabilization in log-space. Concurrent single-step-weighting recipes (iw-SFT, DFT, TALR, Proximal-SFT) improve offline pass@1 but rank-reverse post-RL — suffix-based continuation weighting is what actually matters. Up to +14.6% pass@8 on AIME-2025 across Qwen2.5/3 and DeepSeek-Distill 0.6B–8B under identical GRPO; transfers off-domain (SynLogic → Enigmata). Parameter-space analysis shows PEAR gradients have smaller principal angle with GRPO and smaller offline-online NSS drift than SFT.
  • [2026-08-05] Behavior Injection: Preparing Language Models for Reinforcement Learning: BRIDGE (Cen et al., NeurIPS 2025) reframes SFT as RL preparation and takes the data augmentation axis — inject explicit exploration/exploitation behaviors into the SFT dataset, motivated by a per-step RL-influence decomposition into rollout accuracy α and a data co-influence coefficient. BRIDGE has lower pre-RL accuracy than Vanilla/PP-Aug/RC-Aug baselines on PromptBench but wins final in-distribution + OOD scores after identical RFT — the first sharp empirical demonstration of the “weaker SFT wins post-RL” phenomenon.

Open questions

Section titled “Open questions”
  • Are the three axes (loss reweighting, data augmentation, sequence selection) additive, redundant, or substitutes? No filed paper stacks two of them under matched RL compute.
  • Does the sequence-selection recipe compose with the loss-reweighting recipe by construction (i.e. selecting the tail and then reweighting the tail’s tokens by importance ratios), or does the coverage gain from filtering already realize the same effect PEAR’s suffix-ratio weighting reaches on the full stream?
  • Is the “stronger SFT ≠ better post-RL” phenomenon a property of the SFT objective (token-level NLL under μ) or of the SFT data (behavior policy μ mismatched from target π_θ)? PEAR treats it as the former, BRIDGE treats it as the latter, TailSFT treats it as a coverage-vs-accuracy tradeoff — three consistent framings but no head-to-head ablation isolates the dominant cause.
  • Does the “pass@K coverage predicts post-RL performance” claim from TailSFT: Filtered Fine-Tuning Improves Post-Training Performance generalize beyond math and coding, and beyond OLMo-3 7B? PEAR and BRIDGE report post-RL wins without explicitly framing coverage as the intermediate objective.
  • Does the TailSFT diagnostic that predicts “will TailSFT help here?” generalize to predicting whether PEAR or BRIDGE will help — i.e. is there a shared pre-run signature for the three-axis family?
  • How does this thread interact with on-policy distillation (On-Policy Distillation, MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training)? OPD is a different offline recipe that also targets the offline-to-online distribution mismatch (via student rollouts + dense teacher supervision); PEAR/BRIDGE/TailSFT sit upstream of OPD and RL.
  • Robot RL transfer: the phenomenon is theoretically severe in robotics (teleop/scripted controller as μ, stochastic simulator-noise policy as π_θ), but no filed paper reports SFT-for-RL-readiness experiments on VLA / manipulation policies. Which of the three axes ports first?

Papers

Section titled “Papers”

2026-08

Section titled “2026-08”

2026-02

Section titled “2026-02”

2025-05

Section titled “2025-05”