TailSFT: Filtered Fine-Tuning Improves Post-Training Performance

TL;DR

TailSFT is a filtered-SFT recipe that drops already-fit sequences from the SFT training stream mid-run, concentrating gradient updates on the tail of the data distribution the model has not yet modeled. The framing: pass@K coverage of the SFT checkpoint predicts post-RL performance better than pass@1, so an SFT objective that grows coverage — rather than one that grows likelihood on already-fit examples — is a better initialization for subsequent RL. On OLMo-3 7B, TailSFT lifts pass@16 by up to 17% absolute on math and coding evaluations at minimal extra compute, and those higher-coverage checkpoints translate to up to 4% absolute pass@1 gains after identical GRPO. The paper also introduces a lightweight diagnostic for predicting when TailSFT will help.

Key claims

Pass@K coverage of an SFT checkpoint, not pass@1, is the load-bearing predictor of post-RL performance — SFT recipes optimizing pass@1 can plateau or hurt when RL is chained on top [§1].
TailSFT modifies standard SFT by filtering out sequences the model already fits during training, so gradient updates focus on the under-modeled tail of the data distribution [§Abstract, §Method].
Filtering criterion and its design choices are justified through controlled experiments and theoretical analysis (not merely one or the other) [§Abstract].
On OLMo-3 7B, TailSFT often improves pass@16 on math and coding by up to 17% absolute over baseline SFT [§Results].
The compute overhead of the filtering pass is minimal relative to the base SFT run [§Results].
Higher-coverage TailSFT checkpoints consistently translate to up to 4% absolute pass@1 gains after subsequent identical GRPO — i.e., the coverage gain is usable by downstream RL, not just an offline artifact [§Results].
The paper ships a lightweight diagnostic that predicts settings in which TailSFT is most likely to help vs. be neutral [§Abstract, §Discussion].
Broader position: intermediate SFT checkpoints should be judged by how well they support subsequent training stages, not by their standalone evaluation metrics — a stage-aware development philosophy [§Discussion].

Method

TailSFT is a drop-in modification of standard supervised fine-tuning. During training, each sequence is periodically scored under the current model; sequences whose loss (or some per-token confidence measure) falls below a threshold are considered “already fit” and are filtered from subsequent updates for the remainder of that epoch or scoring window. Training then continues on the remaining, under-modeled sequences — the tail of the data distribution the model has not yet absorbed. The specific filtering criterion is design-justified in the paper via controlled experiments plus theoretical analysis, but the mechanism is deliberately simple: no extra model, no reward shaping, no reweighting scheme, just a per-sequence pass/no-pass gate that concentrates gradient updates on the harder examples.

The setup is orthogonal to loss reweighting (PEAR-style importance ratios) and to data augmentation (BRIDGE-style behavior injection) — TailSFT edits which sequences the SFT loss sees, not how each token in a seen sequence is weighted and not what data enters the stream in the first place. The paper positions the recipe as coverage-focused rather than accuracy-focused: growing pass@K is the target, and the filter is the mechanism that redirects capacity from likelihood-sharpening on fit sequences to likelihood-installation on unfit ones.

The evaluation setup uses OLMo-3 7B as the base model and pairs SFT with a downstream GRPO stage under identical RL compute across TailSFT and baseline SFT initializations. Reported metrics are pass@1 and pass@16 on math and coding benchmarks, both at the SFT stage and after downstream GRPO.

Results

Pass@16 on math and coding evaluations: up to +17% absolute gain over baseline SFT on OLMo-3 7B [§Results].
Downstream RL (GRPO) after TailSFT initialization: up to +4% absolute pass@1 gain over the same RL applied to a baseline-SFT checkpoint [§Results].
Compute overhead of TailSFT filtering vs. baseline SFT: minimal — the filter reuses per-batch loss values [§Results].
A diagnostic is proposed that identifies (before the full RL run) which settings will benefit from TailSFT — turning “should we use TailSFT here?” into a pre-run check rather than an ablation [§Discussion].

Why it’s interesting

TailSFT is the third filed paper on the wiki explicitly framing SFT as preparation for RL rather than as an end in itself, and it stakes out a data-selection position that the other two — Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning (PEAR, loss reweighting via off-policy importance ratios) and Behavior Injection: Preparing Language Models for Reinforcement Learning (BRIDGE, data augmentation with exploration/exploitation behaviors) — do not. The three papers cover the natural design axes of the SFT-for-RL problem: PEAR modifies how each token in a seen sequence is weighted, BRIDGE modifies what data enters the stream, and TailSFT modifies which sequences the loss sees. Reading them together, the shared empirical thesis is sharp — pass@K coverage matters more for post-RL performance than pass@1, and standard SFT is systematically anti-aligned with that objective.

The most Luma-transferable contribution is the coverage-vs-accuracy reframing plus the lightweight diagnostic: the diagnostic converts “should we filter our SFT stream?” from an expensive ablation into a pre-run check, which matters for robot RL settings where each end-to-end SFT→RL cycle is costly and where the same underlying phenomenon — a strong BC/SFT checkpoint that stalls under downstream RL — has been observed in team practice (per the Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning pointer note). Complements On-Policy Distillation‘s SFT-as-off-policy framing by giving a concrete pre-RL surgery on the SFT distribution itself, and complements Spurious Rewards: Rethinking Training Signals in RLVR‘s “RLVR elicits pretraining behaviors” thesis by targeting the SFT stage as the place where those behaviors get their coverage.