Skip to content
Bud's Wiki

Behavior Injection: Preparing Language Models for Reinforcement Learning

TL;DR

Section titled “TL;DR”

BRIDGE (Behavior Injection Data augmentation for RL-readiness of Generative pipelinEs) is a data-centric SFT-stage recipe that prepares LLMs for reinforcement fine-tuning (RFT) by seeding exploratory and exploitative behaviors into the SFT dataset rather than optimizing for SFT accuracy. Motivated by the empirical observation that some LLMs improve substantially under RFT while others plateau or degrade, the authors analyze the per-step RL objective and identify two governing factors — rollout accuracy and a data co-influence coefficient — and construct SFT data that jointly targets both. On iGSM and PromptBench with Qwen-2.5 and Llama-3.2 bases, BRIDGE-initialized checkpoints have lower SFT accuracy than baselines like Premise-Permutation Augmentation and Reasoning-Chain Augmentation, but consistently deliver the largest ΔRFT gains, winning the final in-distribution and OOD scores.

Key claims

Section titled “Key claims”
  • LLMs respond inconsistently to identical RFT: same recipe produces substantial improvement in some models and stagnation or degradation in others [§1].
  • The per-step influence of the RL objective decomposes into two factors that jointly govern RFT improvement: sampling rollout accuracy α and a data co-influence coefficient that measures how strongly RL training data affects generalization [§Insights, Eq. 5].
  • Behavior injection augments SFT data with explicit exploration and exploitation behaviors, producing a “behavior-injected SFT model” that is more RL-ready than a “behavior-scarce” one trained on the original data [Fig. 1].
  • BRIDGE has relatively lower pre-RL accuracy than baselines on PromptBench but obtains “significantly more remarkable performance growth” during RL, achieving the best final score on both in-distribution and OOD sets [Table 2].
  • BRIDGE has the fastest reward growth during RL training across both iGSM and PromptBench, which the authors attribute jointly to sampling accuracy and to the data co-influence coefficient [Fig. 3].
  • Baselines evaluated: Vanilla SFT, Premise Permutation Augmentation (PP-Aug), and Reasoning Chain Augmentation (RC-Aug); BRIDGE beats each on post-RL performance despite not beating each on SFT-stage accuracy [Tables 1–2].

Method

Section titled “Method”

The paper first derives an approximate per-step influence expression ΔJ for the RL update on a given evaluation set, which factorizes into (i) a rollout-accuracy term α (probability the SFT policy produces correct rollouts on RL training prompts) and (ii) a data co-influence term that captures how updates on one training instance affect generalization to held-out prompts.

Given this diagnosis, BRIDGE constructs SFT data whose trajectories exhibit both exploratory behaviors (trying alternative paths, hypothesizing subgoals) and exploitative behaviors (committing to a promising path and finishing it). The augmented dataset is then used for standard SFT, with the RFT stage unchanged from baseline. The pipeline overview in Fig. 1 contrasts the “behavior-scarce” trajectory (original SFT data) with the “behavior-injected” trajectory (augmented data), and shows how BRIDGE sits between the base model and the downstream RL stage as a data-augmentation intervention on the SFT corpus, not a modification of the RL algorithm.

Evaluation is done on iGSM (grade-school math synthetic reasoning environment) and PromptBench (arithmetic + logical reasoning), which are chosen specifically to disentangle problem-solving ability from knowledge storage and to avoid data contamination. Bases are Qwen-2.5 and Llama-3.2. SFT is run for 5 epochs, RFT for 100 steps (Qwen 3B) or 200 steps (Qwen 1.5B, Llama 1B). Reported accuracies are strict-match on both in-distribution and OOD test sets.

Results

Section titled “Results”
  • On iGSM and PromptBench, BRIDGE consistently produces the largest RFT-stage gain Δ over the SFT model relative to Vanilla / PP-Aug / RC-Aug baselines [Tables 1–2].
  • On PromptBench specifically, BRIDGE has lower pre-RL accuracy than PP-Aug and RC-Aug but ends up best-in-class after RFT on both in-distribution and OOD [Table 2, quoted in Slack post].
  • BRIDGE has the fastest reward growth curve during RL training, which the authors decompose into contributions from both α (sampling accuracy) and the data co-influence coefficient — the analysis in §3 predicts this and the empirical curves in Fig. 3 confirm it.
  • Behavior injection helps across both Qwen and Llama bases and across both benchmarks, consistent with the paper’s claim that the failure mode it targets (“some models plateau under RFT”) is a data-side rather than model-side pathology [Tables 1–2].

Why it’s interesting

Section titled “Why it’s interesting”

This is one of two papers Sirou posted together and belongs to a fast-growing thread on the wiki that reframes the SFT stage as preparation for RL, not as an end in itself. Its central empirical finding — that BRIDGE has lower SFT accuracy but wins after RL — is exactly the “stronger SFT checkpoint can underperform weaker one after identical RL” phenomenon that Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning (the second Sirou paper) uses to motivate PEAR. The two papers approach the same problem from complementary angles: BRIDGE modifies the data (inject exploratory/exploitative trajectories); PEAR modifies the loss weighting (importance-weight tokens by target/behavior likelihood ratio). Both explicitly argue that pre-RL accuracy is a misleading proxy.

Also complements Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe and The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation, which analyze policy-distillation failure modes with mutual-information tooling, and Spurious Rewards: Rethinking Training Signals in RLVR and Understanding Reasoning from Pretraining to Post-Training, which argue RLVR elicits pretraining behaviors rather than installing new ones — BRIDGE takes this seriously by pre-installing the target behaviors during SFT. For Luma’s robot RL work (per Sirou’s Slack note), the recipe is directly transferable: instead of maximizing BC accuracy on the training set, augment the demonstration corpus with explicit exploration/exploitation behavior traces that the downstream RL stage can then amplify.

See also

Section titled “See also”