Skip to content
Bud's Wiki

Bridging Offline and Online Reinforcement Learning for LLMs

TL;DR

Section titled “TL;DR”

A FAIR/NYU study that sweeps the offline → semi-online → fully-online axis for LLM post-training, comparing DPO and GRPO on a verifiable task (NuminaMath) and a non-verifiable one (WildChat + Athene-RM-8B reward model) with Llama-3.1-8B-Instruct as the seed. Headline finding: offline DPO lags badly, but semi-online DPO with periodic generator/reference sync (s=100, even s=32 steps between syncs) matches fully-online DPO and GRPO on both task types — i.e. the “fully online” requirement of recent RLHF folklore is not load-bearing. They also show that joint training on verifiable + non-verifiable rewards in one run beats single-task training on the non-verifiable side, without hurting math much, suggesting the two reward shapes are complementary rather than competing.

Key claims

Section titled “Key claims”
  • Offline DPO substantially underperforms every online or semi-online variant on both task types: on math, MATH500 jumps 53.7 → ~58–59 going from offline to semi/online; on WildChat, AlpacaEval LC winrate (GPT-4-1106) jumps 53.2 → 81.6 / 83.1 [Table 1, Table 2].
  • Semi-online DPO with periodic sync (s=100 generator+reference updates on math, smaller s on WildChat) matches fully-online DPO on both verifiable and non-verifiable tasks; the gap between s=100 and s=1 (fully online) is within noise [Table 1, Table 2, §4.1].
  • Online DPO marginally outperforms GRPO in both task settings (e.g. AlpacaEval LC 83.1 vs 75.2 with GPT-4-1106 judge on WildChat) — i.e. for this scale / setup the loss choice between DPO and GRPO matters less than online-ness [Table 1, Table 2].
  • GRPO is restricted to purely on-policy here because the standard PPO importance-sampling proof uses a single-step advantage, while GRPO’s advantage is sequence-level; the authors decline to do off-policy GRPO until the proof is extended [§2.1].
  • Scaling GRPO group size beyond a modest number does not help further on math [Appendix Table 5].
  • Joint training on verifiable (NuminaMath) + non-verifiable (WildChat) rewards from the seed model in one run reaches MATH500 57.3 / AlpacaEval LC 65.6 / ArenaHard 57.1 — preserving most math gain while exceeding WildChat-only on non-verifiable evals [Table 3].
  • Cross-task zero-shot transfer fails: a WildChat-only DPO checkpoint drops MATH500 from 47.4 → 35.0 (chasing the reward-model signal at the cost of math), and a NuminaMath-only checkpoint only marginally improves AlpacaEval (32.0 → 36.2) [Table 3]. Mixing rewards in training is the load-bearing move.
  • Entropy collapse is observed during online DPO and GRPO on the verifiable task regardless of algorithm, except for offline DPO; entropy regularization at multiple coefficient scales did not cleanly fix it [Fig. 3, Fig. 6, §4.2].
  • A bimodal response-length distribution (very-short vs. very-long) correlates with response-length collapse and performance degradation when reference-model sync is disabled or training speed is increased [Fig. 2, Fig. 9, §4.2].
  • The NLL-over-chosen auxiliary term that Iterative DPO (Pang et al. 2024) and ORPO use does not help in this semi-online setup; the authors attribute this to chosen log-probs not decreasing substantially in their runs, removing the failure mode that NLL was patching [Fig. 10, §4.2].
  • A “GroupDPO” loss that averages DPO over all correct×incorrect pairs from a GRPO-style group does not beat single-pair random sampling on the verifiable task [Eq. 4, Fig. 8].

Method

Section titled “Method”

The paper holds the seed model (Llama-3.1-8B-Instruct), evaluation suites (Math500/NuminaMath/AMC23 for verifiable; AlpacaEval 2.0 LC + ArenaHard with both GPT-4-1106 and GPT-4o judges for non-verifiable), and reward instrumentation (Math-Verify for verifiable; Athene-RM-8B as the open-source RM for non-verifiable) fixed, and varies (a) the loss — offline DPO, semi-online DPO at sync rate s, online DPO, GRPO — and (b) the task — NuminaMath math vs. WildChat instruction following, individually and combined. For DPO, preference pairs are constructed online from the current generator’s rollouts: best-vs-worst on the RM-scored WildChat pool, and a random correct vs. random incorrect for math (skipping prompts where the pool is all-correct or all-incorrect, which is also what zeroes out the GRPO advantage). Semi-online means the generator (and optionally the reference) are re-synced to the training model every s update steps, sweeping s from 1 (fully online) to 100/32 (semi-online) to ∞ (offline). The training stack is fairseq2 + vLLM on 32×H200 trainers + 8×H200 generators (16 generators for combined-task runs).

Results

Section titled “Results”
  • Verifiable math (NuminaMath training, Llama-3.1-8B-Instruct seed) [Table 1]: Seed MATH500/NuminaMath/AMC23 = 47.4 / 33.9 / 23.7. Offline DPO → 53.7 / 36.4 / 28.8. Semi-online DPO (s=100) → 58.9 / 39.3 / 35.1. Online DPO → 58.7 / 39.6 / 32.9. GRPO → 58.1 / 38.8 / 33.6. The semi-online and online runs cluster within ~1 point on MATH500 and within standard error on the others.
  • Non-verifiable WildChat [Table 2]: AlpacaEval LC (GPT-4-1106): Seed 27.3 → Offline 53.2 → Semi-online 81.6 → Online 83.1 → GRPO 75.2. ArenaHard (GPT-4-1106): Seed 32.0 → Offline 39.4 → Online 62.8 → GRPO 59.1. Online DPO has the highest AlpacaEval LC; GRPO is best on ArenaHard under the GPT-4o judge (54.3).
  • Joint training [Table 3]: starting from a WildChat-DPO checkpoint and finetuning on NuminaMath gives the best AlpacaEval LC (71.9) and ArenaHard (62.3) of any single-run config, while only losing ~4 points MATH500 vs. NuminaMath-only. Starting from the seed and training on the mixed batch (NuminaMath + WildChat, ~2:1 due to skipped math prompts) gets MATH500 57.3 / AlpacaEval LC 65.6 / ArenaHard 57.1 — i.e. competitive on math and better than WildChat-only on AlpacaEval.
  • Failure modes: response-length collapse (math) when ref-sync is disabled; length inflation (WildChat) from RM length-bias hacking, mitigated by length-normalized checkpoint selection; entropy collapse in all online runs except offline DPO; bimodal length distribution as a predictor of collapse [§4.2, Figs. 2, 3, 6, 7, 9].

Why it’s interesting

Section titled “Why it’s interesting”

This is the cleanest filed paper-level evidence that the “fully online” requirement of post-training is overstated for the LLM-RL recipes that the Reasoning RL cluster is converging on — semi-online DPO with infrequent sync matches GRPO and online DPO on both math and instruction-following while admitting embarrassingly-parallel rollout generation. That has direct implications for compute economics of the recipes catalogued under Reasoning RL (especially streaming-rollout-heavy ones like MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling‘s long-horizon ReAct trajectories), and it weakens the case for the algorithmic specialness of GRPO vs. semi-online DPO at this scale. The joint verifiable + non-verifiable result is also a counterpoint to the “RM-only training drops math” finding that Incorrect Baseline Evaluations Call Into Question Recent LLM-RL Claims has been pushing skepticism around — same direction, but here the mix is what saves you. The entropy-collapse observation echoes the broader stability-and-collapse thread the wiki has been tracking from a pretraining angle (Why Gradients Rapidly Increase Near the End of Training, Generative Modeling via Drifting) and gives an RL-side data point that scheduler/regularizer choices that work in pretraining don’t transfer straightforwardly.

See also

Section titled “See also”