FlashREINFORCE: Critic-Free Single-Rollout Asynchronous RL for Agentic Language Models

TL;DR

FlashREINFORCE is NVIDIA’s critic-free, single-rollout-per-prompt asynchronous RL recipe for agentic LMs: replace GRPO’s sibling-group baseline with a batch-mean baseline over independent prompts (one rollout each), keep token-level importance sampling for action correction, and add a sequence-level KL-proxy trust region plus sample-mean (not token-mean) loss reduction to handle the length-amplified negative-feedback and history-distribution-drift problems that appear when there are no sibling rollouts. The paper reports stable long-CoT training through 6,000 asynchronous updates at policy lag ≈4 on R1-Distill-Qwen-1.5B, +1.7 points over the GRPO baseline on Qwen2.5-Math-1.5B with half the rollouts, and stable tool-use RL on the 30B MoE Qwen3-30B-A3B at policy lag 8 where GRPO’s group-synchronization barrier is expensive.

Key claims

Group-based advantage estimation (GRPO) has three costs at fixed rollout budget: it limits prompt coverage (a budget of G rollouts covers only G/k prompts for group size k), introduces a group-synchronization barrier for asynchronous workers, and is difficult to apply to online interactions that yield a single trajectory [§1].
FlashREINFORCE’s one-batch REINFORCE with reward centering — advantage A_i = R_i − mean(R over batch of B independent prompts) — provides signed feedback in the single-rollout regime; the positive-only ablation (no negative feedback) collapses to 0 tool calls and 9.8 mean accuracy on Qwen2.5-7B-Instruct tool use [Table 5a].
Under token-level IS alone, the surrogate-vs-true-return error is bounded by accumulated policy drift |J(π) − ℒ_μ(π)| ≤ A_max · T² · κ̄ where κ̄ is the mean per-step KL between learner and behavior — motivating a sequence-level mask that removes an entire trajectory’s contribution rather than individual token terms [§2.2, Prop. B.2].
The sequence trust region uses a per-token Bernoulli-KL proxy on the sampled action d_{i,t} = p log(p/q) + (1−p) log((1−p)/(1−q)), averaged along the trajectory, thresholded at δ; this reward-independent proxy uses only stored behavior probabilities (no full distribution storage) [§3.2, Eq. 6–8].
Sample-mean loss reduction — average within a trajectory before averaging across trajectories — weights each trajectory 1/B rather than T_i / Σ T_j, removing the explicit length multiplier on negative feedback; the token-mean ablation collapses into 3,947-token responses with 45.3% truncation vs 2,518 tokens / 17.2% truncation for sample-mean, at matched training budget [§3.3, Table 5b].
Sequence-level admission is not equivalent to token-level admission at the same threshold: on Qwen2.5-Math-1.5B, token-local masking collapses at step 1,237 (train reward 0.18→0.01, entropy 0.37→5.6) while sequence-level admission remains stable through 2,160 updates — token gating leaves stored histories used by other terms intact, sequence rejection removes the full contribution [Table 6].
Fresh-batch updates (128 completed trajectories per update) beat sequential four-minibatch schedules (512 trajectories, four updates on drifted learner) by 1.65 points at equal data and optimization budget — the drift from sequential updates on the same rollout batch is measurable [Table 4].
Scaling results: 6,000 asynchronous updates at lag ≈4 on R1-Distill-Qwen-1.5B lifting AIME24/25 21.7→33.7 [Fig. 2]; 38.0 five-benchmark mean on Qwen2.5-Math-1.5B with 256k rollouts vs GRPO’s 36.3 with 512k [Table 1]; +6.7 points and 3.25 tool calls/trajectory (vs GRPO’s 0.00 — GRPO stops calling the tool) on Qwen2.5-7B-Instruct at step 600 [Table 2a]; stable 30B MoE training through 1,450 updates at lag 8, +6.8 points over GRPO at matched 102.4k-rollout budget [Table 2b].
ALFWorld: 98.3%/96.5% seen/unseen at G=1 vs the strongest published baseline C-RF+NTF (also G=1) at 90.5%/86.3%; prompt diversity alone does not explain the gap since both are single-rollout [Table 3].
Joint ablation on ALFWorld: removing both token IS and sequence trust drops step-40 seen/unseen success from 81.0/77.6 to 35.0/27.6 — behavior correction and sequence trust contribute jointly, not additively, at the operating point [§4.3.3].
The KL-proxy trust region does not have to be tight: on Qwen2.5-Math, threshold δ=3×10⁻³ rejects only 0.13% of trajectories and still preserves stable training over 4,000 steps — the gate screens drift outliers rather than restricting ordinary IS corrections [§4.3.3, §A.3.3].

Method

Setup. Rollout workers periodically pull policy snapshots and independently submit one completed trajectory per prompt as it finishes. The learner batches the next B completions (which may span several behavior snapshots) and executes a single optimizer step per fresh batch. For each token, the stored behavior probability μ_i(a_i,t | h_i,t) is kept as-is (not recomputed) to avoid training-inference-mismatch drift.

Three components (Fig. 1 in paper).

One-Batch REINFORCE with reward centering. Compute Ā = mean(R_j over batch B) and per-trajectory advantage A_i = R_i − Ā. Each fresh batch supplies its own baseline, receives one update, and is discarded — no replay, no sibling-group wait.
Sequence trust region. For each token, form the Bernoulli-KL proxy in Eq. (6), average along the trajectory to get D̄_i, and admit the trajectory iff D̄_i ≤ δ (Eq. 8). Rejection is stop-gradient at trajectory granularity.
Sample-mean optimization. The final loss is

ℒ(θ) = −(1/B) Σ_i m_i · (A_i / T_i) · Σ_t stopgrad(ρ_{i,t}) · log π_θ(a_{i,t} | h_{i,t})

where ρ_{i,t} = π_θ / μ_i is the token IS ratio (with stopgrad giving the same gradient as the detached-ratio implementation). The 1/T_i inside the sum plus 1/B outside is the sample-mean weighting; token-mean would replace this with 1/Σ_j T_j.

No critic, no clipping, no reference-model forward pass. The KL-proxy gate replaces PPO’s clipping-based trust, and the batch-mean advantage replaces both GRPO’s sibling baseline and a learned value.

Results

Long-CoT training. R1-Distill-Qwen-1.5B trained on DPPO’s 1,460-problem MATH subset at policy lag ≈4 with 128 trajectories per update; AIME24/25 mean avg@32 rises 21.7→33.7 over 6,000 cumulative updates, with a new AIME24 peak of 38.85 at step 5,896 in the with-trust continuation [Fig. 2, Table 9].

Critic-free reasoning-baseline comparison. Qwen2.5-Math-1.5B, five-benchmark mean at step 2,000: FlashREINFORCE 38.0 with 256k rollouts vs GRPO 36.3 and C-RF+NTF 35.3 at 512k [Table 1]. Task-level split: Minerva +4.6 over GRPO, Olympiad +9.6, AIME25 +0.8; MATH-500 and AMC23 slightly behind GRPO but ahead of C-RF+NTF.

Python-tool RL. Qwen2.5-7B-Instruct at step 600: mean 37.0 vs GRPO 30.3, tool calls 3.25 vs 0.00 (GRPO stops calling within ~200 steps) [Table 2a]. Qwen3-30B-A3B (MoE) at step 800 with δ=10⁻² at lag ≈8: mean 67.1 vs GRPO 60.3 at lag ≈1 [Table 2b].

ALFWorld. 12.8k training trajectories, step 200: 98.3% seen / 96.5% unseen — +7.8 / +10.2 over C-RF+NTF (also G=1) and +19.7 / +19.7 over GRPO (G=8) [Table 3].

Ablations. Signed vs positive-only feedback: 36.2 vs 9.8 mean, 3.30 vs 0.00 tool calls [Table 5a]. Sample-mean vs token-mean continuation: reward 0.477 vs 0.403, response length 2,518 vs 3,947 tokens, truncation 17.2% vs 45.3% [Table 5b]. Sequence- vs token-local trust: stable through 2,160 steps vs collapse at 1,237 [Table 6]. Sequence trust with vs without: 32.2% vs 30.6% AIME24/25 at step 4,992; continued training without trust yields no further improvement (30.6% still), with trust reaches 33.7% [Table 7].

Why it’s interesting

Sits directly against the GRPO-as-default framing that dominates Reasoning RL: FlashREINFORCE keeps the group-relative-advantage skeleton (batch-mean baseline) but drops the sibling-rollout requirement, so it commutes with asynchronous single-trajectory workers and applies to online interactions where GRPO structurally cannot. Complements NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4‘s async-GRPO stage in the Nemotron 3 Ultra post-training pipeline — same async-training-decoupled-from-inference direction, but FlashREINFORCE argues the group-baseline itself should go in the async regime, not just the sync barrier around it.

Two claims are of particular interest to the Luma team. First, the sample-mean-vs-token-mean loss-reduction ablation ([Table 5b]) is the sharpest filed reproduction of the token-aggregation debate raised in Training frontier knowledge work agents: A 397B RL training guide with SkyRL‘s prompt_mean vs token_mean finding, but where Mercor showed +3.9 pts on knowledge-work tasks, this paper shows the failure mode: token-mean at long-CoT causes a coupled length-blowup + truncation-rate collapse (2,518→3,947 tokens, 17.2%→45.3% truncation) rather than a small accuracy hit. Second, the sequence-level KL-proxy trust region is a concrete alternative to PPO/CISPO-style ratio clipping for the drift-outlier problem, and the ablation shows it’s a strictly different intervention from token-level masking at the same threshold — a datapoint for anyone tempted to substitute per-token importance-weight clipping for trajectory-level admission.