Defeating the Training-Inference Mismatch via FP16

TL;DR

A one-line-of-code intervention against a recurring RL fine-tuning failure mode: switch the model from BF16 to FP16. The authors (Sea AI Lab + NUS) argue that the “training–inference mismatch” — the small log-probability gap between the rollout engine (e.g. vLLM) and the gradient engine (e.g. FSDP) that destabilizes RL post-training of LLMs — is fundamentally a precision problem, not an algorithmic one. BF16’s 7-bit mantissa rounds aggressively; FP16’s 10-bit mantissa does not. Reverting to FP16 with standard dynamic loss scaling reduces the sequence-level log-probability ratio mismatch by ~24× and yields stable optimization, faster convergence, and higher final reward across multiple RL algorithms (GRPO, GSPO, DAPO), models, and frameworks (VeRL, Oat).

Key claims

Root cause of the training-inference mismatch is floating-point precision, not algorithmic — the BF16 7-bit mantissa introduces large rounding errors that diverge between the training pass and the inference pass; FP16’s 10-bit mantissa eliminates the gap [§1, §2].
FP16 reduces the sequence-level log-probability ratio mismatch between inference policy µ and training policy π by approximately 24× compared to BF16 [Fig. 2].
The fix is “a few lines of code change” — no model architecture changes, no learning-algorithm modifications, no new RL objective — relying on already-mature dynamic loss scaling to handle FP16’s narrower dynamic range [§1, §2].
FP16 yields more stable training reward curves than BF16 across diverse settings: validated on two independent RL frameworks (VeRL and Oat) [Fig. 1].
The recipe generalizes across RL algorithms — improvements hold under GRPO, GSPO, and DAPO — and across diverse base LLMs and tasks [Abstract, §4].
BF16’s larger dynamic range, which motivated its adoption for pretraining, is not the binding constraint at RL post-training time because the policy weights are already converged and bounded; precision fidelity is what matters [Abstract — “range resilience vs. precision fidelity”].
Prior fixes that target the symptom (importance-ratio corrections, truncated importance sampling, engineering alignments between rollout and training engines) become unnecessary when the underlying numerical mismatch is removed [§1, §2].

Method

The intervention is to perform RL fine-tuning entirely in FP16 (both training-engine forward/backward and inference-engine rollout) rather than the BF16 default that ships with most modern frameworks. The authors keep everything else fixed — same RL algorithm, same rollout engine, same model architecture, same learning rate, same KL schedule. Dynamic loss scaling (now standard in PyTorch / Apex) handles FP16’s smaller dynamic range automatically by tracking gradient overflow/underflow and adjusting a scaling factor. The empirical evidence is twofold: (i) direct measurement of the token-level probability distribution and sequence-level log-probability ratio mismatch between inference and training policies (Fig. 2) shows FP16 collapses the distribution onto the no-mismatch line; (ii) end-to-end training reward curves across multiple algorithms and frameworks (Fig. 1, Table 1+) show FP16 trains stably where BF16 either trains slowly, plateaus, or collapses. Code released at github.com/sail-sg/Precision-RL.

Results

~24× reduction in sequence-level log-probability ratio mismatch under FP16 vs BF16 [Fig. 2].
Training reward is consistently higher and more stable under FP16 across GRPO / GSPO / DAPO on VeRL and Oat frameworks [Fig. 1].
Results reproduce on two independent RL frameworks (VeRL, Oat), arguing the effect is framework-agnostic [§4].
No quantified throughput penalty reported relative to BF16 (FP16 GEMMs are equally well-supported on H100/A100 hardware).

Why it’s interesting

This sits directly in the Training stability at scale cluster’s “stability comes from removing the right tuning knob” thread — like QK-Clip: Taking Muon Further on the Scaleup Journey (QK-Clip eliminates the spectral-norm tuning axis) and Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method (Quantile Balancing eliminates the MoE aux-loss tuning axis), but the knob it removes is precision format and the failure mode it defeats is RL-specific rather than pretraining-specific. It pairs naturally with Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention, which mechanistically explained how BF16 addition rounding in Flash Attention’s P V corrupts gradients during pretraining; this paper extends that “BF16 has a real precision deficit at the mantissa” story from pretraining to RL fine-tuning, where the symptom is the training-inference mismatch rather than mid-training loss explosion. It is also a direct counterpoint to the BF16-as-default consensus that hardened around DeepSeek-V3.1-Base — long-context-extended V3 base with UE8M0 FP8 microscaling‘s UE8M0 FP8 microscaling and Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling‘s NVFP4 work — those papers chase smaller formats for pretraining throughput, but FP16 reasserts that for the post-training regime, larger mantissa beats wider exponent. The Slack thread (@Terrance) flags the natural open question — does this transfer to diffusion RL post-training (DanceGRPO, HuDA on Wan 2.1 in Human detectors are surprisingly powerful reward models) and to distillation (DMD, FreeFlow), where the same “trained engine vs. inference engine” mismatch exists in different form?