The 4-bitter Lesson: Balancing Stability and Performance in NVFP4 RL

TL;DR

humans& (with RadixArk and NVIDIA) release the first open-source end-to-end NVFP4 RL recipe that trains stably at frontier scale on Qwen3-30B-A3B with DAPO-math-17k. The recipe combines three interventions layered on top of NVIDIA’s NVFP4 pretraining recipe: (1) dequantized backward — the BF16 backward pass consumes DQ(Q(w)) rather than the raw BF16 weight, so gradient chain-rule matches the quantized forward; (2) Four-Over-Six (4/6) applied to both weights and activations, with a bit-exact FP16 fast-path kernel giving ~2.8× speedup and >99.97% scale-choice agreement vs the reference implementation; (3) selective high precision — the shared MoE expert and last ~15% of layers stay in BF16, with the recipe respected across checkpoint, training, rollouts, and weight sync. The combined recipe preserves BF16-quality reward curves while enabling NVFP4 both in trainers and samplers, and the same quantization path is used to serve trained checkpoints as online NVFP4 (0.924 correlation with FP8 deployment on a GLM-5.1-judged multi-turn benchmark).

Key claims

The NVFP4 pretraining recipe is not sufficient for RL because RL has a different bias–variance profile — the policy gradient is already noisy from rollouts, advantage estimation, rewards, KL, and staleness, so quantization must be low-error, not merely unbiased, and stochastic rounding (fine for pretraining) trades bias for variance that RL cannot absorb [§Baseline — Why is the NVFP4 pretraining recipe not sufficient for RL?].
Per-tensor NVFP4 activation scales create two RL-specific pathologies: (a) the same token quantizes differently depending on which other tokens are in the batch, and (b) later tokens in a sequence can leak information into earlier tokens via the shared scale — the recipe uses per-token FP32 activation scales (following Composer 2 Technical Report) computed inside the fused quantization kernel [§Baseline recipe].
Only the MoE layers are aggressively quantized (they carry ~97% of parameters in DeepSeek-V3-style architectures — 656B of 671B in the reference calculation), with all other layers in BF16; the baseline keeps the backward pass in BF16 as a conservative starting point [§Baseline recipe].
Dequantized backward — replacing y = x · w_bf16 in the backward chain with y = x · DQ(Q(w_bf16)) — resolves the forward/backward mismatch that produced gradient-norm spikes under the “quantized forward, BF16 backward” baseline while preserving BF16 backward stability [§Improving Gradient Stability].
Validated first with MXFP8 (finer FP8 grid makes the mechanism cleaner): dequantized backward gives cleaner, less noisy gradients than both fully-MXFP8 backward and full-precision BF16 backward throughout training [§Validation with MXFP8, Fig. showing three-recipe gradient norm].
Under NVFP4, dequantized backward reduces gradient bias (backward matches forward) but the coarser 4-bit grid raises gradient variance; under vanilla SGD this trade lets learning go faster but occasionally diverges. With Adam’s momentum + adaptive second moment absorbing the variance, dequantized backward produces “far fewer large gradient spikes” than the higher-precision BF16 backward baseline [§Validation with NVFP4].
Four-Over-Six (4/6) from Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling is applied to both weights and activations (the original paper applied it only to activations in pretraining); in RL, starting from already-trained weights makes weight quantization error disproportionately impactful [§Four-Over-Six for RL Weights and Activations].
The reference 4/6 selection kernel is FP32-scalar-arithmetic-bound and stalls; the humans& implementation performs the FP4→FP16 dequant and FP4×FP8 multiply losslessly in FP16 using PTX cvt.rn.f16x2.e2m1x2, cvt.rn.f16x2.e4m3x2, and mul.rn.f16x2 (BF16 lacks the mantissa bits for exact E2M1×E4M3), scales the target down by inverse of amax/(E2M1_MAX·E4M3_4OVER6_MAX) once instead of scaling each candidate up twice, and keeps error accumulation in FP32 — yielding ~2.8× speedup and >99.97% scale-choice agreement with the strict FP32 reference [§Four-Over-Six for RL Weights and Activations].
Adding 4/6 does not measurably increase rollout time because the extra work is fused into the quantization kernel and hidden under existing memory-access latencies [§Fig. 15, Four-Over-Six for RL].
4/6 must be a bit-exact contract across trainer (TransformerEngine) and sampler (FlashInfer): quantized values, scales, and error accumulation/comparison must be bitwise identical regardless of weight layout or swizzling, otherwise the trainer/sampler mismatch reopens under RL [§Four-Over-Six for RL — bit-exact contract].
Adding 4/6 meaningfully stabilizes reference KL vs NVFP4 without 4/6 during RL [§Fig. 17].
Selective layer precision: keeping ~15% of final layers in BF16 (per the NVIDIA NVFP4 pretraining paper) and keeping the always-active shared MoE expert in higher precision is a favorable memory/logprob trade for MoE architectures — the shared expert runs on every token so its precision cost per parameter is high, but keeping it precise is disproportionately impactful [§Selective Layer Precisions]. Precision selection must be respected across checkpointing, training, rollouts, and weight updates.
The dequantization overhead reduction merged into TransformerEngine (PR #2865, joint with NVIDIA) avoids storing the extra quantized tensor copies that lower-precision training normally keeps, cutting peak memory of a 2k×2k → 8k NVFP4 linear from 150 MB to 45 MB (–70%) [§Improving Gradient Stability — Implementation].
Any two out of three interventions still show gradient spikes; all three combined produce a “remarkably stable” gradient norm across five runs whose reward curves closely track the BF16 reward curve [§Putting it all together].
The same rollout quantization path enables online NVFP4 serving — post-training checkpoints are quantized live with no separate calibration/QAT/model-conversion pipeline; on an internal GLM-5.1-judged multi-turn tool-use benchmark, correlation between the FP8 deployment and the NVFP4 quantized deployment is 0.924 [§A Pleasant side effect: Online NVFP4 Serving].
Configuration argument in SGLang is --quantization nvfp4_online on any existing BF16/FP8 model-serving command; the feature is open-sourced (PR #26083) [§A Pleasant side effect: Online NVFP4 Serving].
Position for future hardware: humans& want MXFP8 activations × NVFP4 weights — activations are harder to quantize than weights, and existing NVIDIA GPUs ship MXFP8×MXFP4 but MXFP4 weights lose precision because MXFP4 uses UE8M0 scales, so the target combination waits on future silicon [§Future Improvements].

Method

The system is a joint modification of trainer (Megatron-LM + TransformerEngine + Miles), sampler (SGLang + FlashInfer), and RL orchestration around long-horizon multi-agent rollouts. Model is Qwen3-30B-A3B; training data is DAPO-math-17k at 8k sequence length; the reference architecture is DeepSeek-V3-style MoE where 256 experts × 3 projections per layer × 58 layers = ~656B parameters (~97% of the 671B total) — MoE-only quantization already captures most of the memory benefit.

For each MoE linear the forward stores weights in NVFP4 (16-value blocks with per-block FP8 E4M3 scales and a per-tensor FP32 global scale) and quantizes activations per-token with an FP32 scale computed across the hidden dimension inside a fused kernel. The kernel emits packed FP4 values, the per-16-value FP8 scales, and the per-token FP32 scale in one pass, so quantization stays local to each token and shares no state across the batch.

The backward pass keeps operands in BF16 but reconstructs the dequantized quantized weight rather than using the pre-quantization BF16 weight, so the local Jacobian matches what the forward actually computed. In TransformerEngine this is exposed as a switch alongside the quantized-backward and vanilla-BF16-backward modes.

Four-Over-Six adds a second candidate quantization per block using amax/4 instead of amax/6, evaluates blockwise MSE (kept in FP32) against the original values, and picks the lower-error branch. The bit-exact fast-path fuses this into the same quantization kernel using PTX cvt instructions to keep dequantization and FP4·FP8 multiplication in FP16 registers, scaling the target down by the constant inverse instead of scaling both candidates up. The same numerical contract is implemented across TransformerEngine (training) and FlashInfer (inference) so the trainer’s and sampler’s quantized weights are bitwise identical.

Selective precision keeps (a) the last ~15% of transformer layers and (b) the shared/always-active MoE expert in BF16, with layout metadata propagated through the checkpoint format, weight update path, and rollout worker. Online NVFP4 serving reuses the trainer’s quantization kernel to convert a BF16/FP8 checkpoint at load time in SGLang.

Results

Gradient stability: with all three interventions (dequantized backward + 4/6 + selective precision), grad-norm curves across five NVFP4 runs are visually flat and match BF16; dropping any one intervention brings back visible spikes [§Putting it all together, Figs. of grad-norm five-run overlay].
Reward parity: reward curves for the five combined-recipe NVFP4 runs closely track the BF16 reference across the training window [§Putting it all together].
Rollout throughput: 4/6 adds no measurable rollout time; the fused kernel hides its extra work under existing memory-access latencies [§Fig. 15].
4/6 kernel: ~2.8× speedup over the naive/strict FP32-reference approach, >99.97% match rate on scale selection across error types and quantization modes on the reference benchmark [§Four-Over-Six for RL].
Peak memory: TransformerEngine NVFP4 linear peak memory reduces from 150 MB to 45 MB (–70%) on a proxy 2k×2k input / 8k output workload after the dequant-overhead PR [§Improving Gradient Stability — Implementation].
Online serving correlation: 0.924 correlation between FP8 and NVFP4-online multi-turn rollout scores under GLM-5.1-as-judge on an internal tool-use benchmark [§A Pleasant side effect].
Hardware ceiling (dense Tensor Core PFLOPS per GPU, humans&’ HGX-spec table): B200 2.25 (BF16) / 4.5 (FP8); B300 2.25 / 4.5 / 13.5 (FP4); Rubin NVL8 4 / 17.5 / 35 — motivating up-to-9× throughput at 4-bit vs 16-bit on Rubin [§Introduction, footnote 3].

Why it’s interesting

This is the first filed end-to-end NVFP4 RL recipe for a frontier-scale MoE LLM, closing three gaps at once. First, on the algorithm side, it operationalizes and validates dequantized backward — a chain-rule-consistency fix that’s absent from published NVFP4 pretraining work and that the paper argues is specifically load-bearing for RL because the noise budget is smaller than pretraining’s. Second, it lifts Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling out of pretraining (where 4/6 was applied only to activations) and pushes it into both weights and activations for RL, with a bit-exact contract across trainer and sampler that the original paper did not need. Third, it publishes the concrete kernel engineering — PTX-cvt-based FP16 fast path, per-token FP32 activation scaling fused into the quant kernel, ~70% peak-memory reduction from removing duplicate quantized tensor storage in TransformerEngine — that turns “NVFP4 works in principle” into “NVFP4 works in a shipping RL codebase.”

Against the wiki’s Reasoning RL cluster it is directly downstream of Composer 2 Technical Report, which independently converged on the same per-token FP32 activation scale (Cursor’s Composer 2 discovered that per-tensor NVFP4 scales diverge RL and inter-token scales bias gradients); humans& explicitly cites Composer 2 and extends the recipe to symmetric trainer/sampler contracts and to 4-bit quantization of both operands. Against Defeating the Training-Inference Mismatch via FP16 the tension is sharp: that paper argues RL post-training wants more precision than BF16 (upgrade to FP16 for +10 mantissa bits); this one argues RL can tolerate less than BF16 (downgrade weights to NVFP4) if the forward/backward chain rule is respected and 4/6 handles near-max values. Both may be simultaneously right — FP16 for the sensitive small last-layer parameters, NVFP4 for the fungible MoE bulk — and the “selective precision” thread in humans&’s recipe is a partial mediation of that debate.