Adaptive Block-Scaled Data Types (IF4 / IF3 / IF6)

TL;DR

IF4 is a new 4-bit block-scaled data type for LLM quantization that, per group of 16 values, picks at quantize time between FP4 (E2M1) and a scaled INT4 representation — whichever has lower error. The selection bit is stuffed into the sign of the FP8 E4M3 scale factor (unused in NVFP4), so IF4 has zero memory overhead vs NVFP4 while reducing mean squared quantization error from 9.0×10⁻³ to 6.2×10⁻³ on normally-distributed inputs [Table 1]. The same construction generalizes to IF3 and IF6. The authors design and synthesize an IF4 MAC unit in 28nm CMOS to argue this is implementable in next-gen accelerators with 4.7% per-MAC latency overhead [§5.2]. Of practical interest to anyone training in W4A4G4 (DeepSeek-V3-style FP4 pretraining) — IF4 closes most of the gap to BF16 in PTQ and outperforms NVFP4 / NVINT4 / MXFP4 / Four-Over-Six during 340M dense-Transformer pretraining.

Key claims

NVFP4’s quantization error is concentrated on the near-maximal values in each group of 16, a structural consequence of FP4’s non-uniform step sizes; this is what Four-Over-Six (4/6) tried to mitigate by sometimes scaling groups to a maximum of 4 instead of 6 [§1, Fig. 1].
4/6 has two intrinsic costs: scaling a group to max-4 wastes the FP4 codes for ±6 (2 of only 15 useful values), and forces the FP32 tensor-level scale to drop by 1.5×, cutting tensor dynamic range by 42.9% [§3].
IF4 replaces 4/6’s “scale to 4” branch with “represent the group as scaled INT4”. Both branches end up scaled to the same [0, 6] range — INT4 groups are pre-multiplied by 6/7 since INT4 spans [−7, 7] — so dynamic range is preserved at 100% of NVFP4’s [Table 1].
The branch choice (FP4 vs INT4) is encoded in the sign bit of the per-group E4M3 scale factor, which NVFP4 leaves unused; this is the structural trick that gives IF4 zero memory overhead [§3].
IF4 quantization error on N(0,1) inputs is 6.2×10⁻³ MSE vs 9.0 for NVFP4, 7.4 for NVINT4, 7.5 for NVFP4-with-4/6, and 13.2 for MXFP4 [Table 1].
On 340M dense Transformer pretraining with QK-Norm + Hadamard-on-weight-grads + stochastic-rounding-on-act-grads, IF4 beats NVFP4 / NVINT4 / MXFP4 / 4/6 on average across Qwen3.5 0.8B/9B/27B downstream-task averages, both with and without Hadamard-RTN [Table 3, §4.1].
Most of IF4’s pre-training win over NVFP4 traces to its preference for INT4 on Hadamard-transformed inputs — when the NVFP4 recipe is modified to quantize weight-grad inputs with NVINT4, performance approaches IF4; conversely IF4’s lead grows when paired with MS-EDEN, which Hadamard-transforms all backward-pass inputs [Fig. 4, Fig. 5(a)].
W4A4 post-training quantization on Nemotron-3 Nano (4B, 30B-A3B) and Qwen3.5 (9B, 27B, 35B-A3B, 122B-A10B, 397B-A17B): IF4 outperforms NVFP4, NVINT4, MXFP4, and 4/6 in nearly all WikiText-2 / C4 perplexity and 5-task accuracy comparisons; on Qwen3.5-122B-A10B, IF4 ties BF16 at 76.4 average [Table 2, Table 3].
IF (Int/Float) formats dominate the perplexity-vs-memory Pareto frontier across multiple bit-widths and block sizes — IF4 beats NVFP4 and NVINT4 at 4.5 bits, IF3 beats NVFP3/NVINT3 at 3.5 bits, IF3-BS8 beats NVFP3-BS8/NVINT3-BS8 at 4 bits [§5.1, Figs. 2 and 6].
The synthesized IF4 MAC adds ~4.7% latency over a plain NVFP4 MAC in 28nm CMOS, traced almost entirely to one extra FP32 multiplication on the range-alignment scaling path; impact at the system level is expected to be much smaller since modern AI workloads are memory-bound, not MAC-latency-bound [§5.2, Appendix C].

Method

IF4 inherits NVFP4’s structure: 16-value groups, one E4M3 (FP8) per-group scale, plus a tensor-wide FP32 scale. The novelty is per-group adaptivity. For each group, the encoder computes both an FP4 quantization and a scaled-INT4 quantization, picks whichever yields lower MSE against the original group, and writes the selection bit into the sign of that group’s E4M3 scale factor. Decoding flips on the sign: NVFP4 codes go through a small LUT (the 16 E2M1 values are non-uniform, LUT is the natural circuit); INT4 codes go through a shifter-based decode and pick up an extra ×(6/7) range-alignment factor so the dequantized values land on the same [0, 6] interval as the FP4 path. When both operands of a MAC happen to be INT4, the alignment factor becomes (6/7)² = 36/49; when only one is INT4, it’s 6/7; when both are FP4, no alignment is needed.

For training, the recipe follows NVIDIA’s NVFP4 pretraining recipe: quantize weights+activations+gradients in every linear except the last four hidden layers, apply the random Hadamard transform on weight-gradient computations to suppress outliers, use stochastic rounding when quantizing activation gradients, and apply a √2 factor on gradient scale factors (from Panferov et al. / MS-EDEN) to debias E4M3-induced stochastic-rounding bias. The forward pass dequantizes IF4 → FP32 → BF16 before each emulated matmul. For post-training quantization, the paper uses round-to-nearest in all linear + MoE layers except the LM head; weights and activations are both quantized.

The hardware feasibility argument is grounded in a SystemVerilog implementation of a 16-element IF4 MAC, synthesized in 28nm CMOS and benchmarked against a parallel NVFP4-only MAC. The decoded weight×activation product is held in FP16; per-group scale-factor products are computed once in parallel on a separate datapath and broadcast across the 16 multiplications; the range-alignment scaling is the only piece that needs FP32. Accumulation is FP32 until reset.

Results

W4A4G4 pretraining loss (340M dense, QK-Norm + Hadamard-RTN), average over Qwen3.5-0.8B/9B/27B downstream tasks [Table 3]:
- BF16: 65.7
- MXFP4: 60.6
- NVINT4: 62.8
- NVFP4: 63.3
- 4/6: 63.6
- IF4: 63.8
With Hadamard transform applied at quantization time: NVINT4 jumps to 63.8 and IF4 to 64.3 — the Hadamard-INT4 synergy is what IF4 captures structurally [Table 3].
W4A4 PTQ on Qwen3.5-122B-A10B: IF4 = 76.4 average across ARC-E/ARC-C/HellaSwag/LAMBADA/PIQA — matching BF16 (76.4) and beating NVFP4 (76.0), NVINT4 (76.0), 4/6 (75.7), MXFP4 (75.5) [Table 3].
W4A4 PTQ on Qwen3.5-35B-A3B (WikiText-2 / C4 perplexity): IF4 = 8.07 / 23.12, NVFP4 = 8.14 / 23.18, 4/6 = 8.11 / 23.11, NVINT4 = 8.31 / 23.63, BF16 = 7.70 / 22.17 [Table 2].
Quantization error on N(0,1) (MSE × 10⁻³) [Table 1]: IF4 6.2 vs NVFP4 9.0 — a 31% reduction with zero memory overhead.
Pareto frontier [§5.1, Figs. 2/6]: IF formats consistently sit below NVFP and NVINT curves across 9B, 27B, 35B-A3B, 122B-A10B Qwen3.5 sizes — IF4 at 4.5 bits/parameter dominates NVFP4 and NVINT4 at the same bit budget.
MAC area / latency [§5.2, Appendix C]: IF4 MAC ≈ 4.7% higher latency than NVFP4 MAC in 28nm CMOS; one extra FP32 multiply on the range-alignment path is the dominant overhead.
Code: https://github.com/mit-han-lab/fouroversix (shared with the prior Four-Over-Six paper).

Why it’s interesting

This is the natural next step beyond Four-Over-Six (same first author, same lab) and removes 4/6’s two structural costs — wasted FP4 codes and lost dynamic range — by changing the representation rather than the scale. The “spend the unused sign bit” trick is elegant and is what makes the proposal cleanly hardware-targetable for Blackwell-successor accelerators. Connects to Enabling Up to 41% Faster Pre-training: MXFP8 and DeepEP for DeepSeek-V3 on B200 with TorchTitan, which reports the open-stack +41% throughput win for MXFP8-on-grouped-GEMMs on DeepSeek-V3 671B — the same trajectory of “low-precision formats are the dominant lever for B200-era training”, except IF4 is the 4-bit story while that result was 8-bit-on-grouped-GEMMs only. IF4’s success on Hadamard-transformed inputs slots into the Training stability at scale thread that A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training opened — Gated Attention’s coupling between training-stability fixes and W4A4-friendliness predicts that every W4A4 format-design lever should be evaluated against post-Hadamard activation distributions, not raw ones; IF4 essentially confirms this prediction by structurally favoring INT4 on Hadamard-flat inputs and FP4 on raw outlier-heavy inputs.

Less directly comparable to Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments / TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate — those quantize the KV cache for inference (data-oblivious online scheme, no calibration), while IF4 quantizes weights+activations+gradients for end-to-end W4A4G4 training. The shared insight is that block-scaled formats need to be adaptive to the local distribution: TurboQuant by composing QJL + PolarQuant, IF4 by per-block FP4/INT4 selection.