Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding

TL;DR

Nemotron-Labs-Diffusion (NLD) is an NVIDIA tri-mode language-model family — 3B/8B/14B, in base / instruct / vision-language variants — where one set of weights can be served in three modes by changing only the attention pattern: standard AR (causal), parallel diffusion (bidirectional masked), and self-speculation (diffusion drafts, AR verifies, shared KV cache). No separate draft model, no architectural change, no extra weights. The 8B reports 5.9× tokens per forward over Qwen3-8B with matched accuracy and 3.3× wall-clock vs AR on GB200; the VLM 8B reaches 3.6–7.5× TPF with ≤0.1% accuracy drop versus AR. Released under the NVIDIA Nemotron Open Model License with HF weights, vLLM/SGLang/Docker Model Runner serving snippets, and a LoRA-enhanced drafter adapter for additional acceptance-length gains.

Key claims

A single set of weights, trained with a joint AR + masked-diffusion objective, can be deployed in three decoding modes — AR, parallel diffusion, and self-speculation — selected at inference by switching only the attention mask, with no architectural change and no auxiliary draft model [Abstract, §1, §3].
The AR and diffusion objectives are complementary on the same checkpoint: diffusion provides lookahead/planning, AR provides left-to-right linguistic priors; ablations on 25B-token continued pretraining show joint training beats either objective alone [§2, Table 1].
Loss-averaging strategy under the diffusion objective matters in the same way that GRPO-vs-DAPO loss aggregation matters for on-policy RL, because the number of masked tokens varies across sequences in a batch [§2, “Global loss averaging”].
Self-speculation — the same model drafts via diffusion and verifies via AR with shared KV cache — outperforms multi-token-prediction (MTP) approaches in both acceptance rate and real-device efficiency [Abstract, §3, §6].
A speed-of-light analysis under an optimal sampler shows current self-speculation samplers still leave parallelism on the table: an oracle sampler would correctly predict 76.5% more tokens per forward than the deployed self-speculation mode [Abstract, §4].
NLD-8B decodes 5.9× more tokens per forward than Qwen3-8B at matched accuracy, translating to ≈4× higher throughput on SPEED-Bench under SGLang on a GB200 [Abstract, §6].
Native self-speculation reaches 5.46 average acceptance length per draft step on SPEED-Bench; with a LoRA-tuned drafter it reaches 6.82, vs 2.75 for Eagle3 and 4.24 for Qwen3-9B-MTP [Tables, §6].
On diffusion-friendly categories (coding, math, reasoning, multilingual) the acceptance-length gap widens: 8.69 for NLD-LoRA vs 2.81 for Eagle3 [§6].
NLD-14B with LoRA-tuned linear self-speculation achieves 66.36% average accuracy at 5.96× TPF, exceeding Qwen3-14B’s 65.17% in pure AR mode [§6].
The vision-language variant (NLD-VLM-8B) reaches 3.63×–7.45× TPF in linear self-speculation, with the higher end on >200-token responses, at a 0.1% average accuracy drop vs AR [§6].
Real-device wall-clock: NLD-8B at concurrency 1 reaches 850 tok/s on GB200 (3.3× over 253 tok/s AR baseline and 2.4× over 360 tok/s Eagle3); custom CUDA kernels push this to 1015 tok/s (4×) [HF card §Highlights].
On DGX Spark, NLD-8B at concurrency 1 reaches 112 tok/s (2.7× over 41.8 tok/s AR) under w4a16 quantization [HF card §Highlights].
Self-speculation moves decode from memory-bound toward compute-bound: weights are loaded once per forward and reused to produce multiple committed tokens, increasing arithmetic intensity at low concurrency [HF card §Highlights].

Method

NLD trains a single transformer with a joint loss: a standard causal next-token AR loss plus a masked-diffusion loss (the model must predict randomly masked tokens given the rest under a bidirectional mask). The two objectives are summed with per-objective averaging chosen carefully — the report explicitly draws an analogy to GRPO vs DAPO loss aggregation, since the number of masked tokens varies per example and naive averaging skews the gradient. At inference the operator chooses the mode by selecting the attention mask: a strict lower-triangular mask for AR, a bidirectional mask over an active block for diffusion, or a hybrid for self-speculation. In self-speculation, the diffusion side proposes a block of tokens in parallel under bidirectional attention, then those tokens are re-attended causally and verified against the AR distribution; accepted tokens are committed; the shared KV cache means no second model is needed and no extra weights are loaded. An optional LoRA adapter (linear_spec_lora in the HF release) can be attached only during drafting to further raise acceptance length, traded off against a small compute increase. The model family is shipped as 3B / 8B / 14B dense Transformers, each with base, instruct, and vision-language variants, all using the same architecture and the same checkpoint can be served via vLLM, SGLang, or Docker Model Runner.

Results

Headline efficiency on SPEED-Bench (GB200, SGLang, single user): NLD-8B reaches 5.9× tokens per forward (TPF) over Qwen3-8B at matched accuracy, ~4× throughput; native acceptance length 5.46 (LoRA: 6.82) vs Eagle3 2.75 and Qwen3-9B-MTP 4.24; on the coding/math/reasoning/multilingual subset NLD-LoRA averages 8.69 acceptance length vs 2.81 for Eagle3. At 14B with LoRA-tuned linear self-speculation, average accuracy is 66.36% at 5.96× TPF, beating Qwen3-14B-AR (65.17%). NLD-VLM-8B in linear self-speculation reaches 3.63×–7.45× TPF (top end on responses >200 tokens) with 0.1% average accuracy loss vs AR. Real-device wall-clock at concurrency 1: GB200 — 850 tok/s vs 253 tok/s AR (3.3×) and 360 tok/s Eagle3 (2.4×); custom CUDA kernels lift to 1015 tok/s (4×). DGX Spark — 112 tok/s vs 41.8 tok/s AR (2.7×) under w4a16. The speed-of-light analysis (with an oracle sampler under the same architecture) reports a further 76.5% headroom in TPF over the deployed self-speculation sampler, framed as future research.

Why it’s interesting

NLD is the second clean datapoint on the wiki — after WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference — for the hypothesis that the right way to make diffusion LMs actually win in production is to keep them deployable on existing AR-serving infrastructure (vLLM/SGLang/KV cache, FlashAttention-shaped attention) rather than to ship a parallel parallel stack. WeDLM’s contribution was Topological Reordering so a DLM runs under strict causal attention; NLD’s contribution is the orthogonal one — train one model that can choose AR-causal or diffusion-bidirectional or a hybrid at each forward, and use the diffusion mode as a draft inside a self-speculation pipeline that replaces MTP/Eagle3. The 5.46→6.82 acceptance-length jump from a small LoRA on the drafter is also a notable connection to Parameter-Efficient Finetuning: the same trick (LoRA-as-drafter-tuning) that TorchSpec: Speculative Decoding Training at Scale industrialises at K2.5 scale is now used internally on a single-checkpoint self-speculation model. Compared to Speculative Speculative Decoding‘s “parallelize the draft and verify on separate hardware” attack on the same wasted-verification bottleneck, NLD attacks it by making the draft and verify use the same weights, which removes the second-model-fit/distribution-drift question entirely. Open release surface area (3B/8B/14B × base/instruct/VLM, HF weights, multi-backend serving, LoRA adapter) puts it firmly in the Open foundation-model releases pattern alongside DeepSeek-OCR 2 and the Qwen3-VL pipeline pairings.