Skip to content
Bud's Wiki

Introspective Diffusion Language Models

TL;DR

Section titled “TL;DR”

I-DLM converts pretrained AR models (Qwen3-8B, Qwen3-32B) into diffusion language models that retain AR-style introspective consistency — i.e. agreement with their own previously generated tokens — by training under strict causal attention with logit shift and an all-masked objective. The decode-time payoff is Introspective Strided Decoding (ISD), which generates N tokens at MASK positions while verifying the previous N−1 tokens against an anchor distribution in the same forward pass, accepting via min(1, p/q). I-DLM-8B is the first DLM to match same-scale AR quality across 15 benchmarks (matching Qwen3-8B on most, beating LLaDA-2.1-mini 16B by +26 on AIME-24 and +15 on LiveCodeBench-v6) while delivering 2.9–4.1× higher throughput than prior DLMs at high concurrency. A gated-LoRA variant (R-ISD) provides bit-for-bit lossless acceleration over the base AR model. Because attention stays strictly causal, the engine plugs into SGLang with paged KV cache, continuous batching, and CUDA Graphs unchanged.

Key claims

Section titled “Key claims”
  • “Introspective consistency” — whether a model accepts (under top-k argmax or sampling) its own previously generated tokens when re-evaluated — is the structural advantage of AR training that prior DLMs lack: SDAR reports 0.699 vs. I-DLM 0.984 [§Motivation, project page].
  • ISD generates N tokens per forward pass at MASK positions while verifying the prior N−1 committed tokens via the acceptance rule min(1, p(x)/q(x)), where p comes from the current forward (anchor) and q from the previous forward — same guarantee shape as speculative decoding but inside a single model [§Method, project page].
  • I-DLM-8B matches or exceeds Qwen3-8B on 14 of 15 benchmarks: AIME-24 69.6 (Qwen3-8B 73.1, LLaDA-2.1-mini 43.3), LiveCodeBench-v6 45.7 (Qwen3-8B 50.3, LLaDA-2.1-mini 30.4), MATH-500 96.8 (Qwen3-8B 95.8), IFEval 84.7 (tied) [Results table, project page].
  • At stride N=4 with measured acceptance p≈0.9, I-DLM’s tokens-per-forward (TPF) ≈ 2.96 in the memory-bound regime, and its compute efficiency TPF²/queries-per-forward ≈ 1.22 — i.e. each FLOP produces more useful output than AR, so parallel decoding actually saves compute, unlike SDAR which sits at ~0.31 [§Speedup Factor Explorer, project page].
  • At high concurrency (batch 64), I-DLM delivers 2.9–4.1× higher throughput than LLaDA-2.1-mini and SDAR; batching-efficiency slope is 549 for I-DLM vs. 84 for SDAR, meaning I-DLM stays in the memory-bound regime where parallel decoding wins, while SDAR saturates compute early [§Throughput, project page].
  • R-ISD (Residual ISD) adds a rank-128 gated LoRA active only at MASK positions; verify positions use base-only weights, so output is bit-for-bit identical to the base AR model with measured overhead factor α ≈ 1.12 [§Lossless R-ISD, project page].
  • Training recipe is cheap: 4.5B tokens, 8 H100 GPUs, 2 epochs with a stride curriculum (N=2 then N=3); input is the concatenation [x_t | x_0] of a fully-masked sequence with the clean sequence, attended under strict causal masking with auto-balanced CE on both masked and clean positions [§Training, project page].
  • Serving is a strict drop-in for SGLang — no custom kernels — and stacks (i) paged KV cache + continuous batching, (ii) CUDA Graph capture for +42–76%, (iii) stationary-batch decode-loop scheduling for +11–21%, (iv) argmax proposals for +11–15%, (v) paged-only attention kernel for +10–14%, totaling 2.1–2.5× over a naive baseline [§Serving, project page].

Method

Section titled “Method”

I-DLM is a post-training conversion of a pretrained AR model. Training concatenates a fully-masked sequence x_t (the “draft side”) with the clean sequence x_0 (the “verify side”) into one long input [x_t | x_0], applies strict causal attention across the joint sequence so that every MASK position in x_t can see all of x_0’s clean prefix, and trains with an auto-balanced cross-entropy on both the masked positions and the clean positions. A stride curriculum (N=2 then N=3) is used over 4.5B tokens on 8 H100s for 2 epochs. The result is a model that behaves identically to AR at clean positions (logit shift makes position 1 always accepted) and produces high-quality proposals at MASK positions.

At decode time, ISD runs strided blocks of size N. Each forward pass (a) computes anchor distribution p at the previously generated positions (verify-step) and (b) proposes new tokens at the next N MASK positions from distribution q (generate-step). Accepted tokens — those passing u ~ U(0,1), u < p(x)/q(x) — are committed; on first rejection the block restarts from that position. Per-position acceptance compounds geometrically: position k accepts with probability p^(k-1). Memory-bound TPF is (2 + p + ... + p^(N−2)) / (2 − p^(N−1)). R-ISD (lossless) replaces the proposal pathway with a rank-128 gated LoRA active only at MASK positions, leaving verify positions on base weights; the acceptance rule then yields bit-for-bit AR output at overhead α ≈ 1.12.

Results

Section titled “Results”

End-to-end quality, I-DLM-8B vs. Qwen3-8B (same scale AR teacher) vs. LLaDA-2.1-mini (16B, prior DLM SOTA):

  • AIME-24: 69.6 / 73.1 / 43.3 — first DLM within striking distance of AR at this benchmark.
  • AIME-25: 60.8 / 65.4 / 43.3.
  • LiveCodeBench-v6: 45.7 / 50.3 / 30.4.
  • MATH-500: 96.8 / 95.8 / 85.0 (exceeds AR teacher).
  • MMLU-Pro: 73.1 / 75.1 / 64.8.
  • IFEval: 84.7 / 84.7 / 83.2 (tied with AR teacher).
  • I-DLM-32B reaches 83.3 on AIME-24 and 57.1 on LCB-v6, beating Qwen3-32B on multiple math/code benchmarks [Results table, project page].

Throughput at concurrency C=64: 2.9–4.1× over LLaDA-2.1-mini and SDAR. Compute-efficiency analysis (TPF² / queries-per-forward, with AR = 1.0): I-DLM ≈ 1.22 (>1, parallel decoding saves compute), SDAR ≈ 0.31 (decode is wasted compute) — explains why SDAR throughput plateaus and I-DLM scales [§Speedup Factor Explorer, project page].

Why it’s interesting

Section titled “Why it’s interesting”

This is the most direct counterpoint yet to the “diffusion LMs always lag AR” stylized fact, and it shares conceptual machinery with two threads already on the wiki. (a) The introspective-consistency / strided-decode framing is essentially speculative decoding inside a single model: ISD’s min(1, p/q) acceptance is the same rule used in target-model verification in Speculative Decoding: Performance or Illusion? and Speculative Speculative Decoding, but here the “draft” and “target” are the same network on different positions of the same forward. (b) The cache-friendly DLM serving story is the same diagnosis as WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference: keep strict causal attention, and standard SGLang/vLLM stacks (paged KV, CUDA graphs, continuous batching) carry over unchanged — but I-DLM additionally inherits AR-quality at same scale, which WeDLM only matches on average benchmarks (74.72 vs Qwen3-8B-Instruct 72.61). Combined with the gated-LoRA R-ISD variant for bit-for-bit lossless acceleration, the design also fits the Parameter-Efficient Finetuning pattern of LoRA-as-deployment-knob. Open question: how does R-ISD compose with SSD/Saguaro’s draft↔verify parallelization — both target wasted verification compute but from opposite directions of the pipeline.

See also

Section titled “See also”