TiDAR: Think in Diffusion, Talk in Autoregression

TL;DR

TiDAR is an NVIDIA hybrid LLM architecture that drafts tokens in parallel via diffusion (“Thinking”) and verifies them autoregressively via rejection sampling (“Talking”) — all in a single forward pass under a structured causal+bidirectional attention mask. By exploiting “free token slots” in the memory-bound decode regime, it gets diffusion’s parallelism without diffusion’s quality loss, since the AR pass guarantees chain-factorized-joint quality. Continually pretrained from Qwen2.5-1.5B (50B tokens) and Qwen3-8B (150B tokens), TiDAR matches AR quality on coding/math/reasoning while reporting 4.71× (1.5B) and 5.91× (8B) higher tokens/second on a single H100 — and unlike prior dLLMs supports exact KV caching.

Key claims

Sequence is partitioned into three sections — prefix (causal, KV-cached), last-step proposals (causal, AR-verified via rejection sampling), and next-step pre-drafts (bidirectional, diffusion drafted under fully-masked inputs) — and all three are handled in one forward with a hybrid attention mask [§3.1, Fig. 2].
Training uses a fully-masked diffusion section (every token in the diffusion block is <mask>), giving dense diffusion loss equal in count to the NTP loss and enabling straightforward loss balancing with λ=1 [§3.1].
Both AR-NTP loss on the causal prefix and diffusion loss on the masked block are computed on the same sample — Block Diffusion cannot do this due to label leakage from intra-block bidirectional attention, so the joint-loss training is a TiDAR-specific affordance [§3.1].
Drafting is one-step diffusion (not multi-step denoising); one step is reported sufficient because the rejection-sampling stage will fix any low-quality draft anyway [§3.2].
Pre-drafting conditions on all possible outcomes of the current rejection-sampling step (covering each accept-count branch), so the next-step draft is ready no matter how many tokens get accepted — eliminates the draft↔verify serialization that limits classical SD [§3.2].
TiDAR supports exact KV caching (unlike LLaDA/Dream/Fast-dLLM) without recomputing any token’s KV — KV from causal-attention positions is reused across steps, and only evicted if the sampled prefix is shorter than the drafted length [§3.3].
TiDAR-1.5B (continued-pretrained from Qwen2.5-1.5B on 50B tokens, block sizes 4/8/16): 44.03% average on HumanEval/HumanEval+/MBPP/MBPP+/GSM8K/Minerva-Math vs Qwen2.5-1.5B 41.64% and Block-Diffusion-1.5B 38.41%, at an average 7.45 tokens per NFE [Table 2].
TiDAR-8B (continued-pretrained from Qwen3-8B on 150B tokens, block size 16) trusting diffusion: 65.31% coding/math average vs Qwen3-8B 68.09%, Dream-7B 58.74%, LLaDA-8B 41.78% — at 8.25 tokens per NFE [Table 2].
On likelihood tasks (MMLU/ARC/Hellaswag/PIQA/Winogrande), TiDAR-8B averages 75.40% beating Qwen3-8B at 74.25%, Dream-7B at 71.86%, and LLaDA-8B at 68.06% — and likelihood is computed via the AR mode in a single NFE, sidestepping the Monte-Carlo masking estimator that prior dLLMs need [§4.2.2, Table 3].
Single-H100 wall-clock at batch size 1: TiDAR-1.5B is 4.71× tokens/sec vs Qwen2.5-1.5B AR, and TiDAR-8B is 5.91× vs Qwen3-8B AR, beating EAGLE-3 speculative decoding and Block-Diffusion on the same benchmarks [§4.3, Fig. 4].
The “free token slots” effect is empirically validated: extending the forward by ~16 mask tokens at small batch keeps latency nearly flat because decode is memory-bound, but the extra slots produce useful drafted tokens at no extra wall-clock cost [§1, Fig. 1].

Method

The architecture is a standard Transformer with a custom attention mask: prefix tokens see causal attention (AR-NTP path), and a trailing block of fully-masked tokens sees bidirectional attention to the prefix + intra-block (diffusion path). During training, every sample contributes both an NTP loss on the prefix and a per-position diffusion loss on the masked block; the input is a doubled sequence (original + fully-masked corruption appended). The loss is λ·L_AR + L_Diff with λ=1. Continual pretraining starts from a frozen-architecture Qwen2.5/Qwen3 checkpoint, with block sizes 4/8/16, cosine schedule, bf16, 50B/150B token budgets, on H100s with Megatron-LM + Torchtitan.

Inference proceeds in steps. Each step appends two blocks to the prefix: (a) the previous step’s drafted tokens, attended causally so AR rejection sampling can evaluate p_AR at every position; and (b) a new block of <mask> tokens, attended bidirectionally, that draws the next draft from p_Diff conditioned on every possible accept-count outcome of (a) — leveraging the Apple-MTP “branch-on-all-outcomes” trick so no rollback is needed. After the forward, accepted tokens are committed (KV retained), the matching pre-draft branch is selected, and the loop continues. A Flex-Attention block-mask of size (q_len × q_len + max_seq) is precomputed once and sliced per step.

Results

Quality: TiDAR-1.5B reaches the average quality of its Qwen2.5-1.5B teacher on coding/math (44.03 vs 41.64) and beats Block Diffusion-1.5B (38.41) — i.e. the parallel-decoding tax disappears. TiDAR-8B is within 2.8 points of Qwen3-8B on coding/math (65.31 vs 68.09) and beats Dream-7B (58.74) and LLaDA-8B (41.78) by large margins [Table 2]. On likelihood (MMLU/ARC/Hellaswag/PIQA/Winogrande) TiDAR-8B actually exceeds Qwen3-8B (75.40 vs 74.25) [Table 3].

Efficiency: at H100 batch 1, TiDAR-1.5B averages 4.71× and TiDAR-8B averages 5.91× tokens/sec over the AR teacher [§4.3]. EAGLE-3 SD on Qwen3-8B-Instruct is the strongest published baseline and TiDAR-8B still beats it on wall-clock. Tokens-per-NFE: 7.45 for 1.5B, 8.25 for 8B.

Why it’s interesting

TiDAR is the second NVIDIA tri-mode-style paper on the wiki and lands directly on top of Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding — both train a single checkpoint that can serve in AR, diffusion, and self-speculative modes; both target the “5–6× tokens/s over the AR teacher at matched quality” headline. The key methodological difference is that TiDAR pre-drafts during verification by branching on all outcomes of the current rejection sample, whereas NLD’s self-speculation is a more conventional accept-then-redraft loop — making TiDAR more aggressive at hiding draft latency, closer in spirit to Speculative Speculative Decoding‘s pre-speculation cache.

This also reframes how the dLLM line on Diffusion Language Models should be read: TiDAR side-steps the parallelizability/quality tradeoff that DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation and Introspective Diffusion Language Models tried to fix within the diffusion paradigm by simply outsourcing quality to AR rejection sampling. It complements WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference‘s “diffusion under causal attention is cache-friendly” thesis by adding “diffusion + AR in one model is cache-friendly and lossless” — fitting the prefix-cacheability framing of LLM Inference Efficiency directly.