Speculative Decoding: Performance or Illusion?

TL;DR

A systematic study of speculative decoding (SD) inside a production-grade inference engine (vLLM v0.10.1.1), benchmarking five SD variants — n-gram, EAGLE, EAGLE-3, draft-model, and MTP — across four models (Llama3.1-8B, Llama3-70B, Qwen3-8B, GLM-4.5-Air-106B), six datasets, and batch sizes 1–128. Every variant beats no-SD at small batch, but speedup decays with batch size and the decay is sharper for larger models. Verification cost (running the target model on rejected tokens) dominates the time breakdown — 42–95% across configurations — while drafting is a small fraction. The paper computes a position-aware theoretical upper bound on SD speedup; combining variants adaptively to exploit position-level acceptance complementarity lifts end-to-end speedup to 4.9× over no-SD on Llama-family targets. Outputs are not bitwise identical to no-SD even under greedy decoding, attributed to kernel/floating-point nondeterminism.

Key claims

Prior SD evaluations are batch-size=1 prototypes; this paper is the first systematic SD study inside a production inference engine (vLLM) with CUDA graphs, KV cache management, continuous batching, and chunked prefill enabled [§1, §3.1].
Every SD variant beats the no-SD baseline at small batch, but the relative speedup decays monotonically as batch size grows because the system becomes compute-bound and rejected-token verification time is no longer free [§3.2, Fig. 1].
The speedup decay with batch size is steeper for larger models: EAGLE on ShareGPT loses 4.3% (1.68 → 1.61) going from batch 1 to 32 on Llama3.1-8B but 14.0% (1.96 → 1.72) on Llama3-70B [§3.2].
Draft-model-based SD wins on the 70B target but underperforms EAGLE-3 and even n-gram on the 8B target — because the draft/target single-forward time ratio is ~12.5% at 70B vs ~37.5% at 8B, so proposing eats the budget [§3.2].
n-gram is generally the weakest variant except on code-editing (InstructCoder), where token reuse between prompt and output makes lookup competitive with or better than learned drafters [§3.2, §7].
On reasoning workloads (AIME22–24, GPQA-Main) with long outputs, EAGLE-3 leads (1.64–1.80× on Qwen3-8B-Thinking) and n-gram closes the gap (1.50–1.58×) because long contexts and repetitive symbolic patterns boost reuse; MTP underperforms because the released GLM-4.5-Air weights ship only the first MTP module, which is reused autoregressively and degrades quickly across drafted positions [§3.2].
Verification time dominates execution: 42–95% across SD variants and batch sizes; drafting is <2% for n-gram, 12–20% (batch 1) to 3–7% (batch 512) for EAGLE/EAGLE-3, and 21–47% (batch 1) for draft-model SD; sampling is <1.7% throughout [§5.1, Fig. 3].
vLLM scheduling/system overhead is 3–12% of total execution time and shrinks with model and batch size as fixed costs amortize [§5.1].
Outputs of SD vs no-SD are not bitwise identical even under greedy decoding (temperature=0, top_p=1, top_k=-1); attributed to GPU kernel nondeterminism and floating-point variation per He (2025), so the paper reports token throughput rather than per-request latency [§3.1, Tab. 1].
A simulator grounded in measured per-position acceptance statistics computes the theoretical upper bound on SD speedup; the gap between measured and bound highlights wasted verification on rejected tokens as the primary remaining opportunity [§1, §8].
Different SD variants achieve high acceptance at different token positions; adaptively combining them per-position lifts end-to-end speedup to 4.9× over no-SD, beyond any single variant in isolation [§1].
Acceptance behavior is variable at three levels: within a request, across requests, and across datasets — so dataset-averaged acceptance numbers obscure the actual deployment behavior [§4].

Method

The benchmark fixes vLLM v0.10.1.1 with all production optimizations enabled (CUDA Graphs, KV-cache mgmt, continuous batching, chunked prefill) on H100s — 1×H100 for 8B models, 4×H100 (TP=4) for 70B/106B. Five SD variants are evaluated: draft-model (Llama3.2-1B for Llama3-70B, Qwen3-0.6B for Qwen3-8B, distilled), EAGLE / EAGLE-3 (draft-model-free, auxiliary heads predicting 3 tokens/step), MTP (co-trained heads on GLM-4.5-Air), and n-gram (prompt-lookup-decoding with proposal lengths 3 and 5). Six datasets span summarization (CNN/DailyMail), chat (ShareGPT), code editing (InstructCoder), grade-school math (GSM8K), and reasoning (AIME22–24, GPQA-Main). Decoding is greedy (T=0). For each configuration the authors record (i) end-to-end token throughput, (ii) a per-stage time breakdown (drafting / verification / rejection sampling / vLLM overhead), (iii) per-position acceptance rates across requests. The reported simulator replays measured per-position acceptance into a closed-form speedup model derived from Leviathan et al. (Eq. 1), allowing them to compute what speedup would be achievable if verification only operated on tokens that would ultimately be accepted.

Results

End-to-end (batch 1, GSM8K, Llama3.1-8B): EAGLE 1.73× over no-SD; collapses to 1.21× at batch 128 [§3.2].
Reasoning (Qwen3-8B-Thinking): EAGLE-3 1.64–1.80× on GPQA-Main / AIME22–24; n-gram 1.50–1.58× [§3.2].
MTP on GLM-4.5-Air-106B: 1.3–1.8× on GPQA-Main, below the upper bound because only the first MTP module is released open-source and is reused autoregressively [§3.2].
n-gram on InstructCoder: outperforms EAGLE and EAGLE-3 on Llama3.1-8B and Qwen3-8B — code reuse dominates [§3.2].
Time breakdown (CNN/DailyMail, 500 requests): verification 42–95%; n-gram drafting <2%; EAGLE/EAGLE-3 drafting 3–20%; draft-model drafting 16–47% on 8B and 3–21% on 70B; sampling <1.7%; vLLM overhead 3–12% [§5.1].
Memory (FP16): n-gram adds zero static memory; EAGLE +0.47 GiB (8B), EAGLE-3 +0.79 GiB (8B); 1B draft model adds the model itself (~2 GiB) on top of the 70B target [Tab. 2].
Adaptive combination: per-position selection across SD variants reaches 4.9× over no-SD on Llama-family, vs. <2× for the best single variant in many settings [§1, §8].
Output nondeterminism (Tab. 1): generation length on the same prompt set varies across batch sizes and SD variants even at greedy; not consistently longer or shorter than baseline — so per-token-throughput is the right metric, not per-request latency.

Why it’s interesting

A clean evidence-based reset on the “SD speedup” narrative. Prior numbers in the literature (2–3× routinely; 5–6× claimed for tree-based drafters) collapse to ~1.2–1.8× once you put the method into vLLM and turn batching on — which is exactly the deployment regime any serious inference service runs. The 42–95% verification-time fraction is the headline diagnostic: SD is now bottlenecked not by drafting but by running the target model on tokens that will be thrown away, which is the opening that Speculative Speculative Decoding (Saguaro/SSD) attacks by parallelizing draft and verify on distinct hardware, and that TorchSpec: Speculative Decoding Training at Scale attacks upstream by training a higher-acceptance draft model in the first place. The dataset-dependence finding (n-gram wins on InstructCoder; draft-model wins on 70B; EAGLE-3 wins on reasoning; MTP wins on GLM-4.5-Air when full MTP heads ship) means choosing an SD variant in production is a workload-routing problem, not a one-size answer. The output-nondeterminism observation under greedy is also worth flagging for anyone designing eval harnesses: SD vs no-SD outputs are not reproducible even at T=0, so SD-introduced quality regressions can be very hard to attribute. The 4.9× adaptive-combination headline is the most actionable forward direction: a position-level router across SD variants is on the table for any inference team willing to instrument acceptance per position.