Speculative Speculative Decoding

TL;DR

Standard speculative decoding (SD) accelerates LLM inference by overlapping a small draft model with a parallel verification pass of the target, but draft and verify remain sequentially dependent: the next speculation can’t start until the current verification finishes. Speculative Speculative Decoding (SSD) parallelizes them by having the draft model pre-speculate, in parallel with ongoing verification, against the most likely verification outcomes; if the realized outcome was pre-speculated, draft latency is hidden entirely. The optimized instantiation, Saguaro, builds the speculation cache with a geometric fan-out, biases the draft sampling distribution to make the residual (bonus-token) distribution easier to predict, and switches its cache-miss fallback strategy as a function of batch size. Reported: average 30% faster than the strongest SD baselines (incl. SGLang/vLLM, EAGLE-3) and up to 5× faster than autoregressive decoding; SSD shifts the latency–throughput Pareto frontier, not just latency.

Key claims

SD is rate-limited by sequential drafting–verification: drafting must wait for the verifier to return before producing the next speculation; SSD overlaps the two on distinct hardware [§1, §3.1].
The space of possible verification outcomes is ~ $V^{\gamma}$ where $V$ is vocab size and $\gamma$ the lookahead; SSD reformulates outcome prediction as constrained optimization — pick a budgeted subset of likely outcomes and pre-speculate against each [§4.1].
The optimal Saguaro cache shape under a power-law cache-hit-rate assumption is a capped geometric fan-out across speculation positions: spend more outcome guesses at earlier positions where acceptance is likely, less where it isn’t [Theorem 12].
The bonus token at each round is drawn from the residual distribution $\propto \max(p_t - p_d, 0)$ , which is hard to predict at higher temperatures; Saguaro sampling biases the draft distribution to concentrate the residual on the most likely draft tokens, increasing cache hit rate at the cost of acceptance rate [§4.2, Theorem 15].
The acceptance-rate vs. cache-hit-rate tradeoff is navigable by a single down-weighting hyperparameter $\beta$ ; lower $\beta$ gives higher cache-hit, lower acceptance, and the end-to-end optimum is empirical [Definition 14, Fig. 5].
Cache-miss fallback strategy must change with batch size: at small batch, reuse the (slow, accurate) primary speculator just-in-time; at batch ≥ $B^*$ , switch to a low-latency fallback (e.g. random tokens or n-gram speculator), because the whole batch stalls on the slowest miss [Theorem 17].
SSD is lossless and strictly dominates SD at parity: with primary = backup = the SD draft model, SSD ≥ SD; with primary slower-but-higher-acceptance and backup faster, the SSD speedup over SD is sandwich-bounded by latency hidden and acceptance gain [Corollaries 8, 9].
End-to-end on Llama-3, Saguaro is on average 30% faster than the strongest open SD baselines (SGLang/vLLM, EAGLE-3) and up to 5× vs. autoregressive [Fig. 7, §5].
SSD shifts the latency–throughput Pareto frontier — biggest gains at low batch, but improvements hold even at larger batch sizes despite SSD doing more compute per batch element [§5, Fig. 7 right].
SSD composes with tree-based speculative decoding (Medusa-style) and improved draft architectures (EAGLE, GliDe, LongSpec) — those add acceptance/draft-speed gains without changing the SSD overlap structure [§2.2, Appendix E].

Method

SSD runs the speculator and verifier on distinct hardware. While the verifier processes round- $i$ tokens, the speculator builds a speculation cache $C_{i+1}$ : a dictionary from candidate verification outcomes $\tau$ (= accepted-tokens prefix length + bonus-token identity) to fully-prepared next-round speculations. When the verifier returns the realized outcome $\tau^*$ , a cache hit lets the speculator immediately send the corresponding pre-speculated tokens to the verifier — drafting latency hidden. A miss falls back to a backup speculator. Because the residual distribution depends on the draft distribution, Saguaro deliberately down-weights the most likely draft logits (parameter $\beta$ ) when sampling the speculation; this concentrates the residual on those same tokens, raising the chance that the bonus token sampled by the verifier is one Saguaro pre-cached. The fan-out budget across the $\gamma$ lookahead positions is allocated as a capped geometric series (Theorem 12), reflecting the geometric distribution of accepted-string lengths. Fallback strategy is batch-size-aware: just-in-time slow speculator at low batch, fast/low-latency speculator (n-gram or random) at large batch where cache misses become near-certain.

Results

Headline. Average ~30% wall-clock speedup over SD baselines (including SGLang/vLLM with EAGLE-3) and up to 5× vs. autoregressive on Llama-3 across Alpaca / GSM8k / UltraFeedback / HumanEval; matched on Qwen-3 (Appendix F) [§5, Fig. 7].
Cache-hit prediction. The Saguaro residual-prediction scheme reaches up to ~90% accuracy at predicting the bonus token from draft logits [§4.1.1].
Geometric fan-out. Beats uniform fan-out on both cache-hit rate and end-to-end decode speed, with the gap widening at higher temperatures [Fig. 4].
Sampling trade-off. Lower $\beta$ drives higher cache-hit at lower acceptance; the end-to-end optimum sits in the interior of $[0,1]$ [Fig. 5].
Fallback at scale. At larger batch sizes a fast (e.g. random-token) backup beats a slow neural one used just-in-time; this is why Saguaro retains ~20% speedup over SD even at larger batches despite doing more draft compute per element [§4.3.3, Fig. 6].
Pareto. SSD is not just lower latency but also higher throughput per device than SD at low/medium batch [Fig. 7 right].

Why it’s interesting

A clean structural inefficiency in speculative decoding — that drafting and verification, themselves designed to overlap with the target model, still serialize against each other — and a concrete framework + theory + system to close it. The “speculate about your own speculative-decoder’s verification outcome” framing is also worth absorbing as a general technique: any pipeline with branched outcomes and idle drafting compute admits the same construction. Sits alongside Inference-Time Hyper-Scaling with KV Cache Compression as a lossless inference-acceleration result that pushes the Pareto frontier rather than trading quality for speed — DMS is a parameter-light retrofit of the KV cache, SSD is hardware-level overlap of draft and verify. Complements WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference‘s argument that prefix-cacheability dominates token-throughput in serving — both papers are reframing where the real inference bottleneck lives. SSD’s “primary slow + accurate, backup fast + cheap at large batch” structure recapitulates the compute-matching intuition of Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling but applied to inference rather than synthetic-data generation.