Skip to content
Bud's Wiki

LLM Inference Efficiency

TL;DR

Section titled “TL;DR”

Lossless or near-lossless speed-ups for serving LLMs — methods that change how the target model is invoked at decode time without changing what it outputs in distribution. Three sub-threads are converging on the same diagnosis: under production serving (vLLM/SGLang with continuous batching, CUDA Graphs, KV-cache management), the dominant cost is target-model verification compute and KV-cache memory traffic, not drafting or arithmetic. The filed papers split into (a) speculative decoding — drafting + verification, with empirical evidence that verification dominates 42–95% of execution time and that drafting/verification overlap is the next opportunity, (b) cache-prefix-friendly non-AR decoders — diffusion LMs that keep a standard causal mask so they remain compatible with vLLM/FlashAttention/PagedAttention, and (c) upstream training infrastructure for the draft side of SD. Distinct from inference-time-scaling, which trades more compute for better outputs; this concept is about producing the same output faster.

Key claims

Section titled “Key claims”
  • In production serving (vLLM v0.10.1.1, all optimizations enabled), every speculative decoding variant beats no-SD at small batch but the speedup decays monotonically with batch size and the decay is steeper for larger models — at batch 128 on Llama3.1-8B/GSM8K, EAGLE drops from 1.73× to 1.21× (Speculative Decoding: Performance or Illusion? §3.2, Fig. 1).
  • Verification — running the target model on proposed tokens — dominates the SD time breakdown at 42–95% across configurations; drafting is <2% for n-gram, 12–20% for EAGLE/EAGLE-3 at batch 1 (collapsing to 3–7% at batch 512), and 21–47% for draft-model SD at batch 1; rejection sampling is <1.7% (Speculative Decoding: Performance or Illusion? §5.1, Fig. 3).
  • SD variant selection is workload-dependent, not universal: n-gram wins on code-editing (InstructCoder, by token reuse), draft-model wins on 70B targets, EAGLE-3 wins on reasoning workloads with long outputs, and MTP wins on GLM-4.5-Air when full multi-head MTP weights are released (Speculative Decoding: Performance or Illusion? §3.2).
  • Adaptive per-position combination of SD variants lifts end-to-end speedup to 4.9× over no-SD on Llama-family targets, far beyond any single variant in isolation — the per-position acceptance complementarity across variants is the largest remaining opportunity (Speculative Decoding: Performance or Illusion? §1, §8).
  • Speculative decoding is rate-limited by sequential drafting↔verification: the next speculation cannot start until the current verification finishes. Speculative Speculative Decoding (SSD/Saguaro) parallelizes them by pre-speculating against the most likely verification outcomes on distinct hardware, hiding draft latency on cache hits (Speculative Speculative Decoding §1, §3.1).
  • SSD/Saguaro achieves average ~30% wall-clock speedup over the strongest open SD baselines (SGLang/vLLM, EAGLE-3) and up to 5× vs. autoregressive decoding on Llama-3; effects hold at low/medium batch and shift the latency–throughput Pareto frontier (Speculative Speculative Decoding §5, Fig. 7).
  • Optimal Saguaro cache shape under a power-law cache-hit-rate assumption is a capped geometric fan-out across speculation positions: spend more outcome guesses early where acceptance is likely (Speculative Speculative Decoding Theorem 12).
  • Producing a high-quality EAGLE-3 draft model for trillion-parameter targets is itself bottlenecked by storage (~700 TB to precompute hidden states for 100k samples on Kimi K2.5) or co-located memory (target weights eat 575 GB on 8×H100, capping context to ~4096 tokens). TorchSpec disaggregates inference and training across clusters with Mooncake RDMA streaming hidden states, training a Kimi K2.5 EAGLE-3 draft in 1500 H200-hours on 600k samples and unlocking 44k-token context on a single H100 / 200k on a single B200 (TorchSpec: Speculative Decoding Training at Scale §Background, §Long Context Support).
  • The TorchSpec-trained K2.5 EAGLE-3 draft delivers +60% throughput at batch=1, +30% at batch=8, +26% at batch=16 with lookahead=3 — i.e. the draft remains good enough to justify SD overhead at multi-tenant batch sizes, not just at batch=1 (TorchSpec: Speculative Decoding Training at Scale §Introduction).
  • In KV-cached serving, the right efficiency metric is prefix cacheability P_c = N_new / N_total — the fraction of processed token instances that end up as cache-reusable committed prefix — not tokens-per-forward. This reframes the diffusion-LM vs. AR debate and explains why many parallel-decoding papers don’t translate per-step token counts into wall-clock wins (WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference §3.1, Eq. 4).
  • WeDLM achieves prefix cacheability by topological reordering: observed tokens are physically moved to the prefix while RoPE position ids preserve logical positions, so masked tokens attend to all observed context under an unmodified causal mask — directly compatible with FlashAttention/PagedAttention/CUDA Graphs without kernel changes (WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference §4.1, §5.2).
  • WeDLM-8B (continued-pretrained from Qwen3-8B on 100B tokens) served via vLLM reaches ~3× end-to-end speedup on complex reasoning and >10× on low-entropy generation versus the same vLLM-served Qwen3-8B baseline, while matching or exceeding base on quality (WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference §1, §6.4, Table 1).
  • Output nondeterminism is real even under greedy decoding (T=0, top_p=1, top_k=-1): SD vs no-SD outputs are not bitwise identical, with generation length varying across batch sizes and SD variants. Attributed to GPU kernel parallelism and floating-point variation; implies token-throughput, not per-request latency, is the correct metric (Speculative Decoding: Performance or Illusion? §3.1, Tab. 1).
  • Inference-efficiency comparisons are extremely sensitive to baseline implementation parity: under a reproducible symmetric setup, TurboQuant’s published 6×/8× speed-and-memory advantages over RaBitQ for KV-cache quantization do not hold — RaBitQ outperforms TurboQuant in most inner-product, ANN, and KV-cache settings, the TurboQuant baseline for RaBitQ ran single-threaded on CPU while TurboQuant ran on A100 GPU (undisclosed), and several of TurboQuant’s own reported runtimes cannot be reproduced from the released artifact (Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments §1, §4.1, §4.2).
  • A variance bound on a quantization estimator is not the same caliber of guarantee as a sub-Gaussian tail bound: TurboQuant’s Theorem 2 (variance) converts via Chebyshev to bit-width scaling 1/ε² · 1/δ², whereas RaBitQ’s sub-Gaussian tail gives 1/ε² · log(1/δ) and matches the Alon–Klartag (FOCS 2017) optimal lower bound (Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments §3, Lemma 3.1).
  • Aggressive weight-level quantization can be pushed to 1-bit end-to-end (embeddings, attention projections, MLPs, LM head — no FP16 escape hatches) and still claim benchmark parity with FP16 8B baselines at ~1/14 the size: PrismML’s 1-bit Bonsai 8B reports 1.15 GB footprint, 6.2× faster on RTX 4090, 8.4× faster on M4 Pro, 44 tok/s on iPhone 17 Pro Max, 4–5× lower energy per token, and a 70.5 average across 6 benchmark categories — all vendor-reported, pre-whitepaper, and dependent on custom Q1_0 / MLX dequantization kernels in forked runtimes (1-bit Bonsai 8B: End-to-End 1-bit Language Models from PrismML).
  • Real-time streaming inference (200ms chunks of audio/video/text) introduces a regime that existing LLM inference libraries are not optimized for: frequent small prefills + decodes with per-turn metadata overhead. Thinking Machines’ streaming-session primitive treats each chunk as an independent request that the server appends to a persistent in-GPU sequence, avoiding repeated memory reallocation and metadata-computation overhead. A version of this has been upstreamed to SGLang as PR #19171 (Interaction Models: A Scalable Approach to Human-AI Collaboration §Inference optimization).
  • A gather + gemv MoE kernel beats grouped-GEMM in the streaming small-batch regime, where there isn’t enough shared work per expert to amortize the gather-scatter-combine bookkeeping; the same diagnosis appears in PyTorch’s “supporting Mixtral in gpt-fast” and in Better MoE model inference with warp decode‘s warp-decode kernel (Interaction Models: A Scalable Approach to Human-AI Collaboration §Inference optimization).

Recent contributions

Section titled “Recent contributions”
  • [2026-08-31] Sliding-window beats linear attention: Jolicoeur-Martineau et al. argue Sliding Window Attention + attention sinks (StreamingLLM recipe) is a strictly better inference-memory-reduction lever than post-training a pretrained LLM into a linear-attention model — same or better on general tasks, 2–10× stronger on long-context reasoning (NIAH, BABILong), with zero post-training cost. Adds a “don’t retrofit, mask instead” position to the page’s inference-memory-reduction thread; leaves from-scratch hybrid-linear designs (Kimi Linear, Qwen3-Next) explicitly out of scope.
  • [2026-08-27] Redwood: A Frontier AI Accelerator Designed, Verified, and Deployed from Scratch in 2 Weeks by AI: Redwood (Architect Labs) is an AI-designed tile-based near-memory-dataflow accelerator projected at 3.4× tokens/s/W vs a measured Jetson Orin Nano baseline on Qwen3-0.6B at Samsung 8nm-class (49 vs 28 tokens/s at 1.335 W vs 2.59 W, ~2.88 mm²). Adds a co-designed accelerator lever to this page’s existing threads (speculative decoding, KV/weight quantization, streaming sessions) — the perf/watt comes from tile-local scratchpads, DMA-orchestrated data movement, FlashAttention-4’s emulated-softmax reused on shared SIMD lanes, and CTM-to-CTM message-based scheduling that removes centralized arbitration. Measured FPGA baseline is 12.1 tokens/s on VPK180 against a 21.73 tokens/s roofline; decode is confirmed strongly memory-bound (44.65 ms DRAM vs 12.29 ms arithmetic per token in the roofline).
  • [2026-08-26] Qwen3.8-Flash-Next — Hybrid Attention with QSA, Gated Residual, N-gram Embedding: Qwen3.8-Flash-Next’s Qwen Sparse Attention (QSA) selects at the micro-block level with a 512-block / 2048-token budget via an MQA-style indexer, cutting long-context softmax cost where per-token attention becomes the bottleneck; native 262K context, extensible to 1M via YaRN factor=4.0.
  • [2026-08-18] DFlash 2: Keep Drafting Parallel: Inco AI’s DFlash 2 keeps DFlash’s one-forward-pass parallel drafter but adds a 2M-param pairwise path selector over top-16 candidates and a 16.5M-param dynamic depthwise conv (two-tap, Canon-Layers-style, reaching one predecessor) around attention/FFN sublayers — the diagnosis being that DFlash’s top-16 already contains the right token 99.5% of the time at position 0, and suffix decay is a local problem attention shouldn’t have to spend capacity on. Together +1.3% cycle latency for +21% acceptance-length over DFlash and +9% over DSpark; hits 2.7–3.4× (Qwen3.8-27B) and 3.1–4.6× (Muse Glimmer) throughput over autoregressive at batch 1 on SGLang. Community DFlash 2 drafter for Muse Glimmer beats the official DFlash 1 drafter Meta shipped with the model (5.70 vs 4.44 avg tokens/pass), sharpening the drafter-in-the-box packaging pattern already seen in Introducing Muse Glimmer: An Open Agentic Model That Runs on Your Device and 1-bit Bonsai 27B — GGUF: 27B-class Reasoning at 1.125 Bits Per Weight.
  • [2026-08-10] Introducing Muse Glimmer: An Open Agentic Model That Runs on Your Device: Muse Glimmer ships ~4-bit weight quantization + a small DFlash-based speculative-decoding drafter as a co-released artifact in a single open release — the same “drafter-in-the-box” packaging seen in 1-bit Bonsai 27B (DSpark drafter). Full-precision 30B (>55 GB) compresses to <20 GB LM weights, leaving room for KV cache, perception encoder, and drafter within a 24/32 GB consumer-GPU envelope; validated as “minimal to no degradation on agentic tasks”. Adds DFlash (arXiv 2602.06036) to this cluster’s watch list — not yet filed.
  • [2026-08-08] πR²: Reactive Real-time Flow Policies: πR² (Park & Tulsiani, CMU) is the robotics-side twin of the streaming/small-prefill regime this page tracks: for large flow-matching VLAs (GR00T-N1.7), naive per-tick replanning is blocked by the VLM backbone dominating per-call latency (image + VLM ≈ 100 ms vs ~5 ms for one denoising step). πR² gets 25 Hz replanning (~4× faster than baseline, 40 ms tick on RTX A5000) without compressing the backbone — the lever is structural decoupling (async fast/slow channel: proprioception refreshed every tick, VLM features cached in a background thread with a learned integer-delay embedding) plus a latency-adaptive staircase noise schedule that lets the action head run 1 NFE per call regardless of K, parameterized by measured delay τ so one trained model handles a range of latencies. Sibling to Sunday Robotics’ finding (Sunday Robotics finds on-device VLA inference beats cloud during ACT-2 eval (Cheng Chi)) that home-VLA cloud inference is tail-latency-limited by network rather than per-step compute — same async-inference regime attacked from the model side rather than the deployment side. Complements the async-inference training-recipe answer of Real-Time Robot Execution with Masked Action Chunking (REMAC’s LoRA + prefix-mask training) and the output-space answer of B-spline Policy: Accelerating Manipulation Policies via B-spline Action Representations (B-spline Policy).
  • [2026-08-07] Real-Time Robot Execution with Masked Action Chunking: REMAC (Wang et al., Cisco Research) is the robotics-side counterpart to the LLM-streaming problem this page tracks — how to hide inference latency behind async prediction without accuracy loss. On flow-matching VLAs, naive async chunking creates two distinct failure modes (inter-chunk discontinuity + intra-chunk inconsistency where the executed prefix was conditioned on a stale observation); REMAC fixes both at training time via a LoRA correction (≤1.5% params) under a prefix-mask + self-conditioning curriculum + residual-alignment loss, plus a prefix-preserved inference sampler. Adds zero inference latency by merging LoRA into the backbone. Under the paper’s example numbers, a π₀-class model already needs 76 ms per action on an RTX 4090 vs a 20 ms control period (50 Hz), so sync inference is unattainable and async is the only viable regime — this makes the training-time robustness recipe the load-bearing lever rather than model compression.
  • [2026-07-31] Interlatent to open-source full-stack robotics deployment infrastructure — cloud inference pipelines and robot-driving stack: Interlatent pre-announces an open-source cloud inference pipeline for robotics + robot-driving plane, staking the framing that “no public examples” of efficient cloud inference for robotics exist. Directionally aligned with the Embodied.cpp diagnosis (Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots) that VLA deployment is a structurally different serving contract from throughput-first LLM inference, and complementary to Sunday Robotics’ filed field-evidence (Sunday Robotics finds on-device VLA inference beats cloud during ACT-2 eval (Cheng Chi)) that residential WiFi/ISP tail latency, not per-step compute, is the binding constraint for single-robot cloud VLA serving. Announcement only — no code, no benchmarks, target repo not publicly readable at filing.
  • [2026-07-31] Speeding up end-to-end inference with self-improving agents: Asari AI Labs reports agent-driven full-stack optimization of vLLM v0.23 serving DeepSeek v4 Pro and GLM 5.2 on 8×B200, with up to 16% throughput/interactivity gains across concurrency levels — most produced changes have no upstream equivalent in vLLM v0.26, and the gain surface is specifically M=1 skinny GEMM kernels, glue-code paths, and launch overhead (i.e., the seams between components that manual optimization has left underserved). Adds two threads to this page: (a) an agent-produced-optimizations lever distinct from speculative decoding, cache compression, and weight quantization, targeting parts of the stack that Amdahl’s law says still matter in production; (b) a stringent distribution-matching correctness gate on token-level probability distributions over an AllenAI C4 prompt set — the strictest shippability contract yet filed on this page, and a possible resolution to the open question about how reproducibility should be defined given SD’s non-bitwise-identical greedy outputs.
  • [2026-07-30] How enabling two settings tripled our scores on the ARC-AGI-3 benchmark: OpenAI reports a 6× output-token reduction on the ARC-AGI-3 public set from turning on two Responses API defaults — retained reasoning across turns and canonical compaction instead of rolling-window truncation — while task score simultaneously moves from 13.3% → 38.3% RHAE. Concrete deployment instance of the O(L²) → O(L) validated-compaction argument from Agentic Context Management: Solving Agent Memory and Cost by Treating Them as Lifecycle and Architecture Problems: harness-level context-management is a cost lever that increases quality rather than trading it, distinct from the decode-time speedups (SD, KV compression, weight quantization) tracked elsewhere on this page.
  • [2026-07-30] TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAM: TurboVLA reports the sharpest datapoint yet for “remove the LLM from the perception-to-action path” as an efficiency lever on VLAs: replacing the standard V → L → A LLM-centric pathway with a direct V + L → A mapping (independent V and L encoders + a lightweight bidirectional interaction module + compact decoder) collapses a LIBERO-competitive VLA to 0.2B params, 0.9 GB inference VRAM, and 31.2 ms policy latency on a consumer RTX 4090 — sub-billion-parameter, sub-GB, 32 Hz on commodity hardware without the LLM decode loop.
  • [2026-07-27] Kimi K3 Technical Report — Open Frontier Intelligence: Kimi K3 tech report — KDA systems co-design contributes fused kernels, KDA Context Parallelism, and state-aware prefix caching (landed in vLLM); Block Attention Residuals reduces inference state from O(Ld) to O(Nd) with online-softmax merging of parallel inter-block and sequential intra-block partial sums.
  • [2026-07-25] Agentic Context Management: Solving Agent Memory and Cost by Treating Them as Lifecycle and Architecture Problems: Adds a conversation-length cost lever distinct from the page’s per-decode threads (SD, KV-cache compression, weight quantization, streaming sessions): argues naive context accumulation costs O(L²) tokens in conversation length while validated compaction gets O(L) with preserved fidelity. Frames memory-lifecycle compaction as a first-class inference-efficiency primitive at the agentic level, and calls out three missing evaluation axes — latency, token efficiency, context-rot resistance — that overlap with this concept’s throughput/latency focus but haven’t been wired into agent memory benchmarks like LongMemEval / LoCoMo.
  • [2026-07-23] GLM-5.2-Vision-NVFP4 — bolting MoonViT onto frozen GLM-5.2 with a 49.5M projector: Blackwell-only serving recipe for a community 744B-MoE VLM: NVFP4 text weights via nvidia/GLM-5.2-NVFP4, DSA attention backend, FP8 KV cache, full 1M-token context on 8×B200 (256k on 4×B200) with an out-of-tree SGLang plugin for the new Glm5vForConditionalGeneration architecture.
  • [2026-07-17] Sunday Robotics finds on-device VLA inference beats cloud during ACT-2 eval (Cheng Chi): Sunday Robotics deployment observation: at single-robot residential-network scale, cloud inference for a home VLA is tail-latency-limited by the WiFi/ISP hop, not by the model’s per-step compute — Cheng Chi reports moving ACT-2 to on-device inference two days into their eval after concluding the trained model was more reliable than the average home network. First wiki datapoint on the latency-tail-first, single-request regime that sits orthogonal to the throughput-first speculative-decoding / KV-cache / streaming-session threads on this page, and field-evidence support for Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots‘s claim that VLA deployment is a structurally different serving contract from LLM inference.
  • [2026-07-16] Kimi K3 — Open Frontier Intelligence (2.8T MoE with KDA + AttnRes): Kimi K3 pairs MXFP4 weights + MXFP8 activations (QAT from SFT), KDA prefix-cache in vLLM, Mooncake disaggregated inference (>90% cache-hit rate reported on coding workloads), and supernode-scale (≥64) expert parallelism to serve a 2.8T model at 0.30/0.30 / 3.00 / $15.00 per MTok (cache-hit input / cache-miss input / output).
  • [2026-07-14] Xiaomi-Robotics-U0: Unified Embodied Synthesis with World Foundation Model: FlashAR+ combines diagonal-parallel decoding, lightweight acceleration adaptation post-training, and vLLM paged-KV-cache batch scheduling to accelerate autoregressive image/video generation on a 38B model up to 82.9×, cutting 1024² latency from 450.77s to 5.44s.
  • [2026-07-14] 1-bit Bonsai 27B — GGUF: 27B-class Reasoning at 1.125 Bits Per Weight: PrismML scales end-to-end binary weights from 8B up to a 27B-class Qwen3.6-27B derivative at 1.125 bpw / 3.9 GB — 76.11 average across 15 thinking-mode benchmarks (89.5% of FP16), with math holding at 91.66 and coding at 81.88, while conventional Qwen3.6-27B IQ2_XXS at 2.8 bpw collapses to 72.73. First filed weight-quantization release that also ships its own speculative-decoding drafter (DSpark, 1.37× lossless on H100), addressing the open question of whether 1-bit and SD compose. Also demonstrates that a hybrid-linear backbone survives extreme quantization: full 262K context fits in ~9.4 GB peak with 4-bit KV cache.
  • [2026-07-10] The 4-bitter Lesson: Balancing Stability and Performance in NVFP4 RL: Online NVFP4 serving reuses the RL trainer’s rollout quantization kernel to convert BF16/FP8 checkpoints at load time in SGLang (--quantization nvfp4_online) — no calibration, QAT, or model-conversion pipeline. On an internal GLM-5.1-judged multi-turn tool-use benchmark, correlation between FP8 and NVFP4-online deployments is 0.924, and the same 4/6 + selective-precision improvements carry into inference.
  • [2026-07-09] Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity: SDM decouples linear-RNN state capacity from per-token FLOPs via sparse addressing — the compute cost of a large memory bank becomes gather/scatter rather than dense outer-product accumulation. Extends the “cache-size, not parameter-count, is the efficiency target” argument of Jet-Nemotron to a “state-slot count, not state-vector width” reframing on the linear-attention branch.
  • [2026-07-06] Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots: Embodied.cpp argues LLM-serving runtimes (vLLM/SGLang/llama.cpp) fit request-response batch-N throughput but not the embodied-deployment contract of multi-rate closed-loop execution, batch-1 latency-first inference, and image+action+language+world-latent I/O — proposes a five-layer VLA/WAM runtime with GGUF weights and one CPU/CUDA/NPU backend abstraction, first filed data point in this regime distinct from the throughput-first LLM serving optimizations logged elsewhere on this page.
  • [2026-06-24] Next Forcing: Causal World Modeling with Multi-Chunk Prediction: First filed video-AR instance of MTP-as-speculative-decoding: Next Forcing retains its auxiliary multi-chunk-prediction heads at inference to predict the next video chunk in parallel with the current one, yielding 2× inference acceleration — the video-diffusion analogue of DeepSeek/GLM MTP speculative decoding.
  • [2026-06-23] ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?: ImageWAM (robotics) avoids decoding a target frame at inference and instead feeds the image-editing denoiser’s KV cache straight to a flow-matching action expert — same cache-as-output trick the wiki has seen for LLM/diffusion serving, here at the recipe layer of the model; reports ~1/6 FLOPs and ~1/4 latency vs video-based WAMs.
  • [2026-06-20] HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization: HydraHead targets long-context decoding cost by hybridizing FA/LA along the head axis within a layer rather than across layers — preserves FA only on retrieval-critical heads (selected via interpretability analysis), converts the rest to LA, with a scale-normalized fusion module. Reaches a 7:1 LA-to-FA ratio matching a 3:1 layer-wise hybrid’s long-context quality at 512K, on a 15B-token distillation budget — implying layer-wise hybrids may be over-allocating FA capacity at the head level.
  • [2026-06-18] M*: A Modular, Extensible, Serving System for Multimodal Models: Stanford’s M* — a unified serving runtime for composite multimodal models (UMMs, SpeechLMs, Omni, VLAs, world models) built around the Walk Graph abstraction. Each model is a graph of component nodes with named Walks selected per request, and Sequential / Parallel / Loop / Stream are first-class primitives. Generalizes vLLM-Omni’s flat stage-DAG (explicitly identified as a restricted subset) so that loops can span any subgraph and parallelism (CFG fan-out) is expressed natively rather than via per-model torch.distributed glue. Continuous batching and CUDA-graph replay apply to diffusion steps, world-model rollouts, and AR decode uniformly. Benchmark headlines: ≈2.7× Qwen3-Omni TTS throughput vs vLLM-Omni and ≈4× vs SGLang-Omni at B=16; ≈1.6× faster first token on BAGEL image→text; ≈1.3× higher throughput vs VoxServe on Orpheus TTS; up to 12.5× on V-JEPA 2 rollout from treating the rollout as a Loop with persistent KV cache. The 12.5× number is the largest single multiplier on this page and is purely structural (no kernel changes). Closes the page’s open question of “what does the framework-level analog of vLLM-Omni + SGLang Diffusion + a hand-written bridge look like” — the answer is one Walk Graph runtime.
  • [2026-05-26] Laguna M.1/XS.2 Technical Report: Laguna XS.2 ships with FP8, INT4, and NVFP4 quantization recipes for the MoE layers plus FP8 KV cache and a documented mixed-precision strategy aimed specifically at low-VRAM-device deployment.
  • [2026-05-26] PrismML — 1-bit and Ternary Bonsai Image 4B — local-hardware diffusion image generation: PrismML teases Bonsai Image 4B in 1-bit and ternary variants — the first signal that the BitNet-style end-to-end low-bit weight discipline used in the LLM Bonsai release (March 2026) is being extended to the diffusion image-generation regime. Tweet is a launch teaser only, no benchmarks/whitepaper/model card yet; tracked as a directional datapoint, not a result.
  • [2026-05-26] FaaScale: Unlocking Fast LLM Scaling for Serverless Inference: FaaScale (MLSys 2026) attacks the serverless-LLM cold-start tail rather than per-instance decode speed: production traces show >10× minute-scale request spikes against 30+ s SSD→GPU loads and 36–64% host-memory cache miss rates, breaking caching-only strategies. PipeCast multicasts fine-grained model blocks via a binomial-pipeline + circular-shift k-way schedule over GDR/RDMA and forms cross-node pipeline-parallel execution instances during transfer so the first complete model is assembleable across receivers after b/k steps instead of b — and inference starts before any single node holds the full model. Reports 2.4–5× P90 TTFT improvement and 17.8–31.3% GPU-cost reduction vs ServerlessLLM / FaaSNet / NCCL on real LLM traces. A new “make instance creation fast” lever distinct from speculative decoding, KV-cache compression, weight quantization, or prefix-cache-friendly decoding.
  • [2026-05-26] LT2: Linear-Time Looped Transformers: LT2 — looped Transformers retrofitted with linear/sparse attention to remove the L × N² inference cost. Ouro-hybrid-1.4B is produced by converting a pretrained Ouro looped checkpoint with only ~1B tokens of continued training, then runs at linear-time inference while reportedly matching industry 4B models. Adds a “convert a looped checkpoint to subquadratic attention” lever distinct from speculative decoding, cache compression, or weight quantization.
  • [2026-05-24] Interaction Models: A Scalable Approach to Human-AI Collaboration: Thinking Machines Lab announces a real-time multimodal interaction model (276B MoE / 12B active) running on 200ms micro-turns. Two new levers for this page that the existing speculative-decoding / cache-compression / weight-quantization threads don’t cover: (a) streaming sessions — chunk-as-request with server-side sequence append, upstreamed to SGLang (PR #19171), tackling the frequent-small-prefill regime; (b) a gather + gemv MoE kernel for small-batch streaming, citing the same diagnosis as PyTorch’s mixtral-in-gpt-fast and Cursor’s warp decode. Reaches 0.40s FD-bench V1 turn-taking latency — the first datapoint on this page that’s bottlenecked by interactive latency floor rather than aggregate throughput.
  • [2026-05-23] 1-bit Bonsai 8B: End-to-End 1-bit Language Models from PrismML: PrismML emerges from stealth with the 1-bit Bonsai family — 8B/4B/1.7B Apache-2.0 LLMs whose entire network is 1-bit (embeddings, attention, MLP, LM head). 1.15 GB footprint, ~10.6× “intelligence density” (their negative-log-error/GB metric) over FP16 Qwen3 8B, claimed parity with FP16 8B instruct models at 70.5 avg across 6 categories. Adds a lossy weight-quantization lever to a page previously focused on lossless decode-time speed-ups; the speedup comes from running a much smaller model on commodity hardware rather than from cache or speculation tricks. Caveats: vendor-reported, no whitepaper yet, requires forked llama.cpp and mlx for the Q1_0 / 1-bit kernels. Worth tracking against the methodological warning of Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments — independent symmetric-baseline reproduction will matter.
  • [2026-05-23] Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments: Public technical-note rebuttal by the RaBitQ authors to the ICLR 2026 TurboQuant paper. Three findings: (i) TurboQuant and RaBitQ share the same JL-rotation-then-per-coordinate-quantize primitive (which TurboQuant doesn’t acknowledge); (ii) TurboQuant’s variance-only bound yields a 1/δ² bit-width scaling under Chebyshev, exponentially worse than RaBitQ’s optimal sub-Gaussian log(1/δ); (iii) under symmetric A100 + C++-RaBitQ-library setup, RaBitQ beats TurboQuant on most inner-product, ANN, and KV-cache benchmarks, and TurboQuant’s own reported wall-clock numbers cannot be reproduced from its released code (the original RaBitQ baseline was single-core CPU vs A100 GPU TurboQuant — undisclosed). Adds a baseline-parity / disclosure dimension to the concept that complements the speculative-decoding “verification dominates production wall-clock” lesson.
  • [2026-05-23] Speculative Decoding: Performance or Illusion?: First systematic SD evaluation inside vLLM with full production optimizations. Five SD variants × four models × six datasets × batch sizes 1–128. Headline findings: (i) verification time dominates 42–95%, (ii) speedup decays monotonically with batch and steeper at larger model sizes, (iii) SD variant choice is workload-dependent — n-gram wins on code, draft-model wins on 70B, EAGLE-3 wins on reasoning, MTP wins when full heads ship, (iv) per-position adaptive combination lifts speedup to 4.9× over no-SD, (v) greedy outputs are not bitwise reproducible under SD due to kernel-level nondeterminism — token throughput, not per-request latency, is the correct metric. Companion: simulator and code at https://github.com/SpecDecode-Bench.
  • [2026-05-23] TorchSpec: Speculative Decoding Training at Scale: TorchSpec — torch-native framework for training EAGLE-3 draft models at scale by fully disaggregating the inference cluster (target model producing hidden states via vLLM/SGLang) from the training cluster, with Mooncake handling RDMA/TCP zero-copy transfer. Eliminates ~700 TB of disk I/O and frees training GPUs of the 575 GB target weights, unlocking 200k-token context on a single B200. Trains a Kimi K2.5 EAGLE-3 draft in 1500 H200-hours; deployed draft delivers +60%/+30%/+26% throughput at batch 1/8/16. Pattern likely generalizes to RL rollouts and reward-model inference.
  • [2026-05-23] Speculative Speculative Decoding: Saguaro/SSD — recognizes that draft and verify still serialize against each other in SD and parallelizes them on distinct hardware via a speculation cache keyed on verification outcomes. Capped-geometric fan-out across speculation positions (Theorem 12); residual-distribution biasing via β-down-weighting; batch-size-aware cache-miss fallback. Reported average 30% over EAGLE-3/SGLang/vLLM SD and up to 5× over autoregressive on Llama-3 — and Pareto-shifts latency–throughput, not just latency.
  • [2026-05-22] WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference: Defines prefix cacheability as the right efficiency metric for KV-cached serving and shows masked diffusion LMs can be reorganized to be prefix-cache-friendly via Topological Reordering — physical-vs-logical position decoupling under standard causal attention. WeDLM-8B reaches ~3× over vLLM-served Qwen3-8B on reasoning and >10× on low-entropy generation without kernel changes.

Open questions

Section titled “Open questions”
  • Are SSD’s draft↔verify parallelization, TorchSpec’s higher-acceptance draft training, and per-position adaptive variant routing additive? All three target the same “verification on rejected tokens is wasted” diagnosis but from different sides of the pipeline. No filed paper combines two of them.
  • The 4.9× adaptive-combination upper bound (Speculative Decoding: Performance or Illusion?) requires per-position acceptance prediction at runtime. Is that prediction cheap enough not to eat its own gains? The simulator runs offline; the deployment policy is not specified.
  • TorchSpec produces drafts that hold up at batch=16 with +26% throughput, but the systematic benchmark suggests SD speedup decays sharply beyond batch ~32 on 8B targets. At what target batch size does production SD stop justifying its memory overhead even with a perfect draft model?
  • Output nondeterminism under greedy decoding (Speculative Decoding: Performance or Illusion? §3.1) breaks reproducibility for eval harnesses. Is there a deterministic SD variant that’s still fast, or do all eval pipelines now need to switch from per-prompt latency to token-throughput? Implications for offline benchmarking are unclear.
  • WeDLM proves DLMs can match vLLM-AR wall-clock by being cache-friendly, but the comparison is against the base Qwen3-8B, not Qwen3-8B + SD + KV compression. Does a fully-optimized AR + SD + DMS stack still lose to WeDLM at >10× on low-entropy generation, or does it close the gap?
  • SD variant selection is workload-dependent. A production inference router would need to route between EAGLE-3 / draft-model / n-gram / MTP at runtime based on workload signal. No filed paper proposes such a router or characterizes the routing signal.
  • The MTP underperformance on GLM-4.5-Air is attributed specifically to only the first MTP module being released open-source. If the full multi-head MTP weights were released, would MTP categorically beat EAGLE-3 on reasoning?
  • The RaBitQ-vs-TurboQuant rebuttal raises a methodological open question for the wiki: which inference-efficiency claims in the existing literature would survive symmetric baselines? The pattern (single-threaded reference impl on CPU vs heavily optimized proposed method on GPU) is not unique to TurboQuant; how often does it apply elsewhere? Bonsai 8B’s vendor-reported 6.2× / 8.4× / 44 tok/s numbers (against unspecified FP16 baselines) are a fresh candidate for this scrutiny.
  • Does end-to-end 1-bit quantization compose with SD/SSD? A 1-bit 8B draft for a 70B FP16 target could in principle cut draft-side memory dramatically, but no filed work tests this — and the Bonsai release doesn’t disclose whether its training recipe produces a model whose distribution is close enough to a teacher for SD acceptance to remain high.
  • The real-time streaming regime (200ms chunks, sub-second turn latency) has very different bottlenecks from batch serving: chunk-as-request append overhead, MoE kernel choice at batch ≈ 1, and bitwise trainer-sampler alignment for deterministic state. Does any of the speculative-decoding or cache-compression machinery transfer to this regime, or does it need an orthogonal stack (Interaction Models: A Scalable Approach to Human-AI Collaboration)?

Papers

Section titled “Papers”

2026-11

Section titled “2026-11”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-08

Section titled “2025-08”

2025-07

Section titled “2025-07”

2025-06

Section titled “2025-06”

2025-05

Section titled “2025-05”

2025-04

Section titled “2025-04”

2024-12

Section titled “2024-12”

Unknown date

Section titled “Unknown date”