Skip to content
Bud's Wiki

LLM Inference Efficiency

TL;DR

Section titled “TL;DR”

Lossless or near-lossless speed-ups for serving LLMs — methods that change how the target model is invoked at decode time without changing what it outputs in distribution. Three sub-threads are converging on the same diagnosis: under production serving (vLLM/SGLang with continuous batching, CUDA Graphs, KV-cache management), the dominant cost is target-model verification compute and KV-cache memory traffic, not drafting or arithmetic. The filed papers split into (a) speculative decoding — drafting + verification, with empirical evidence that verification dominates 42–95% of execution time and that drafting/verification overlap is the next opportunity, (b) cache-prefix-friendly non-AR decoders — diffusion LMs that keep a standard causal mask so they remain compatible with vLLM/FlashAttention/PagedAttention, and (c) upstream training infrastructure for the draft side of SD. Distinct from inference-time-scaling, which trades more compute for better outputs; this concept is about producing the same output faster.

Key claims

Section titled “Key claims”
  • In production serving (vLLM v0.10.1.1, all optimizations enabled), every speculative decoding variant beats no-SD at small batch but the speedup decays monotonically with batch size and the decay is steeper for larger models — at batch 128 on Llama3.1-8B/GSM8K, EAGLE drops from 1.73× to 1.21× (Speculative Decoding: Performance or Illusion? §3.2, Fig. 1).
  • Verification — running the target model on proposed tokens — dominates the SD time breakdown at 42–95% across configurations; drafting is <2% for n-gram, 12–20% for EAGLE/EAGLE-3 at batch 1 (collapsing to 3–7% at batch 512), and 21–47% for draft-model SD at batch 1; rejection sampling is <1.7% (Speculative Decoding: Performance or Illusion? §5.1, Fig. 3).
  • SD variant selection is workload-dependent, not universal: n-gram wins on code-editing (InstructCoder, by token reuse), draft-model wins on 70B targets, EAGLE-3 wins on reasoning workloads with long outputs, and MTP wins on GLM-4.5-Air when full multi-head MTP weights are released (Speculative Decoding: Performance or Illusion? §3.2).
  • Adaptive per-position combination of SD variants lifts end-to-end speedup to 4.9× over no-SD on Llama-family targets, far beyond any single variant in isolation — the per-position acceptance complementarity across variants is the largest remaining opportunity (Speculative Decoding: Performance or Illusion? §1, §8).
  • Speculative decoding is rate-limited by sequential drafting↔verification: the next speculation cannot start until the current verification finishes. Speculative Speculative Decoding (SSD/Saguaro) parallelizes them by pre-speculating against the most likely verification outcomes on distinct hardware, hiding draft latency on cache hits (Speculative Speculative Decoding §1, §3.1).
  • SSD/Saguaro achieves average ~30% wall-clock speedup over the strongest open SD baselines (SGLang/vLLM, EAGLE-3) and up to 5× vs. autoregressive decoding on Llama-3; effects hold at low/medium batch and shift the latency–throughput Pareto frontier (Speculative Speculative Decoding §5, Fig. 7).
  • Optimal Saguaro cache shape under a power-law cache-hit-rate assumption is a capped geometric fan-out across speculation positions: spend more outcome guesses early where acceptance is likely (Speculative Speculative Decoding Theorem 12).
  • Producing a high-quality EAGLE-3 draft model for trillion-parameter targets is itself bottlenecked by storage (~700 TB to precompute hidden states for 100k samples on Kimi K2.5) or co-located memory (target weights eat 575 GB on 8×H100, capping context to ~4096 tokens). TorchSpec disaggregates inference and training across clusters with Mooncake RDMA streaming hidden states, training a Kimi K2.5 EAGLE-3 draft in 1500 H200-hours on 600k samples and unlocking 44k-token context on a single H100 / 200k on a single B200 (TorchSpec: Speculative Decoding Training at Scale §Background, §Long Context Support).
  • The TorchSpec-trained K2.5 EAGLE-3 draft delivers +60% throughput at batch=1, +30% at batch=8, +26% at batch=16 with lookahead=3 — i.e. the draft remains good enough to justify SD overhead at multi-tenant batch sizes, not just at batch=1 (TorchSpec: Speculative Decoding Training at Scale §Introduction).
  • In KV-cached serving, the right efficiency metric is prefix cacheability P_c = N_new / N_total — the fraction of processed token instances that end up as cache-reusable committed prefix — not tokens-per-forward. This reframes the diffusion-LM vs. AR debate and explains why many parallel-decoding papers don’t translate per-step token counts into wall-clock wins (WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference §3.1, Eq. 4).
  • WeDLM achieves prefix cacheability by topological reordering: observed tokens are physically moved to the prefix while RoPE position ids preserve logical positions, so masked tokens attend to all observed context under an unmodified causal mask — directly compatible with FlashAttention/PagedAttention/CUDA Graphs without kernel changes (WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference §4.1, §5.2).
  • WeDLM-8B (continued-pretrained from Qwen3-8B on 100B tokens) served via vLLM reaches ~3× end-to-end speedup on complex reasoning and >10× on low-entropy generation versus the same vLLM-served Qwen3-8B baseline, while matching or exceeding base on quality (WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference §1, §6.4, Table 1).
  • Output nondeterminism is real even under greedy decoding (T=0, top_p=1, top_k=-1): SD vs no-SD outputs are not bitwise identical, with generation length varying across batch sizes and SD variants. Attributed to GPU kernel parallelism and floating-point variation; implies token-throughput, not per-request latency, is the correct metric (Speculative Decoding: Performance or Illusion? §3.1, Tab. 1).
  • Inference-efficiency comparisons are extremely sensitive to baseline implementation parity: under a reproducible symmetric setup, TurboQuant’s published 6×/8× speed-and-memory advantages over RaBitQ for KV-cache quantization do not hold — RaBitQ outperforms TurboQuant in most inner-product, ANN, and KV-cache settings, the TurboQuant baseline for RaBitQ ran single-threaded on CPU while TurboQuant ran on A100 GPU (undisclosed), and several of TurboQuant’s own reported runtimes cannot be reproduced from the released artifact (Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments §1, §4.1, §4.2).
  • A variance bound on a quantization estimator is not the same caliber of guarantee as a sub-Gaussian tail bound: TurboQuant’s Theorem 2 (variance) converts via Chebyshev to bit-width scaling 1/ε² · 1/δ², whereas RaBitQ’s sub-Gaussian tail gives 1/ε² · log(1/δ) and matches the Alon–Klartag (FOCS 2017) optimal lower bound (Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments §3, Lemma 3.1).
  • Aggressive weight-level quantization can be pushed to 1-bit end-to-end (embeddings, attention projections, MLPs, LM head — no FP16 escape hatches) and still claim benchmark parity with FP16 8B baselines at ~1/14 the size: PrismML’s 1-bit Bonsai 8B reports 1.15 GB footprint, 6.2× faster on RTX 4090, 8.4× faster on M4 Pro, 44 tok/s on iPhone 17 Pro Max, 4–5× lower energy per token, and a 70.5 average across 6 benchmark categories — all vendor-reported, pre-whitepaper, and dependent on custom Q1_0 / MLX dequantization kernels in forked runtimes (1-bit Bonsai 8B: End-to-End 1-bit Language Models from PrismML).
  • Real-time streaming inference (200ms chunks of audio/video/text) introduces a regime that existing LLM inference libraries are not optimized for: frequent small prefills + decodes with per-turn metadata overhead. Thinking Machines’ streaming-session primitive treats each chunk as an independent request that the server appends to a persistent in-GPU sequence, avoiding repeated memory reallocation and metadata-computation overhead. A version of this has been upstreamed to SGLang as PR #19171 (Interaction Models: A Scalable Approach to Human-AI Collaboration §Inference optimization).
  • A gather + gemv MoE kernel beats grouped-GEMM in the streaming small-batch regime, where there isn’t enough shared work per expert to amortize the gather-scatter-combine bookkeeping; the same diagnosis appears in PyTorch’s “supporting Mixtral in gpt-fast” and in Better MoE model inference with warp decode‘s warp-decode kernel (Interaction Models: A Scalable Approach to Human-AI Collaboration §Inference optimization).

Recent contributions

Section titled “Recent contributions”
  • [2026-07-17] Sunday Robotics finds on-device VLA inference beats cloud during ACT-2 eval (Cheng Chi): Sunday Robotics deployment observation: at single-robot residential-network scale, cloud inference for a home VLA is tail-latency-limited by the WiFi/ISP hop, not by the model’s per-step compute — Cheng Chi reports moving ACT-2 to on-device inference two days into their eval after concluding the trained model was more reliable than the average home network. First wiki datapoint on the latency-tail-first, single-request regime that sits orthogonal to the throughput-first speculative-decoding / KV-cache / streaming-session threads on this page, and field-evidence support for Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots‘s claim that VLA deployment is a structurally different serving contract from LLM inference.
  • [2026-07-16] Kimi K3 — Open Frontier Intelligence (2.8T MoE with KDA + AttnRes): Kimi K3 pairs MXFP4 weights + MXFP8 activations (QAT from SFT), KDA prefix-cache in vLLM, Mooncake disaggregated inference (>90% cache-hit rate reported on coding workloads), and supernode-scale (≥64) expert parallelism to serve a 2.8T model at 0.30/0.30 / 3.00 / $15.00 per MTok (cache-hit input / cache-miss input / output).
  • [2026-07-14] Xiaomi-Robotics-U0: Unified Embodied Synthesis with World Foundation Model: FlashAR+ combines diagonal-parallel decoding, lightweight acceleration adaptation post-training, and vLLM paged-KV-cache batch scheduling to accelerate autoregressive image/video generation on a 38B model up to 82.9×, cutting 1024² latency from 450.77s to 5.44s.
  • [2026-07-14] 1-bit Bonsai 27B — GGUF: 27B-class Reasoning at 1.125 Bits Per Weight: PrismML scales end-to-end binary weights from 8B up to a 27B-class Qwen3.6-27B derivative at 1.125 bpw / 3.9 GB — 76.11 average across 15 thinking-mode benchmarks (89.5% of FP16), with math holding at 91.66 and coding at 81.88, while conventional Qwen3.6-27B IQ2_XXS at 2.8 bpw collapses to 72.73. First filed weight-quantization release that also ships its own speculative-decoding drafter (DSpark, 1.37× lossless on H100), addressing the open question of whether 1-bit and SD compose. Also demonstrates that a hybrid-linear backbone survives extreme quantization: full 262K context fits in ~9.4 GB peak with 4-bit KV cache.
  • [2026-07-10] The 4-bitter Lesson: Balancing Stability and Performance in NVFP4 RL: Online NVFP4 serving reuses the RL trainer’s rollout quantization kernel to convert BF16/FP8 checkpoints at load time in SGLang (--quantization nvfp4_online) — no calibration, QAT, or model-conversion pipeline. On an internal GLM-5.1-judged multi-turn tool-use benchmark, correlation between FP8 and NVFP4-online deployments is 0.924, and the same 4/6 + selective-precision improvements carry into inference.
  • [2026-07-09] Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity: SDM decouples linear-RNN state capacity from per-token FLOPs via sparse addressing — the compute cost of a large memory bank becomes gather/scatter rather than dense outer-product accumulation. Extends the “cache-size, not parameter-count, is the efficiency target” argument of Jet-Nemotron to a “state-slot count, not state-vector width” reframing on the linear-attention branch.
  • [2026-07-06] Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots: Embodied.cpp argues LLM-serving runtimes (vLLM/SGLang/llama.cpp) fit request-response batch-N throughput but not the embodied-deployment contract of multi-rate closed-loop execution, batch-1 latency-first inference, and image+action+language+world-latent I/O — proposes a five-layer VLA/WAM runtime with GGUF weights and one CPU/CUDA/NPU backend abstraction, first filed data point in this regime distinct from the throughput-first LLM serving optimizations logged elsewhere on this page.
  • [2026-06-24] Next Forcing: Causal World Modeling with Multi-Chunk Prediction: First filed video-AR instance of MTP-as-speculative-decoding: Next Forcing retains its auxiliary multi-chunk-prediction heads at inference to predict the next video chunk in parallel with the current one, yielding 2× inference acceleration — the video-diffusion analogue of DeepSeek/GLM MTP speculative decoding.
  • [2026-06-23] ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?: ImageWAM (robotics) avoids decoding a target frame at inference and instead feeds the image-editing denoiser’s KV cache straight to a flow-matching action expert — same cache-as-output trick the wiki has seen for LLM/diffusion serving, here at the recipe layer of the model; reports ~1/6 FLOPs and ~1/4 latency vs video-based WAMs.
  • [2026-06-20] HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization: HydraHead targets long-context decoding cost by hybridizing FA/LA along the head axis within a layer rather than across layers — preserves FA only on retrieval-critical heads (selected via interpretability analysis), converts the rest to LA, with a scale-normalized fusion module. Reaches a 7:1 LA-to-FA ratio matching a 3:1 layer-wise hybrid’s long-context quality at 512K, on a 15B-token distillation budget — implying layer-wise hybrids may be over-allocating FA capacity at the head level.
  • [2026-06-18] M*: A Modular, Extensible, Serving System for Multimodal Models: Stanford’s M* — a unified serving runtime for composite multimodal models (UMMs, SpeechLMs, Omni, VLAs, world models) built around the Walk Graph abstraction. Each model is a graph of component nodes with named Walks selected per request, and Sequential / Parallel / Loop / Stream are first-class primitives. Generalizes vLLM-Omni’s flat stage-DAG (explicitly identified as a restricted subset) so that loops can span any subgraph and parallelism (CFG fan-out) is expressed natively rather than via per-model torch.distributed glue. Continuous batching and CUDA-graph replay apply to diffusion steps, world-model rollouts, and AR decode uniformly. Benchmark headlines: ≈2.7× Qwen3-Omni TTS throughput vs vLLM-Omni and ≈4× vs SGLang-Omni at B=16; ≈1.6× faster first token on BAGEL image→text; ≈1.3× higher throughput vs VoxServe on Orpheus TTS; up to 12.5× on V-JEPA 2 rollout from treating the rollout as a Loop with persistent KV cache. The 12.5× number is the largest single multiplier on this page and is purely structural (no kernel changes). Closes the page’s open question of “what does the framework-level analog of vLLM-Omni + SGLang Diffusion + a hand-written bridge look like” — the answer is one Walk Graph runtime.
  • [2026-05-26] Laguna M.1/XS.2 Technical Report: Laguna XS.2 ships with FP8, INT4, and NVFP4 quantization recipes for the MoE layers plus FP8 KV cache and a documented mixed-precision strategy aimed specifically at low-VRAM-device deployment.
  • [2026-05-26] PrismML — 1-bit and Ternary Bonsai Image 4B — local-hardware diffusion image generation: PrismML teases Bonsai Image 4B in 1-bit and ternary variants — the first signal that the BitNet-style end-to-end low-bit weight discipline used in the LLM Bonsai release (March 2026) is being extended to the diffusion image-generation regime. Tweet is a launch teaser only, no benchmarks/whitepaper/model card yet; tracked as a directional datapoint, not a result.
  • [2026-05-26] FaaScale: Unlocking Fast LLM Scaling for Serverless Inference: FaaScale (MLSys 2026) attacks the serverless-LLM cold-start tail rather than per-instance decode speed: production traces show >10× minute-scale request spikes against 30+ s SSD→GPU loads and 36–64% host-memory cache miss rates, breaking caching-only strategies. PipeCast multicasts fine-grained model blocks via a binomial-pipeline + circular-shift k-way schedule over GDR/RDMA and forms cross-node pipeline-parallel execution instances during transfer so the first complete model is assembleable across receivers after b/k steps instead of b — and inference starts before any single node holds the full model. Reports 2.4–5× P90 TTFT improvement and 17.8–31.3% GPU-cost reduction vs ServerlessLLM / FaaSNet / NCCL on real LLM traces. A new “make instance creation fast” lever distinct from speculative decoding, KV-cache compression, weight quantization, or prefix-cache-friendly decoding.
  • [2026-05-26] LT2: Linear-Time Looped Transformers: LT2 — looped Transformers retrofitted with linear/sparse attention to remove the L × N² inference cost. Ouro-hybrid-1.4B is produced by converting a pretrained Ouro looped checkpoint with only ~1B tokens of continued training, then runs at linear-time inference while reportedly matching industry 4B models. Adds a “convert a looped checkpoint to subquadratic attention” lever distinct from speculative decoding, cache compression, or weight quantization.
  • [2026-05-24] Interaction Models: A Scalable Approach to Human-AI Collaboration: Thinking Machines Lab announces a real-time multimodal interaction model (276B MoE / 12B active) running on 200ms micro-turns. Two new levers for this page that the existing speculative-decoding / cache-compression / weight-quantization threads don’t cover: (a) streaming sessions — chunk-as-request with server-side sequence append, upstreamed to SGLang (PR #19171), tackling the frequent-small-prefill regime; (b) a gather + gemv MoE kernel for small-batch streaming, citing the same diagnosis as PyTorch’s mixtral-in-gpt-fast and Cursor’s warp decode. Reaches 0.40s FD-bench V1 turn-taking latency — the first datapoint on this page that’s bottlenecked by interactive latency floor rather than aggregate throughput.
  • [2026-05-23] 1-bit Bonsai 8B: End-to-End 1-bit Language Models from PrismML: PrismML emerges from stealth with the 1-bit Bonsai family — 8B/4B/1.7B Apache-2.0 LLMs whose entire network is 1-bit (embeddings, attention, MLP, LM head). 1.15 GB footprint, ~10.6× “intelligence density” (their negative-log-error/GB metric) over FP16 Qwen3 8B, claimed parity with FP16 8B instruct models at 70.5 avg across 6 categories. Adds a lossy weight-quantization lever to a page previously focused on lossless decode-time speed-ups; the speedup comes from running a much smaller model on commodity hardware rather than from cache or speculation tricks. Caveats: vendor-reported, no whitepaper yet, requires forked llama.cpp and mlx for the Q1_0 / 1-bit kernels. Worth tracking against the methodological warning of Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments — independent symmetric-baseline reproduction will matter.
  • [2026-05-23] Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments: Public technical-note rebuttal by the RaBitQ authors to the ICLR 2026 TurboQuant paper. Three findings: (i) TurboQuant and RaBitQ share the same JL-rotation-then-per-coordinate-quantize primitive (which TurboQuant doesn’t acknowledge); (ii) TurboQuant’s variance-only bound yields a 1/δ² bit-width scaling under Chebyshev, exponentially worse than RaBitQ’s optimal sub-Gaussian log(1/δ); (iii) under symmetric A100 + C++-RaBitQ-library setup, RaBitQ beats TurboQuant on most inner-product, ANN, and KV-cache benchmarks, and TurboQuant’s own reported wall-clock numbers cannot be reproduced from its released code (the original RaBitQ baseline was single-core CPU vs A100 GPU TurboQuant — undisclosed). Adds a baseline-parity / disclosure dimension to the concept that complements the speculative-decoding “verification dominates production wall-clock” lesson.
  • [2026-05-23] Speculative Decoding: Performance or Illusion?: First systematic SD evaluation inside vLLM with full production optimizations. Five SD variants × four models × six datasets × batch sizes 1–128. Headline findings: (i) verification time dominates 42–95%, (ii) speedup decays monotonically with batch and steeper at larger model sizes, (iii) SD variant choice is workload-dependent — n-gram wins on code, draft-model wins on 70B, EAGLE-3 wins on reasoning, MTP wins when full heads ship, (iv) per-position adaptive combination lifts speedup to 4.9× over no-SD, (v) greedy outputs are not bitwise reproducible under SD due to kernel-level nondeterminism — token throughput, not per-request latency, is the correct metric. Companion: simulator and code at https://github.com/SpecDecode-Bench.
  • [2026-05-23] TorchSpec: Speculative Decoding Training at Scale: TorchSpec — torch-native framework for training EAGLE-3 draft models at scale by fully disaggregating the inference cluster (target model producing hidden states via vLLM/SGLang) from the training cluster, with Mooncake handling RDMA/TCP zero-copy transfer. Eliminates ~700 TB of disk I/O and frees training GPUs of the 575 GB target weights, unlocking 200k-token context on a single B200. Trains a Kimi K2.5 EAGLE-3 draft in 1500 H200-hours; deployed draft delivers +60%/+30%/+26% throughput at batch 1/8/16. Pattern likely generalizes to RL rollouts and reward-model inference.
  • [2026-05-23] Speculative Speculative Decoding: Saguaro/SSD — recognizes that draft and verify still serialize against each other in SD and parallelizes them on distinct hardware via a speculation cache keyed on verification outcomes. Capped-geometric fan-out across speculation positions (Theorem 12); residual-distribution biasing via β-down-weighting; batch-size-aware cache-miss fallback. Reported average 30% over EAGLE-3/SGLang/vLLM SD and up to 5× over autoregressive on Llama-3 — and Pareto-shifts latency–throughput, not just latency.
  • [2026-05-22] WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference: Defines prefix cacheability as the right efficiency metric for KV-cached serving and shows masked diffusion LMs can be reorganized to be prefix-cache-friendly via Topological Reordering — physical-vs-logical position decoupling under standard causal attention. WeDLM-8B reaches ~3× over vLLM-served Qwen3-8B on reasoning and >10× on low-entropy generation without kernel changes.

Open questions

Section titled “Open questions”
  • Are SSD’s draft↔verify parallelization, TorchSpec’s higher-acceptance draft training, and per-position adaptive variant routing additive? All three target the same “verification on rejected tokens is wasted” diagnosis but from different sides of the pipeline. No filed paper combines two of them.
  • The 4.9× adaptive-combination upper bound (Speculative Decoding: Performance or Illusion?) requires per-position acceptance prediction at runtime. Is that prediction cheap enough not to eat its own gains? The simulator runs offline; the deployment policy is not specified.
  • TorchSpec produces drafts that hold up at batch=16 with +26% throughput, but the systematic benchmark suggests SD speedup decays sharply beyond batch ~32 on 8B targets. At what target batch size does production SD stop justifying its memory overhead even with a perfect draft model?
  • Output nondeterminism under greedy decoding (Speculative Decoding: Performance or Illusion? §3.1) breaks reproducibility for eval harnesses. Is there a deterministic SD variant that’s still fast, or do all eval pipelines now need to switch from per-prompt latency to token-throughput? Implications for offline benchmarking are unclear.
  • WeDLM proves DLMs can match vLLM-AR wall-clock by being cache-friendly, but the comparison is against the base Qwen3-8B, not Qwen3-8B + SD + KV compression. Does a fully-optimized AR + SD + DMS stack still lose to WeDLM at >10× on low-entropy generation, or does it close the gap?
  • SD variant selection is workload-dependent. A production inference router would need to route between EAGLE-3 / draft-model / n-gram / MTP at runtime based on workload signal. No filed paper proposes such a router or characterizes the routing signal.
  • The MTP underperformance on GLM-4.5-Air is attributed specifically to only the first MTP module being released open-source. If the full multi-head MTP weights were released, would MTP categorically beat EAGLE-3 on reasoning?
  • The RaBitQ-vs-TurboQuant rebuttal raises a methodological open question for the wiki: which inference-efficiency claims in the existing literature would survive symmetric baselines? The pattern (single-threaded reference impl on CPU vs heavily optimized proposed method on GPU) is not unique to TurboQuant; how often does it apply elsewhere? Bonsai 8B’s vendor-reported 6.2× / 8.4× / 44 tok/s numbers (against unspecified FP16 baselines) are a fresh candidate for this scrutiny.
  • Does end-to-end 1-bit quantization compose with SD/SSD? A 1-bit 8B draft for a 70B FP16 target could in principle cut draft-side memory dramatically, but no filed work tests this — and the Bonsai release doesn’t disclose whether its training recipe produces a model whose distribution is close enough to a teacher for SD acceptance to remain high.
  • The real-time streaming regime (200ms chunks, sub-second turn latency) has very different bottlenecks from batch serving: chunk-as-request append overhead, MoE kernel choice at batch ≈ 1, and bitwise trainer-sampler alignment for deterministic state. Does any of the speculative-decoding or cache-compression machinery transfer to this regime, or does it need an orthogonal stack (Interaction Models: A Scalable Approach to Human-AI Collaboration)?

Papers

Section titled “Papers”

2026-11

Section titled “2026-11”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-08

Section titled “2025-08”

2025-07

Section titled “2025-07”

2025-06

Section titled “2025-06”

2025-05

Section titled “2025-05”

2025-04

Section titled “2025-04”

2024-12

Section titled “2024-12”

Unknown date

Section titled “Unknown date”