Skip to content
Bud's Wiki

Speeding up end-to-end inference with self-improving agents

TL;DR

Section titled “TL;DR”

Asari AI Labs describes a “co-inventor” agent system that autonomously optimizes the entire production LLM inference stack — kernels, schedulers, load balancers, and configuration — for two large open-weights models (DeepSeek v4 Pro and GLM 5.2) running on 8×B200 with vLLM v0.23. The agents report up to 16% throughput and interactivity gains across concurrency levels, with each concurrency level taking about a day of agent time. Two design choices differentiate it from prior kernel-level autoresearch work: (a) evaluations run against a full production-serving sandbox so speedups don’t vanish “in the seams” between components, and (b) correctness is enforced by statistical distribution-matching on token-level probabilities against the AllenAI C4 corpus, not just benchmark-accuracy parity. The agents also exhibit cross-model insight transfer — a distributed-deadlock lesson learned on DeepSeek v4 Pro saved 44 minutes when the same class of situation reappeared on GLM 5.2.

Key claims

Section titled “Key claims”
  • Optimizing the full inference stack in a single agent loop (rather than component-by-component) is necessary because system-level speedups depend on how the scheduler batches, how the load balancer routes, and how configuration flags propagate to kernels several layers down — a fast isolated kernel can vanish end-to-end (Amdahl’s law) [§What makes optimizing inference systems hard?].
  • On DeepSeek v4 Pro and GLM 5.2 (both served on 8×B200 with vLLM v0.23), agent-optimized stacks improve throughput and interactivity by up to 16% across multiple max-concurrency levels; each concurrency level takes about a day of agent time [§Introduction].
  • As of vLLM v0.26, most of the changes the agents produced have no upstream equivalent, spanning kernels, schedulers, load-balancers, and configuration settings — with M=1 skinny GEMM kernels, glue-code paths, and launch overhead specifically called out as the sources of gain [§Introduction, §Optimization examples].
  • Correctness is enforced via statistical consistency tests on token-level probability distributions between the optimized model and the stock reference, evaluated on a multilingual corpus of prompts from AllenAI C4 — rather than functional tests or benchmark scores like GSM8K/GPQA, which the post argues can pass while model behavior drifts [§Verify].
  • The agents exhibit cross-model insight transfer: a lesson mined from avoiding distributed deadlocks while optimizing DeepSeek v4 Pro generalized to a similar situation optimizing GLM 5.2 and saved ~44 minutes of unnecessary wait time [§Introduction, §Self-improve].
  • The invention loop is discover+build → evaluate → verify → self-improve, run by many agents in parallel; a hierarchical decomposition of end-to-end plus micro benchmarks is used to keep the feedback loop short given multi-hour full evaluations [§The invention loop, §Evaluate].
  • Noise mitigation is explicit: measurement noise from batching shifts, GPU power throttling, and floating-point variation is treated as a first-class problem, with statistical analysis and “anti-cheating checks” applied so a measured gain holds in production [§Evaluate].

Method

Section titled “Method”

The system runs a rigorous discover+build → evaluate → verify → self-improve loop with many agents in parallel. During discover and build, agents propose optimization strategies from a mix of dynamic profiling, static analysis, and learned experiences retrieved from prior runs, prioritizing targets by weighing potential speedup against numerical-accuracy risk. During evaluate, agents execute customized vLLM serving stacks inside a sandbox that replicates the production environment, dispatching to their own optimized kernels and code in place of stock implementations; a hierarchical mix of end-to-end and micro benchmarks keeps the iteration cheap, and statistical checks are layered on to filter out speedups that are artifacts of noise or benchmark gaming. During verify, a stringent distribution-matching correctness check runs statistical consistency tests on token-level probability distributions against a diverse multilingual C4 prompt set. During self-improve, insights are distilled and reused across models — the post lists four transfer categories (antipatterns, kernel exploration, workflow routing, evaluation-harness wiring), with the last two demonstrated as DeepSeek v4 Pro → GLM 5.2 transfers.

Results

Section titled “Results”

Blue-vs-grey throughput and interactivity plots (chart embedded but not statically rendered in the post) claim “consistent improvements across the board” and up to 16% headline gain on both metrics across multiple max-concurrency levels on 8×B200 running vLLM v0.23. Each concurrency level takes ~1 day of agent time. A concrete self-improvement datapoint: the distributed-deadlock lesson transferred from DeepSeek v4 Pro saved ~44 minutes on the GLM 5.2 optimization run. As of vLLM v0.26, most of the changes produced by the agents have no upstream equivalent — with M=1 skinny GEMM kernels, glue-code paths, and launch-overhead reductions specifically identified as agent-native optimization territory. The post shows an example agent run on GLM 5.2 at concurrency=1 with agent-produced code diffs (interactive timeline/diff widgets in the blog, not statically rendered).

Why it’s interesting

Section titled “Why it’s interesting”

This is the first filed instance in the wiki of AI-for-AI-research applied to the production inference stack rather than architecture discovery, paper writing, or training-side optimization — the object being autoresearched is a live vLLM deployment. It complements Sol Video Inference Engine: Agent-Native Full-Stack Acceleration Framework for Efficient Video Generation, which is the closest analogue (agents autoresearching the video-DiT inference stack with five acceleration levers and 2.27–2.77× E2E speedup on B200), by extending the same “agent-driven deployment-stack autoresearch” pattern from video generation to LLM serving. It also directly contrasts with Speculative Decoding: Performance or Illusion?‘s systematic manual study of speculative-decoding variants inside vLLM: Asari’s agents produce niche kernel and glue-code optimizations (M=1 skinny GEMM, launch-overhead reduction) rather than SD-variant selection, but face the same production reality that vLLM v0.23+ leaves headroom mostly in the seams between components. The distribution-matching correctness gate is a stricter contract than the training-parity distribution-matching used elsewhere on the wiki (e.g., On-Policy Distillation) and answers, for one deployment context, the LLM Inference Efficiency open question about how reproducibility should be defined when SD produces non-bitwise-identical greedy outputs.

See also

Section titled “See also”