Skip to content
Bud's Wiki

SGLang Diffusion — bringing SGLang's high-performance serving to diffusion models (LMSYS announcement)

TL;DR

Section titled “TL;DR”

LMSYS (the SGLang team) announces SGLang Diffusion — an extension of the SGLang inference engine to image and video diffusion models. The headline numbers are up to 5.9× faster inference, support for the major open-source families (Wan, Hunyuan, Qwen-Image, Qwen-Image-Edit, FLUX), and an OpenAI-compatible API plus CLI and Python entry points. Built jointly with FastVideo; thanks to NVIDIA and Voltage Park for compute. This is a product/infra announcement (no paper); the operational claim is that the same serving stack that runs production LLMs now covers DiT/MMDiT inference through one OpenAI-compatible front door.

Key claims

Section titled “Key claims”
  • Up to 5.9× faster inference vs unspecified baseline [tweet body].
  • Supports Wan, Hunyuan, Qwen-Image, Qwen-Image-Edit, and FLUX [tweet body].
  • Exposes an OpenAI-compatible HTTP API, a CLI, and a Python API [tweet body].
  • Built with FastVideo to “power the full diffusion ecosystem” [tweet body].
  • Compute sponsors: NVIDIA AI Dev and Voltage Park [tweet body].

Method

Section titled “Method”

Not described in the tweet. The tweet is the top of a thread (16 replies) with linked images that the fetcher did not return. What is observable from the announcement alone: SGLang’s existing primitives — paged KV cache, CUDA Graphs, OpenAI-compatible HTTP front end, continuous batching, and the streaming-session primitive recently upstreamed by Thinking Machines (PR #19171, see Interaction Models: A Scalable Approach to Human-AI Collaboration) — are being repurposed for diffusion serving, where the workload is not autoregressive token decode but a fixed-step DiT denoising loop. The “FastVideo” partnership ties this directly to the Hao AI Lab stack tracked in FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU (NVFP4 linear layers, SM100/SM103 attention kernels, end-to-end graph fusion across the prompt-encode → latent-prep → denoise → decode pipeline).

Results

Section titled “Results”

The only public number is “up to 5.9× faster inference” — baseline, hardware, model, resolution, and step count are not specified in the tweet. No FID/quality numbers. The four attached images presumably contain the benchmark table but were not retrievable via the fetcher.

Why it’s interesting

Section titled “Why it’s interesting”

The most-deployed open-source LLM serving stack is now claiming first-class support for diffusion models. Three reasons this matters for Luma:

  1. It collapses the LLM-vs-diffusion serving split into one runtime. The same OpenAI-compatible endpoint pattern, the same paged-cache / CUDA-graph / continuous-batching primitives, the same telemetry. The image-only sibling — Run FLUX.1-dev three times faster‘s Torch-compile + fused-QKV + channels-last + First-Block-Caching recipe for FLUX.1-dev — is a hand-rolled per-model serving stack; SGLang Diffusion is the same idea generalized into a serving framework with model coverage.
  2. It is built with FastVideo (FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU), which already pushes LTX-2.3 to sub-5-second 1080p T2AV on a single B200. The integration suggests the “system-level lever” of diffusion serving optimization is consolidating around a single open codebase rather than splitting between Modal-style per-deployment recipes and research-lab demos.
  3. It extends the trajectory tracked in LLM Inference Efficiency — the same project that absorbed Thinking Machines’ streaming-session PR (#19171) and that SSD/Saguaro (Speculative Speculative Decoding) targets as the open SD baseline — into diffusion. Whether the 5.9× number survives independent reproduction with disclosed baselines (cf. Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments‘s symmetric-baseline methodology critique) is the open question.

See also

Section titled “See also”