Skip to content
Bud's Wiki

Diffusion serving optimization

TL;DR

Section titled “TL;DR”

Systems-level recipes for serving large diffusion / flow models (DiTs, MMDiTs, video DiTs) at low latency, distinct from algorithmic step reduction (distillation, few-step, AR rewiring). The current frontier combines five levers: (1) Torch compile with max-autotune + fused QKV projections + channels-last memory layout for ~1.5× free speedup; (2) approximate step caching (First Block Caching / TEACache-family) that skips diffusion steps whose post-first-block residual is unchanged; (3) low-precision execution — NVFP4 / FP4 linears on Blackwell-class GPUs exploiting native FP4 tensor cores; (4) attention kernels rewritten for SM100/SM103 to handle 3D spatiotemporal attention at full-HD token counts; (5) end-to-end graph fusion over the entire pipeline (prompt-encode → latent-prep → denoise → decode), plus optimized CPU-side I/O (ffmpeg). Cold-start regression from max-autotune is clawed back via persisted Inductor/Triton caches plus memory snapshots. A consolidation trend is visible: LMSYS now ships SGLang Diffusion built with FastVideo, suggesting the framework-level diffusion serving stack is converging on one open codebase rather than splitting into per-team recipes.

Key claims

Section titled “Key claims”
  • Standard Torch optimizations alone — torch.compile(mode="max-autotune-no-cudagraphs") + fuse_qkv_projections() + channels_last memory format — deliver ~1.5× speedup on FLUX.1-dev at 1024², “driven mostly by the Torch compiler” (Run FLUX.1-dev three times faster §“Apply standard optimizations”).
  • Fusing Q/K/V into one matmul exposes more parallelism and removes the compiler’s alias-analysis burden over three reads of the same input tensor (Run FLUX.1-dev three times faster §“Expose more parallelism to the GPU with fused QKV”).
  • First Block Caching (TEACache-family) gives an additional ~2× by skipping diffusion steps whose first-block residual delta is below threshold; threshold 0.12 on FLUX.1-dev gives “slight changes in results” with “smoother quality/speed tradeoff than quantization” (Run FLUX.1-dev three times faster §“Apply approximate caching”).
  • max-autotune cold start is tens of minutes; persisting TORCHINDUCTOR_FX_GRAPH_CACHE, CUDA_CACHE_PATH, TORCHINDUCTOR_CACHE_DIR, TRITON_CACHE_DIR on a shared volume plus Modal memory snapshots cuts cold-start ~30× (Run FLUX.1-dev three times faster §“Cut cold start latency by 30x”).
  • 1080p video with audio in under 5 seconds end-to-end on a single B200 is achievable for an unmodified LTX-2.3 — system-level only, no distillation (FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU §“FastVideo’s New Real-Time Inference Stack”).
  • NVFP4-quantized DiT linear layers exploit Blackwell’s native FP4 tensor cores while preserving output quality (qualitative claim); framed as essential to fully utilizing B200/B300 throughput (FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU §“Aggressive low-precision execution with NVFP4”).
  • Attention kernels specifically tuned for SM100/SM103 are required to handle 3D spatiotemporal attention at 1080p latent token counts; off-the-shelf attention kernels are not sufficient at this regime (FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU §“Fast attention for video generation”).
  • End-to-end kernel and graph fusion across all four stages — prompt encoding, latent preparation, denoising, decoding — is necessary because real latency is dominated by everything around the denoise loop, not the loop alone (FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU §“End-to-End graph and kernel optimization”).
  • CPU-side I/O (IPC + ffmpeg) is a non-trivial latency contributor at video resolution and requires a CPU-specific optimized build (FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU §“System-Level efficiency beyond the model”).
  • Single-GPU 1080p eliminates the need for sequence parallelism or multi-GPU sharding strategies that are otherwise treated as standard for fast video inference; serving scales by replicating single-GPU containers behind a Rust load balancer (FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU §“Outlook and Conclusion”, §“Serving Optimizations”).
  • The framework-level entry point now exists: SGLang Diffusion brings SGLang’s LLM serving primitives (paged cache, CUDA graphs, continuous batching, OpenAI-compatible API) to Wan / Hunyuan / Qwen-Image / Qwen-Image-Edit / FLUX, claiming up to 5.9× faster inference; built with FastVideo (SGLang Diffusion — bringing SGLang's high-performance serving to diffusion models (LMSYS announcement)).
  • The Apple Silicon / local-Mac path is now covered by an open-source single-CLI tool (MFLUX) that ports Z-Image, FLUX.2, FIBO, SeedVR2, Qwen-Image, and Depth Pro to MLX with quantization, multi-LoRA, in-context and multi-image editing, ControlNet, inpainting, depth conditioning, and upscaling all behind one uv tool install (MFLUX — MLX native implementations of state-of-the-art generative image models README §Models, §Features).
  • A full T2V pipeline (text encoder + first-frame image generator + denoiser + decoder + super-res) can run end-to-end on a Qualcomm Hexagon NPU in 6.7s for 49 frames at 640×1024, with 3.5 GB peak RAM and 4.945B total parameters — by replacing the FLAN-T5-XXL text encoder with a 35×-compressed DistilT5 + ContextAdapter, swapping the codec-latent-VAE decoder for a mobile-friendly one while keeping the original encoder and latent space, pruning >25% of MMDiT blocks with two-stage distillation recovery, and step-distilling via Pyramidal-DMD (Neodragon: Mobile Video Generation using Diffusion Transformer Abstract, §3).
  • For mobile NPUs, the limiting bottleneck of a “small” 226M codec-latent-VAE decoder is not parameter count but the conv3D feature-map buffer footprint — a 7×10×16 latent forward pass takes 3500 ms on Snapdragon X Elite before optimization; conv3D is “indispensable for causal video auto-encoding” so the fix is decoder swap rather than rewrite (Neodragon: Mobile Video Generation using Diffusion Transformer §3.2).
  • Single-GPU consumer-card recipe for video DiTs reaches 97-199× E2E speedup on Wan2.1/Wan2.2 by stacking SageAttention2++ low-bit attention + Sparse-Linear Attention (SLA, ~90% sparsity) + rCM 3-4-step distillation + W8A8 INT8 Linear quantization with hand-written Triton/CUDA norms. Wan2.2-I2V-A14B-720P: 4549s → 38s on a single RTX 5090; Wan2.1-T2V-14B-720P: 4767s → 24s (TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times Fig. 3, §1.1, §2.2). The waterfall attribution puts a number on this page’s structural argument: rCM does 33.3×, SageSLA does 3.45× on top, W8A8 + FusedNorm does 1.14× (Fig. 4).
  • The decode + upsample step is itself a serving target, not just a free cost paid after the denoise loop. PiD replaces the cascaded “VAE decode → external super-resolution” pipeline with a single DMD2-4-step pixel-diffusion decoder; on the 512²→2048² task this runs in ~210 ms on GB200 and under 1 s at 13 GB peak on a consumer RTX 5090, claiming ~6× faster than cascaded diffusion-based super-resolution and 5.9× faster than SeedVR2 specifically (211.2 ms vs 1237.5 ms) at better visual fidelity per project-page side-by-sides (PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion Abstract, project page Quantitative Results). Crucially, the sigma-aware adapter lets the upstream LDM exit its diffusion process early, coupling decode-side compute saving to denoise-side compute saving in one mechanism.

Recent contributions

Section titled “Recent contributions”
  • [2026-06-30] waterloo_intern (ali / Baseten) — Worklog: retrofitting sparsity for the world's fastest video generation kernel (54×): ali (@waterloo_intern, Baseten) tweet promoting a worklog that claims a 54× inference-time speedup on a frontier OSS video model via training-free sparsity retrofit, billed as “the world’s fastest video generation kernel”. Worklog body not retrievable at filing time; baseline, hardware, resolution, and quality metrics undisclosed in the visible tweet. Adds a vendor (Baseten) to the page’s serving-frontier roster and a training-free design point at headline-scale comparable to TurboDiffusion’s co-trained 97–199× — the standing symmetric-baseline critique applies until the worklog is independently retrievable.
  • [2026-06-23] ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?: ImageWAM treats an image-editing diffusion model as a “world model” whose intermediate KV cache is the deliverable — the model never decodes the target frame at inference. Pushes the serving-style “skip the decode, keep the latent” pattern up into a robotics training recipe, reporting ~1/6 FLOPs and ~1/4 latency vs video-based WAMs.
  • [2026-06-18] M*: A Modular, Extensible, Serving System for Multimodal Models: Stanford’s M* extends the framework-level diffusion-serving frontier into the composite-multimodal regime. Where FastVideo / SGLang Diffusion / TurboDiffusion / PiD optimize the denoise loop (and PiD the decode), M*‘s contribution is the graph algebra around it: a Loop primitive that wraps diffusion steps under the same continuous-batching + CUDA-graph machinery as AR decode, a first-class Parallel primitive that runs three-way CFG branches on three GPUs without per-model glue (vs vLLM-Omni’s bespoke CFGParallelMixin), and per-Walk placement so that BAGEL image-gen and image-understanding requests touch different components. Reports ≈1.3× lower latency on BAGEL text→image and up to ≈2.6× on BAGEL image editing vs vLLM-Omni’s default two-stage pipeline (3×H100, CFG-parallel, B=1). The headline vs vLLM-Omni’s best-tuned single-stage config is smaller (≈1.2× on editing) but doesn’t require vLLM-Omni’s whole-model tradeoff of collapsing LLM + ViT + VAE + DiT into the diffusion engine and giving up continuous batching + paged-attention on the text path. New design point on this page: framework-level serving where a single runtime can host the FastVideo recipes as one of its node engines.
  • [2026-06-09] TileRT: Tile-Based Runtime for Ultra-Low-Latency LLM Inference: TileRT — tile-level persistent-kernel runtime for latency-optimized LLM serving on 8×B200 with FP8/FP4 + MTP. Filed here as a precedent for the same persistent-kernel + tile-pipeline design pattern moving into diffusion serving (e.g. SGLang Diffusion, TurboDiffusion); the underlying compiler stack (TileLang/TileScale) is shared across LLM and diffusion workloads in this lineage.
  • [2026-05-26] PrismML — 1-bit and Ternary Bonsai Image 4B — local-hardware diffusion image generation: PrismML announces Bonsai Image 4B in 1-bit and ternary weight variants, explicitly targeted at “high-quality diffusion inference on local hardware: from laptops to phones” — a new lever (end-to-end weight quantization) for the cost-of-diffusion-inference problem. No benchmarks or technical details disclosed in the announcement tweet.
  • [2026-05-25] PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion (PiD, NVIDIA): adds a new design axis to the page — optimize the decode + upsample step of the pipeline rather than (or in addition to) the denoise loop. PiD distills a pixel-diffusion decoder via DMD2 to 4 NFE; that one model replaces (a) the deterministic VAE decoder and (b) the external super-resolution stage of a standard LDM pipeline. 512²→2048² in ~210 ms on GB200, sub-1 s at 13 GB peak on a consumer RTX 5090, with the sigma-aware adapter additionally letting the upstream LDM terminate early so the decode-side and denoise-side savings compound. Works for VAE and SigLIP/DINOv2 (RAE-style) latents, so the same decoder slots into both encoder families. Apache 2.0, weights released. Unlike the other entries on this page, the speedup comes from collapsing pipeline stages rather than from kernel/quantization/cache tricks on a single stage.
  • [2026-05-25] TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times: Tsinghua/Shengshu/UC-Berkeley stack the maximal acceleration recipe on Wan2.1/Wan2.2 video diffusion for a single RTX 5090 consumer GPU (not B200, not H100). Four orthogonal levers merged at training time: (a) full attention → SLA (Sparse-Linear Attention, Top-K 0.1, ≈90% sparsity), fine-tuned in place; (b) 100-step teacher → 3-4-step rCM student, with rCM and SLA finetunes parameter-merged so attention acceleration carries through the distilled weights; (c) inference-time W8A8 INT8 quantization of Linear layers at 128×128 block granularity using INT8 Tensor Cores; (d) hand-written Triton/CUDA replacements for LayerNorm/RMSNorm. Attention is served by SageSLA — SLA composed onto SageAttention2++. Reports 97× / 120× / 170× / 199× E2E speedup on Wan2.1-T2V-1.3B-480P / Wan2.2-I2V-A14B-720P / Wan2.1-T2V-14B-480P / Wan2.1-T2V-14B-720P respectively, beating FastVideo at default settings by 3-4× on the three models FastVideo supports. The Fig. 4 waterfall on Wan2.1-T2V-14B-720P puts numbers on the cluster’s standing intuition: rCM step distillation buys 33.3×, SageSLA buys another 3.45×, W8A8 + FusedNorm buys 1.14×, and the rest is CPU offload recovering an OOM baseline. Limitation: quality is qualitative (side-by-side video grids only, no VBench, no human preference rate, no per-component quality ablation) — the standing recommendation of Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments‘s symmetric-baseline critique applies in full.
  • [2026-05-25] Neodragon: Mobile Video Generation using Diffusion Transformer: Qualcomm AI Research’s Neodragon is the first wiki entry that retargets a full text-to-video pipeline for a mobile NPU (Snapdragon 8 Gen 4 / X Elite Hexagon NPU), as opposed to optimizing a server-class GPU. The recipe is four stacked compressions on Pyramidal-Flow: (a) Text-Encoder Distillation — replace FLAN-T5-XXL (4.726B) with DistilT5 (0.13B) + a 4-layer MLP ContextAdapter (0.13B) trained MSE+cosine on 1.4M text-only prompts; (b) Asymmetric Decoder Distillation — swap the original codec-latent-VAE decoder for a mobile-friendly alternative while keeping the encoder and latent space intact (conv3D feature-map buffers, not parameter count, were the real mobile bottleneck); (c) MMDiT Block Pruning — 2.009B → 1.499B (>25%) with two-stage recovery distillation; (d) Pyramidal-DMD step distillation paired with an SSD-1B first-frame generator and QuickSRNet 2× super-res. End-to-end: 4.945B parameters, 3.5 GB peak RAM, 6.7s latency on a phone NPU, VBench Total 81.61 — competitive with cloud Pyramidal-Flow at 720p (81.72). The base-model choice is motivated by Pyramidal-Flow’s spatio-temporally pyramidal causal attention giving an asymptotic 32× attention-compute saving vs. fully bidirectional Wan/CogVideoX/HunyuanVideo at the same latent size — i.e. AR causal attention is reframed here as a mobile-deployment lever rather than just a long-horizon-rollout one. Adobe Premiere Pro plugin demo on Snapdragon X Elite laptop runs the entire pipeline on-device with no cloud calls.
  • [2026-05-25] MFLUX — MLX native implementations of state-of-the-art generative image models: Filip Strand’s MFLUX ships a from-scratch MLX port of seven SOTA generative image model families (Z-Image, FLUX.2, FIBO, SeedVR2, Qwen-Image, Apple Depth Pro, legacy FLUX.1) behind a single uv tool install CLI. The v0.11.1 release that triggered this filing adds Qwen Image Edit with multi-image support — the local Apple-Silicon counterpart to the cloud-side Qwen-Image-Edit-2509 multi-reference workflow. Feature surface includes quantization, multi-LoRA with scales, LoRA finetuning, in-context editing, virtual try-on, ControlNet (Canny), fill/inpainting, depth conditioning, Redux, and SeedVR2/FLUX-ControlNet upscaling. No quantitative serving benchmarks; the artifact is a working tool, not a paper. Significance: closes the local-Mac side of the serving-optimization frontier that the other three contributions cover from the H100/B200 side.
  • [2026-05-24] SGLang Diffusion — bringing SGLang's high-performance serving to diffusion models (LMSYS announcement): LMSYS announces SGLang Diffusion — an extension of the SGLang inference engine to image and video diffusion models, supporting Wan, Hunyuan, Qwen-Image, Qwen-Image-Edit, and FLUX. Up to 5.9× faster inference (baseline, hardware, and step count not disclosed in the announcement tweet). Critically, it is “built with FastVideo”, which is the systems stack from FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU. Exposes an OpenAI-compatible API plus CLI and Python. Significance: the per-deployment recipe pattern of Run FLUX.1-dev three times faster and the per-lab demo pattern of FastVideo are consolidating into a framework-level serving runtime, served from the same engine that runs production LLMs.
  • [2026-05-23] FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU: Hao AI Lab pushes the video-with-audio frontier of system-level serving to ~4.55s end-to-end for 5s × 1088×1920 × 24FPS TI2AV on a single B200 (3.9× reported over next-fastest 1080p option). Four-layer stack: SM100/SM103-tuned spatiotemporal attention, NVFP4 linears, end-to-end graph fusion across encode/prep/denoise/decode, and CPU-side ffmpeg optimization for the I/O envelope. Algorithmically untouches LTX-2.3 — the speedup is purely systems. NVIDIA Dynamo added FastVideo as a supported diffusion backend.
  • [2026-05-22] Run FLUX.1-dev three times faster: Modal walks through getting FLUX.1-dev under 3 seconds per 1024² image (3× over diffusers baseline). Two layers: ~1.5× from compile + fused QKV + channels-last, then another ~2× from First Block Caching at threshold 0.12. Cold-start regression from max-autotune clawed back ~30× via Inductor/Triton cache persistence + memory snapshots. The most thorough public account of the per-deployment image-DiT serving recipe.

Open questions

Section titled “Open questions”
  • The system-level (FastVideo / SGLang Diffusion) and algorithmic (distillation, few-step, AR rewiring — cf. Helios: Real Real-Time Long Video Generation Model) levers are stackable in principle but no filed work combines them. What is the compounded latency on a distilled LTX or distilled FLUX served through SGLang Diffusion? Neodragon’s stack is this composition end-to-end (distillation + system retargeting), but on a different hardware class. TurboDiffusion (TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times) is a second concrete instance of the composition — algorithmic (rCM + SLA) merged into one weight checkpoint and served with kernel-level (SageSLA, W8A8) acceleration. PiD (PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion) is a third — distillation (DMD2) applied to the decode stage rather than the denoise loop, with the decode-side adapter enabling early LDM exit. The remaining frontier composition would be PiD’s decoder stacked behind a distilled generator, all served through SGLang Diffusion.
  • The headline “up to 5.9×” in the SGLang Diffusion announcement is undisclosed-baseline. The same critique applies to TurboDiffusion’s 97-199× headlines: the comparison is to the unaccelerated Wan reference implementation, not to a strong consumer-GPU serving baseline, and quality is qualitative. The page’s standing recommendation (cf. Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments‘s symmetric-baseline critique) is to wait for an independent reproduction with disclosed hardware, model, resolution, step count, and reference stack — and quantitative quality metrics — before treating any of these numbers as decisive. PiD’s “5.9× faster than SeedVR2” is a single-baseline disclosed comparison on the 512²→2048² decode task; better than aggregate “Nx faster” but still a single-baseline number with project-page-only quality side-by-sides.
  • First Block Caching’s quality–speed tradeoff is measured qualitatively (animations, “images looked better”). At what threshold does it start to materially distort FID / human-eval / video-temporal-consistency, and is the curve smooth as Modal claims or does it cliff somewhere? No filed paper quantifies this against quantization or step-reduction.
  • The video-DiT analog of FBC has higher headroom than the image case in principle, because adjacent video timesteps are more correlated than adjacent image timesteps. No filed paper measures the video upper bound — Hao AI Lab’s FastVideo demo specifically avoids step-caching to preserve quality at the audio synchronization boundary.
  • Approximate caching (FBC), NVFP4 linears, and attention-kernel SM100 tuning all touch different fault lines in output quality. There is no published study isolating each contribution’s quality impact for video DiTs — only the aggregate “preserved” claim. TurboDiffusion’s Fig. 4 ablation isolates latency contributions per lever (rCM 33.3×, SageSLA 3.45×, W8A8 1.14×) but does not isolate quality contributions, leaving the per-component quality story open.
  • How does the SGLang Diffusion / FastVideo runtime compare on serving primitives — continuous batching, paged-cache analog for the denoising loop, request preemption — against bespoke per-model serving (Modal’s recipe, Replicate, fal)? The framework-level pitch only pays off if multi-tenant batched diffusion serving recovers GPU utilization the bespoke recipes can’t.
  • What does the Apple-Silicon / unified-memory side of the cost-of-serving curve look like compared to the NVIDIA datacenter side? MFLUX is the first concrete entry point but there are no head-to-head latency numbers — FLUX.2 [klein] at 1024² on an M-series Mac vs a B200 server has no published comparison.
  • Neodragon’s mobile-NPU regime opens a new question: which of the H100/B200-side techniques transfer to Hexagon-class NPUs? NVFP4 has no direct mobile-NPU analog (Hexagon’s quantization story is INT8/INT16 plus selective FP16); First Block Caching is in principle hardware-agnostic but no filed paper combines it with mobile deployment; full-pipeline graph fusion is the one lever that should transfer cleanly. TurboDiffusion’s W8A8 INT8 design point is closer to the mobile-NPU precision regime than NVFP4 is — does the recipe partially transfer to Hexagon?
  • What is the decode-side serving frontier for video LDMs? PiD is image-only. The video analog would be a single pixel-diffusion model that decodes a Wan/CogVideoX/HunyuanVideo latent and upsamples temporally + spatially in one model. None of the filed papers do this — Vivid-VR / FlashVSR / DUO-VSR are bolted on after the VAE decoder, not in place of it. Whether the early-LDM-exit trick survives the temporal axis is open.

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-09

Section titled “2025-09”

2025-05

Section titled “2025-05”

Unknown date

Section titled “Unknown date”