Skip to content
Bud's Wiki

Diffusion serving optimization

TL;DR

Section titled “TL;DR”

Systems-level recipes for serving large diffusion / flow models (DiTs, MMDiTs, video DiTs) at low latency, distinct from algorithmic step reduction (distillation, few-step, AR rewiring). The current frontier combines five levers: (1) Torch compile with max-autotune + fused QKV projections + channels-last memory layout for ~1.5× free speedup; (2) approximate step caching (First Block Caching / TEACache-family) that skips diffusion steps whose post-first-block residual is unchanged; (3) low-precision execution — NVFP4 / FP4 linears on Blackwell-class GPUs exploiting native FP4 tensor cores; (4) attention kernels rewritten for SM100/SM103 to handle 3D spatiotemporal attention at full-HD token counts; (5) end-to-end graph fusion over the entire pipeline (prompt-encode → latent-prep → denoise → decode), plus optimized CPU-side I/O (ffmpeg). Cold-start regression from max-autotune is clawed back via persisted Inductor/Triton caches plus memory snapshots. A consolidation trend is visible: LMSYS now ships SGLang Diffusion built with FastVideo, suggesting the framework-level diffusion serving stack is converging on one open codebase rather than splitting into per-team recipes.

Key claims

Section titled “Key claims”
  • Standard Torch optimizations alone — torch.compile(mode="max-autotune-no-cudagraphs") + fuse_qkv_projections() + channels_last memory format — deliver ~1.5× speedup on FLUX.1-dev at 1024², “driven mostly by the Torch compiler” (Run FLUX.1-dev three times faster §“Apply standard optimizations”).
  • Fusing Q/K/V into one matmul exposes more parallelism and removes the compiler’s alias-analysis burden over three reads of the same input tensor (Run FLUX.1-dev three times faster §“Expose more parallelism to the GPU with fused QKV”).
  • First Block Caching (TEACache-family) gives an additional ~2× by skipping diffusion steps whose first-block residual delta is below threshold; threshold 0.12 on FLUX.1-dev gives “slight changes in results” with “smoother quality/speed tradeoff than quantization” (Run FLUX.1-dev three times faster §“Apply approximate caching”).
  • max-autotune cold start is tens of minutes; persisting TORCHINDUCTOR_FX_GRAPH_CACHE, CUDA_CACHE_PATH, TORCHINDUCTOR_CACHE_DIR, TRITON_CACHE_DIR on a shared volume plus Modal memory snapshots cuts cold-start ~30× (Run FLUX.1-dev three times faster §“Cut cold start latency by 30x”).
  • 1080p video with audio in under 5 seconds end-to-end on a single B200 is achievable for an unmodified LTX-2.3 — system-level only, no distillation (FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU §“FastVideo’s New Real-Time Inference Stack”).
  • NVFP4-quantized DiT linear layers exploit Blackwell’s native FP4 tensor cores while preserving output quality (qualitative claim); framed as essential to fully utilizing B200/B300 throughput (FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU §“Aggressive low-precision execution with NVFP4”).
  • Attention kernels specifically tuned for SM100/SM103 are required to handle 3D spatiotemporal attention at 1080p latent token counts; off-the-shelf attention kernels are not sufficient at this regime (FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU §“Fast attention for video generation”).
  • End-to-end kernel and graph fusion across all four stages — prompt encoding, latent preparation, denoising, decoding — is necessary because real latency is dominated by everything around the denoise loop, not the loop alone (FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU §“End-to-End graph and kernel optimization”).
  • CPU-side I/O (IPC + ffmpeg) is a non-trivial latency contributor at video resolution and requires a CPU-specific optimized build (FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU §“System-Level efficiency beyond the model”).
  • Single-GPU 1080p eliminates the need for sequence parallelism or multi-GPU sharding strategies that are otherwise treated as standard for fast video inference; serving scales by replicating single-GPU containers behind a Rust load balancer (FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU §“Outlook and Conclusion”, §“Serving Optimizations”).
  • The framework-level entry point now exists: SGLang Diffusion brings SGLang’s LLM serving primitives (paged cache, CUDA graphs, continuous batching, OpenAI-compatible API) to Wan / Hunyuan / Qwen-Image / Qwen-Image-Edit / FLUX, claiming up to 5.9× faster inference; built with FastVideo (SGLang Diffusion — bringing SGLang's high-performance serving to diffusion models (LMSYS announcement)).
  • The Apple Silicon / local-Mac path is now covered by an open-source single-CLI tool (MFLUX) that ports Z-Image, FLUX.2, FIBO, SeedVR2, Qwen-Image, and Depth Pro to MLX with quantization, multi-LoRA, in-context and multi-image editing, ControlNet, inpainting, depth conditioning, and upscaling all behind one uv tool install (MFLUX — MLX native implementations of state-of-the-art generative image models README §Models, §Features).
  • A full T2V pipeline (text encoder + first-frame image generator + denoiser + decoder + super-res) can run end-to-end on a Qualcomm Hexagon NPU in 6.7s for 49 frames at 640×1024, with 3.5 GB peak RAM and 4.945B total parameters — by replacing the FLAN-T5-XXL text encoder with a 35×-compressed DistilT5 + ContextAdapter, swapping the codec-latent-VAE decoder for a mobile-friendly one while keeping the original encoder and latent space, pruning >25% of MMDiT blocks with two-stage distillation recovery, and step-distilling via Pyramidal-DMD (Neodragon: Mobile Video Generation using Diffusion Transformer Abstract, §3).
  • For mobile NPUs, the limiting bottleneck of a “small” 226M codec-latent-VAE decoder is not parameter count but the conv3D feature-map buffer footprint — a 7×10×16 latent forward pass takes 3500 ms on Snapdragon X Elite before optimization; conv3D is “indispensable for causal video auto-encoding” so the fix is decoder swap rather than rewrite (Neodragon: Mobile Video Generation using Diffusion Transformer §3.2).
  • Single-GPU consumer-card recipe for video DiTs reaches 97-199× E2E speedup on Wan2.1/Wan2.2 by stacking SageAttention2++ low-bit attention + Sparse-Linear Attention (SLA, ~90% sparsity) + rCM 3-4-step distillation + W8A8 INT8 Linear quantization with hand-written Triton/CUDA norms. Wan2.2-I2V-A14B-720P: 4549s → 38s on a single RTX 5090; Wan2.1-T2V-14B-720P: 4767s → 24s (TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times Fig. 3, §1.1, §2.2). The waterfall attribution puts a number on this page’s structural argument: rCM does 33.3×, SageSLA does 3.45× on top, W8A8 + FusedNorm does 1.14× (Fig. 4).
  • The decode + upsample step is itself a serving target, not just a free cost paid after the denoise loop. PiD replaces the cascaded “VAE decode → external super-resolution” pipeline with a single DMD2-4-step pixel-diffusion decoder; on the 512²→2048² task this runs in ~210 ms on GB200 and under 1 s at 13 GB peak on a consumer RTX 5090, claiming ~6× faster than cascaded diffusion-based super-resolution and 5.9× faster than SeedVR2 specifically (211.2 ms vs 1237.5 ms) at better visual fidelity per project-page side-by-sides (PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion Abstract, project page Quantitative Results). Crucially, the sigma-aware adapter lets the upstream LDM exit its diffusion process early, coupling decode-side compute saving to denoise-side compute saving in one mechanism.

Recent contributions

Section titled “Recent contributions”
  • [2026-11-05] DreamFree: Future Representation Distillation for Imagination-free World Action Models: DreamFree attacks WAM serving latency at the interface level rather than the per-step level: rather than distill denoising steps or cache cross-chunk residuals (C³ache: Accelerating World Action Models with Cross Inference Chunk Cache C³ache), it moves the future-prediction work entirely to training time via representation distillation from a future-aware teacher, so at inference the WAM backbone never runs its video rollout. 1,632 ms → 91 ms per action chunk on H200 at 14B vs World Action Models are Zero-shot Policies (DreamZero) DreamZero’s full-rollout baseline (17.8× speedup), with real-world DROID success rising 56.0 → 64.0%. Also drops CFG at inference and fits on a single A6000 48 GB. Sibling to RealtimeWAM: One-Step Asynchronous World Action Models RealtimeWAM (~25× H100 speedup via cross-expert pipelining + action step distillation) on the “WAM latency is a serving problem” axis, but attacks it from the input interface side — no future-video is generated at all — rather than parallelizing or distilling the rollout.
  • [2026-10-30] Alex Ksendzovsky — Mischief 1 biocompute image-to-video model launch: Mischief 1 (BioComputingCo + Wonder Studios) launches as a closed-weights image-to-video model priced at $0.005/second on fal and OpenRouter — pitched as the cheapest I2V on both marketplaces and 5× faster than its unnamed base, with the compute substrate claimed to be cultured neurons on electrodes rather than silicon. No technical report, model card, or benchmarks disclosed; filed as a cost/latency market datapoint, not a methods contribution.
  • [2026-10-19] Zero-Shot to New Robots: Introducing Dream-1: Dream-1 (Dream Labs) ships a 14B-class WAM behind a public API at 3/hr/3/hr / 0.0013 per action (Dream-1) and 2/hr/2/hr / 0.0009 per action (Dream-1-flash) with each call returning 1.6s of action — Robocurve reports it 1.7× faster per call than GPT-6 Astra and 16× cheaper per task attempt than Gemini 4 Argon on RoboDojo-RC Tier 1. Puts concrete production economics on the “serve a video-diffusion WAM as a real-time System-1 robot policy” deployment pattern that research-side latency work (RealtimeWAM: One-Step Asynchronous World Action Models RealtimeWAM, C³ache: Accelerating World Action Models with Cross Inference Chunk Cache C³ache, World Action Models are Zero-shot Policies (DreamZero) DreamZero’s own 7 Hz closed-loop at 14B) has been circling.
  • [2026-10-08] Level-of-Token Diffusion: LoT Diffusion extends the ETC-style “spatial-token-reduction fine-tune produces an elastic checkpoint” pattern from Hilbert-cut content-derived layouts to arbitrary externally-specified multiresolution layouts (semantic masks, bounding boxes, texture variance, DoF, agentic plans). Speedup is determined by the layout’s token budget per generation rather than a fixed compression ratio, so one checkpoint serves every budget — same elastic-serving property as ETC but with the layout decoupled from the network’s own features. Headline per-budget numbers were not in the fetched abstract; project page is the next check.
  • [2026-10-05] Rho-1: Collapsing the Multimodal Stack: Rho-1 (Reka Labs) names three serving bottlenecks specific to omni backbones that interleave continuous diffusion rollout with discrete AR decoding: (i) heterogeneous memory layouts — high-throughput discrete KV caches must coexist in HBM alongside transient continuous latent states without fragmentation; (ii) shifting compute intensity — workload swings between compute-bound dense attention during visual synthesis and memory-bandwidth-bound AR decode during text generation within a single request; (iii) hard real-time latency — interactive steering leaves no room for pipeline bubbles, must ingest new instructions and update trajectories within fractions of a second. Reka pitches Reka Cloud as the production surface where these constraints get solved end-to-end, with the token-representation and kernel-scheduling choices of Rho-1 feeding back into the serving stack. No kernel-level or scheduler-level benchmark numbers disclosed.
  • [2026-10-02] Veda: Scalable Video Diffusion via Distilled Sparse Attention: Veda (Han et al., project page) adds a training-free sparse-attention design point for video DiTs that commits to a fixed, KL-distilled oracle-aligned tile mask (rather than Sol-Attn’s on-the-fly thresholding or SVG2’s k-means permutation) and offloads it to a ThunderKittens TMA/WGMMA sparse-gather kernel; 95% sparse Waver-T2V-12B 720P / 241 frames drops 19.4 min → 3.8 min (5.1× E2E), with per-layer latency reported against FlashAttention-3 across Waver and Wan2.1.
  • [2026-10-01] Improving Video Sparse Attention with Fine-grained Router and Sparse Rebasing: VSA2 (Hao AI Lab) is the direct successor to the VSA kernel underlying FastVideo/FastWan — pushes trainable sparse attention from inference-only retrofit into all training stages (pretraining, RL, inference). Two new levers: (1) a fine-grained router permitting per-query variable KV count (dynamic compute, breaking fixed-K/top-K assumption of SVG2 and Sol-Attn), and (2) a Hard-to-Easy curriculum where models trained at higher sparsity than inference generalize down and reportedly beat full attention on motion quality. Adds sparse rebasing: full-attention early-stage checkpoints from progressive low-to-high resolution pretraining can be rebased onto sparse attention mid-training. On 720p: 8.9× attention / 4.62× E2E vs FlashAttention-3, at half the attention compute of previous VSA with lower loss.
  • [2026-09-30] Beyond kernels: Letting agents optimize the whole AI job: FastVideo MiniMax H3 case study: agent identified and eliminated unnecessary CPU↔GPU weight transfers in the VAE decoding stage (5.87s of 8.87s decode time), yielding 2.40× E2E speedup (15.20s → 6.34s median, 4×B200). Upstreamed as FastVideo PR #1867. Complements Sol Video Inference Engine’s 5-lever autoresearch and Asari’s vLLM optimization by targeting an OSS video-diffusion serving path via a portable plugin rather than a lab-controlled stack.
  • [2026-09-30] Sol-H3: Recursive Self-Improvement for MiniMax-H3 Inference Acceleration on Sol-Engine across Cloud and Edge: Sol-H3 is the third Sol-Engine artifact and the first Sol targeting MiniMax H3 — combines an algorithmic cross-resolution two-stage scheduler (low-res global layout → high-res refinement, connected by a learned latent-to-latent map that eliminates the VAE decode/re-encode bridge) with a Recursive Self-Improvement loop over kernel fusions and memory layouts scored jointly on latency+numerical agreement. Reports up to 30× E2E speedup, 20% lower memory, 5s 1344×768 video with audio at 3.5× real time on 8×GB200, and the same clip fully memory-resident on a single DGX Spark in under a minute — the first filed diffusion-serving artifact whose single pipeline explicitly spans cloud (GB200) and edge (DGX Spark) datacenter/workstation classes. Complements the H3-family accelerators already tracked here (FastH3, VideoDeltaNet, TaoMate-H3, Jev-gated sparsity) with the first cross-resolution scheduler + learned latent bridge design point on this base.
  • [2026-09-23] FLUX 3 Action: A 7B World Action Model for Robot Control: FLUX 3 Action reports a serving stack for real-time robotics WAM inference: row-wise dynamic FP8 (+0.05 to -0.76 pp task success vs BF16, >1.4× speedup on local GPUs incl. RTX Pro / 5090), prompt-token KV caching across timesteps (prompt tokens don’t change in mode blocks), static-shape torch.compile with reduced-overhead mode, and step-invariant computation lifted out of blocks into shared prep. Combined with guidance and step distillation, drives end-to-end prediction latency from 246.42 ms BF16 4-step-guidance-on to 32.29 ms FP8 1-step-guidance-off (32 actions covering 2.13 s of motion at 15 Hz). In FP8, step-distilled F3A runs 1.34–2.28× faster than π0.5 BF16 on workstation/datacenter and 1.52–3.95× faster than Cosmos 3 Nano FP8 across consumer/workstation/datacenter GPU tiers — the WAM class hits VLA-class inference RTF via serving-side moves without sacrificing joint video+action prediction.
  • [2026-09-23] InstinctFlash — High-Performance Serving Runtime for Robotics Models: InstinctFlash (General Instinct, YC P26, AGPL-3.0) extends the diffusion-serving frontier into the robotics-policy regime by serving flow-matching / diffusion-transformer WAMs and video-world policies (LingBot-VA, Cosmos3 Edge/Nano DROID, DreamZero DROID) alongside VLAs under one Runtime API — the first filed release where the same runtime targets image-DiT-shaped video-world-model policies and per-chunk-flow VLA action heads through a shared six-layer optimization DAG (MODEL / GRAPH / CACHE / ATTENTION / KERNEL / HARDWARE). Jetson Thor P50 speedups vs PyTorch: 3.24× Cosmos3 Edge DROID and 2.13× Cosmos3 Nano DROID under NUMERIC precision with UniPC4 / CFG3, 1.98× DreamZero DROID under FP8 + 16 steps + BEHAVIORAL-tier dynamic step cache. The 33.78× LingBot-VA headline is the composition of runtime (5.36× FP8) and algorithmic (25V/50A → 2V/4A step distillation) levers — the empirical evidence this page has been asking for on whether algorithmic + systems levers compose without a quality collapse, deferred here to an instinctflash validate non-inferiority certificate rather than reported directly. DreamZero’s opt-in dynamic step cache is exposed as a first-class BEHAVIORAL-tier option, extending the C³ache: Accelerating World Action Models with Cross Inference Chunk Cache C³ache cross-chunk residual-cache pattern to a serving-runtime option surface.
  • [2026-09-20] Kamimoto — Jev-gated dynamic sparsity for MiniMax H3 video generation (41.7% wall-clock reduction on RTX 4070): Kamimoto (@sep_is_heim) demonstrates a genuinely new pole for H3 acceleration — using TypeSafe AI’s Jev as a per-layer runtime sparsity router over {1%, 3%, 5%, 10%} across all 49 attention layers of 4-step MiniMax H3. Reports 6:07 → 3:34 wall-clock on RTX 4070 (41.7% reduction), touching zero weights. First filed datapoint of “decision-model-as-runtime-router” orthogonal to distillation (FastVideo FastH3 V1 — 4-Step Sparse Distilled MiniMax H3 for 14x Blackwell Speedup), architecture retrofit (VideoDeltaNet on MiniMax H3 — Near-Lossless Hybrid Attention for Video Models), and LoRA distillation (TaoMate-H3 — Low-Latency Streaming LoRA Distillation of MiniMax H3 for Joint Audio-Video). Author explicitly concedes fixed 10% sparsity captures much of the win; marginal contribution over a hard-coded schedule remains unmeasured.
  • [2026-09-14] TaoMate-H3 — Low-Latency Streaming LoRA Distillation of MiniMax H3 for Joint Audio-Video: TaoMate-H3 pairs TP2 tensor parallelism with Ulysses4 sequence parallelism (validated on 8×H20-96GB) and explicitly optimizes for first-chunk latency rather than pure throughput: 27.66× first-final-chunk-latent vs 11.45× pure-DiT-time speedup on 480×864 / 10s. Clean KV-cache carryover between chunks is the key streaming primitive that hides most of the latency. Third serving recipe on the MiniMax H3 base after FastVideo’s VSA-H3 sparse-attention stack and VideoDeltaNet’s 5+3 uneven branch-parallel all-to-all, and the first one targeting the streaming regime on this base.
  • [2026-09-11] Elastic Token Compression for Pixel-Space Diffusion Transformers: ETC (Zamfir et al., Würzburg + AWS) adds a spatial-token-count-reduction lever to this page, distinct from the caching / low-precision / kernel / decode-side axes already tracked. Fine-tunes MiniT2I (258M pixel-space DiT) with a Region Token Interface — Hilbert-curve reordering of patches so any contiguous run is a connected 2D region, cuts placed where features change most, one region → one token. Randomized region-count fine-tune gives one elastic checkpoint serving every budget: matches dense-token quality at 2.0× inference speed, close at 2.6×. Frames the compression as reintroducing inside the diffusion model the content-adaptive compression pixel-space diffusion gave up by dropping the VAE — i.e. a serving win specific to VAE-free pipelines that ordinary latent-DiT serving already amortizes.
  • [2026-09-10] FLUX 3 Video Edit [fast] — BFL launches fast, low-cost instruction-based video editor with lip-sync dialogue swap: BFL launches FLUX 3 Video Edit [fast], positioned on the Pareto frontier of quality vs cost — first API-hosted instruction-based video editor to lead with the inference-cost axis; presumably a distilled few-step variant of FLUX 3 Video behind the API.
  • [2026-09-03] FastVideo FastH3 V1 — 4-Step Sparse Distilled MiniMax H3 for 14x Blackwell Speedup: FastH3 Preview v1 (Hao AI Lab + Nuva Lab + NVIDIA FastGen + NVIDIA Enterprise Products) extends the FastVideo serving stack from Wan-family to MiniMax H3’s 33B joint text-to-video-and-audio base. Recommended checkpoint requires FastVideo’s Video Sparse Attention (VSA-H3) kernel backend — dense attention is not a drop-in substitute. Reports 14× headline speedup vs Base H3 dense on 1× B200 and sub-realtime (15s video generated in <13s) on 8× B200 at 1344×768 / 24 FPS with audio. Distillation is 4-step, data-free (prompts only, no target videos), 1k+ B200 training hours. Ships full weights + LoRA + dense/synthetic ablations + FastVideo training + inference code. Distillation-first counterpart to VideoDeltaNet (VideoDeltaNet on MiniMax H3 — Near-Lossless Hybrid Attention for Video Models, filed same day) on the same H3 base — VideoDeltaNet claims better quality via architecture change.
  • [2026-09-03] VideoDeltaNet on MiniMax H3 — Near-Lossless Hybrid Attention for Video Models: VideoDeltaNet-H3 introduces branch-parallel sequence parallelism as a new serving pattern for hybrid diffusion transformers on 8×B200: an uneven 5+3 all-to-all places the softmax branch on 5 GPUs and the VDA linear branch on 3 GPUs (split chosen by per-branch latency profiling), then a reverse all-to-all returns to the original sequence-sharded layout. Reduces 1.62 s/NFE (standard Ulysses) → 1.405 s/NFE (13.3% beyond Ulysses). Combined with per-block architectural change (332.5→125.3 ms per layer via VDA + kernels + FP8) and 8-step DMD2 LoRA distillation from a community turbo LoRA, reaches 74.5× over dense H3 single-GPU and 10.7× over dense 8-GPU. Distinct from FastVideo / SGLang Diffusion / Sol-Engine’s framework-level frontier by exploiting the two-branch structure of hybrid attention; complements the same team’s FastH3 distillation-first retrofit (FastVideo FastH3 V1 — 4-Step Sparse Distilled MiniMax H3 for 14x Blackwell Speedup) filed together.
  • [2026-09-02] C³ache: Accelerating World Action Models with Cross Inference Chunk Cache: C³ache opens a new redundancy axis for this page — across inference chunks in a World Action Model, not within a single denoising trajectory. Caches the DiT residual (block output minus input embedding) at each denoising step and reuses it on subsequent chunks, letting the new observation still flow through via the uncached input embedding while skipping the entire DiT stack. Reports 2.21× wall-clock on Fast-WAM/LIBERO and 2.03× on RoboTwin 2.0 with negligible success drop, restricted to the first ~6 of 10 denoising steps. Extends the existing chunk-local caching canon (First Block Caching / TeaCache / Sol-Attn) into the robot-policy regime; concurrent X-Cache targets the same axis for few-step distilled driving world models.
  • [2026-07-28] Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification: Sol-Attn (NVIDIA + MIT HAN Lab, same team as Sol Video Inference Engine: Agent-Native Full-Stack Acceleration Framework for Efficient Video Generation) is the concrete sparse-attention kernel behind Sol-Engine’s “sparse attention” skill lever — a training-free method that folds routing and approximate correction into a single online-softmax pass. Instead of materializing a proxy-score map and then top-K/top-p selecting blocks (SVG2/SpargeAttention pattern), it thresholds block proxy scores on-the-fly and reuses unselected blocks’ scores as approximate corrections rather than dropping them. B200 numbers vs dense FA3: 2.02× on Wan 2.1-14B, 2.12× on HunyuanVideo-13B, up to 2.4× on LTX 2.3; composed inside Sol-Engine with fusion + caching: 3.48× and 5.08× respectively. Confirms the Sol-Engine story that sparse attention alone is ~2× and the framework composition is where the compounding shows up.
  • [2026-07-24] SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation: One of the most aggressive filed video-serving numbers: 5B pipeline at 13.06s for 720p/5s on one H100 (40 steps), reported as 120× faster than Wan 2.2-A14B on the same hardware. Compiled DiT forward pass is 3.2× a matched full-softmax baseline at 720p/60s (gap grows with duration); Sol-Engine full-stack layer (kernel fusion + caching + sparse attention) adds another 3.58× on top of the hybrid backbone.
  • [2026-07-23] Lucy 2.5 Real-Time AI Video Editing: Decart's 40ms Live Stream Revolution: Lucy 2.5 (Decart, community writeup) claims a DOS inference stack combining MXFP8/NVFP4 4-bit quantization, dynamic sparse attention that skips attention on unchanged frame regions (a temporal-coherence sparsity primitive distinct from Radial-Attention/SVG2-style spatial or semantic sparsity), and deep kernel fusion — reaching sub-40 ms/frame at 30 FPS 720p for real-time video editing (not just generation).
  • [2026-07-22] SANA — Efficient High-Resolution Image and Video Generation family (NVlabs/Sana, sol-engine branch): NVlabs/Sana sol-engine branch — the code companion to the Sol Video Inference Engine paper filed above, bundled inside the broader SANA multi-project repository (SANA / SANA-1.5 / SANA-Sprint / SANA-Video / SANA-WM / SANA-Streaming / Sol-RL). First filed instance of a diffusion-serving framework being distributed as a branch of the model-family repo itself rather than as a standalone serving stack.
  • [2026-07-22] Sol Video Inference Engine: Agent-Native Full-Stack Acceleration Framework for Efficient Video Generation: Sol Video Inference Engine (NVIDIA + MIT HAN Lab) reframes video-DiT acceleration as an agent-native optimization problem — five skill agents (cache / sparse attention / token pruning / quantization / kernel fusion) locally search per-lever, an integrator agent globally composes their candidates, and a human validator checks perceptual quality (since PSNR is not aligned). Reports 2.27× / 2.38× / 2.77× E2E speedup on B200 for Cosmos3-Super 64B / LTX-2.3 22B / SANA-Video 2B respectively vs SGLang, with the same framework generalizing across MoT / two-stage / linear-attention backbones. New design point on this page: the recipe search is the deliverable, not any specific per-deployment recipe. Weaker headline numbers than TurboDiffusion’s 97–199× but training-free rather than training-baked.
  • [2026-07-22] Mage-Flow: An Efficient Native-Resolution Foundation Model for Image Generation and Editing: Mage-Flow-Turbo reaches 0.59s for a 1024² image and 1.02s for a 1024² edit on a single A100 at 4B parameters. Two per-model levers stack cleanly with the framework-level serving frontier: (a) Mage-VAE removes the tokenizer bottleneck with 12–22× fewer MACs/pixel via one-step diffusion-style encode/decode with anchor-latent regularization, and (b) packed CFG evaluates conditional and unconditional branches in one forward pass instead of two. Neither has been composed with FastVideo / SGLang Diffusion / PiD yet.
  • [2026-06-30] waterloo_intern (ali / Baseten) — Worklog: retrofitting sparsity for the world's fastest video generation kernel (54×): ali (@waterloo_intern, Baseten) tweet promoting a worklog that claims a 54× inference-time speedup on a frontier OSS video model via training-free sparsity retrofit, billed as “the world’s fastest video generation kernel”. Worklog body not retrievable at filing time; baseline, hardware, resolution, and quality metrics undisclosed in the visible tweet. Adds a vendor (Baseten) to the page’s serving-frontier roster and a training-free design point at headline-scale comparable to TurboDiffusion’s co-trained 97–199× — the standing symmetric-baseline critique applies until the worklog is independently retrievable.
  • [2026-06-23] ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?: ImageWAM treats an image-editing diffusion model as a “world model” whose intermediate KV cache is the deliverable — the model never decodes the target frame at inference. Pushes the serving-style “skip the decode, keep the latent” pattern up into a robotics training recipe, reporting ~1/6 FLOPs and ~1/4 latency vs video-based WAMs.
  • [2026-06-18] M*: A Modular, Extensible, Serving System for Multimodal Models: Stanford’s M* extends the framework-level diffusion-serving frontier into the composite-multimodal regime. Where FastVideo / SGLang Diffusion / TurboDiffusion / PiD optimize the denoise loop (and PiD the decode), M*‘s contribution is the graph algebra around it: a Loop primitive that wraps diffusion steps under the same continuous-batching + CUDA-graph machinery as AR decode, a first-class Parallel primitive that runs three-way CFG branches on three GPUs without per-model glue (vs vLLM-Omni’s bespoke CFGParallelMixin), and per-Walk placement so that BAGEL image-gen and image-understanding requests touch different components. Reports ≈1.3× lower latency on BAGEL text→image and up to ≈2.6× on BAGEL image editing vs vLLM-Omni’s default two-stage pipeline (3×H100, CFG-parallel, B=1). The headline vs vLLM-Omni’s best-tuned single-stage config is smaller (≈1.2× on editing) but doesn’t require vLLM-Omni’s whole-model tradeoff of collapsing LLM + ViT + VAE + DiT into the diffusion engine and giving up continuous batching + paged-attention on the text path. New design point on this page: framework-level serving where a single runtime can host the FastVideo recipes as one of its node engines.
  • [2026-06-09] TileRT: Tile-Based Runtime for Ultra-Low-Latency LLM Inference: TileRT — tile-level persistent-kernel runtime for latency-optimized LLM serving on 8×B200 with FP8/FP4 + MTP. Filed here as a precedent for the same persistent-kernel + tile-pipeline design pattern moving into diffusion serving (e.g. SGLang Diffusion, TurboDiffusion); the underlying compiler stack (TileLang/TileScale) is shared across LLM and diffusion workloads in this lineage.
  • [2026-05-26] PrismML — 1-bit and Ternary Bonsai Image 4B — local-hardware diffusion image generation: PrismML announces Bonsai Image 4B in 1-bit and ternary weight variants, explicitly targeted at “high-quality diffusion inference on local hardware: from laptops to phones” — a new lever (end-to-end weight quantization) for the cost-of-diffusion-inference problem. No benchmarks or technical details disclosed in the announcement tweet.
  • [2026-05-25] PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion (PiD, NVIDIA): adds a new design axis to the page — optimize the decode + upsample step of the pipeline rather than (or in addition to) the denoise loop. PiD distills a pixel-diffusion decoder via DMD2 to 4 NFE; that one model replaces (a) the deterministic VAE decoder and (b) the external super-resolution stage of a standard LDM pipeline. 512²→2048² in ~210 ms on GB200, sub-1 s at 13 GB peak on a consumer RTX 5090, with the sigma-aware adapter additionally letting the upstream LDM terminate early so the decode-side and denoise-side savings compound. Works for VAE and SigLIP/DINOv2 (RAE-style) latents, so the same decoder slots into both encoder families. Apache 2.0, weights released. Unlike the other entries on this page, the speedup comes from collapsing pipeline stages rather than from kernel/quantization/cache tricks on a single stage.
  • [2026-05-25] TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times: Tsinghua/Shengshu/UC-Berkeley stack the maximal acceleration recipe on Wan2.1/Wan2.2 video diffusion for a single RTX 5090 consumer GPU (not B200, not H100). Four orthogonal levers merged at training time: (a) full attention → SLA (Sparse-Linear Attention, Top-K 0.1, ≈90% sparsity), fine-tuned in place; (b) 100-step teacher → 3-4-step rCM student, with rCM and SLA finetunes parameter-merged so attention acceleration carries through the distilled weights; (c) inference-time W8A8 INT8 quantization of Linear layers at 128×128 block granularity using INT8 Tensor Cores; (d) hand-written Triton/CUDA replacements for LayerNorm/RMSNorm. Attention is served by SageSLA — SLA composed onto SageAttention2++. Reports 97× / 120× / 170× / 199× E2E speedup on Wan2.1-T2V-1.3B-480P / Wan2.2-I2V-A14B-720P / Wan2.1-T2V-14B-480P / Wan2.1-T2V-14B-720P respectively, beating FastVideo at default settings by 3-4× on the three models FastVideo supports. The Fig. 4 waterfall on Wan2.1-T2V-14B-720P puts numbers on the cluster’s standing intuition: rCM step distillation buys 33.3×, SageSLA buys another 3.45×, W8A8 + FusedNorm buys 1.14×, and the rest is CPU offload recovering an OOM baseline. Limitation: quality is qualitative (side-by-side video grids only, no VBench, no human preference rate, no per-component quality ablation) — the standing recommendation of Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments‘s symmetric-baseline critique applies in full.
  • [2026-05-25] Neodragon: Mobile Video Generation using Diffusion Transformer: Qualcomm AI Research’s Neodragon is the first wiki entry that retargets a full text-to-video pipeline for a mobile NPU (Snapdragon 8 Gen 4 / X Elite Hexagon NPU), as opposed to optimizing a server-class GPU. The recipe is four stacked compressions on Pyramidal-Flow: (a) Text-Encoder Distillation — replace FLAN-T5-XXL (4.726B) with DistilT5 (0.13B) + a 4-layer MLP ContextAdapter (0.13B) trained MSE+cosine on 1.4M text-only prompts; (b) Asymmetric Decoder Distillation — swap the original codec-latent-VAE decoder for a mobile-friendly alternative while keeping the encoder and latent space intact (conv3D feature-map buffers, not parameter count, were the real mobile bottleneck); (c) MMDiT Block Pruning — 2.009B → 1.499B (>25%) with two-stage recovery distillation; (d) Pyramidal-DMD step distillation paired with an SSD-1B first-frame generator and QuickSRNet 2× super-res. End-to-end: 4.945B parameters, 3.5 GB peak RAM, 6.7s latency on a phone NPU, VBench Total 81.61 — competitive with cloud Pyramidal-Flow at 720p (81.72). The base-model choice is motivated by Pyramidal-Flow’s spatio-temporally pyramidal causal attention giving an asymptotic 32× attention-compute saving vs. fully bidirectional Wan/CogVideoX/HunyuanVideo at the same latent size — i.e. AR causal attention is reframed here as a mobile-deployment lever rather than just a long-horizon-rollout one. Adobe Premiere Pro plugin demo on Snapdragon X Elite laptop runs the entire pipeline on-device with no cloud calls.
  • [2026-05-25] MFLUX — MLX native implementations of state-of-the-art generative image models: Filip Strand’s MFLUX ships a from-scratch MLX port of seven SOTA generative image model families (Z-Image, FLUX.2, FIBO, SeedVR2, Qwen-Image, Apple Depth Pro, legacy FLUX.1) behind a single uv tool install CLI. The v0.11.1 release that triggered this filing adds Qwen Image Edit with multi-image support — the local Apple-Silicon counterpart to the cloud-side Qwen-Image-Edit-2509 multi-reference workflow. Feature surface includes quantization, multi-LoRA with scales, LoRA finetuning, in-context editing, virtual try-on, ControlNet (Canny), fill/inpainting, depth conditioning, Redux, and SeedVR2/FLUX-ControlNet upscaling. No quantitative serving benchmarks; the artifact is a working tool, not a paper. Significance: closes the local-Mac side of the serving-optimization frontier that the other three contributions cover from the H100/B200 side.
  • [2026-05-24] SGLang Diffusion — bringing SGLang's high-performance serving to diffusion models (LMSYS announcement): LMSYS announces SGLang Diffusion — an extension of the SGLang inference engine to image and video diffusion models, supporting Wan, Hunyuan, Qwen-Image, Qwen-Image-Edit, and FLUX. Up to 5.9× faster inference (baseline, hardware, and step count not disclosed in the announcement tweet). Critically, it is “built with FastVideo”, which is the systems stack from FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU. Exposes an OpenAI-compatible API plus CLI and Python. Significance: the per-deployment recipe pattern of Run FLUX.1-dev three times faster and the per-lab demo pattern of FastVideo are consolidating into a framework-level serving runtime, served from the same engine that runs production LLMs.
  • [2026-05-23] FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU: Hao AI Lab pushes the video-with-audio frontier of system-level serving to ~4.55s end-to-end for 5s × 1088×1920 × 24FPS TI2AV on a single B200 (3.9× reported over next-fastest 1080p option). Four-layer stack: SM100/SM103-tuned spatiotemporal attention, NVFP4 linears, end-to-end graph fusion across encode/prep/denoise/decode, and CPU-side ffmpeg optimization for the I/O envelope. Algorithmically untouches LTX-2.3 — the speedup is purely systems. NVIDIA Dynamo added FastVideo as a supported diffusion backend.
  • [2026-05-22] Run FLUX.1-dev three times faster: Modal walks through getting FLUX.1-dev under 3 seconds per 1024² image (3× over diffusers baseline). Two layers: ~1.5× from compile + fused QKV + channels-last, then another ~2× from First Block Caching at threshold 0.12. Cold-start regression from max-autotune clawed back ~30× via Inductor/Triton cache persistence + memory snapshots. The most thorough public account of the per-deployment image-DiT serving recipe.

Open questions

Section titled “Open questions”
  • The system-level (FastVideo / SGLang Diffusion) and algorithmic (distillation, few-step, AR rewiring — cf. Helios: Real Real-Time Long Video Generation Model) levers are stackable in principle but no filed work combines them. What is the compounded latency on a distilled LTX or distilled FLUX served through SGLang Diffusion? Neodragon’s stack is this composition end-to-end (distillation + system retargeting), but on a different hardware class. TurboDiffusion (TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times) is a second concrete instance of the composition — algorithmic (rCM + SLA) merged into one weight checkpoint and served with kernel-level (SageSLA, W8A8) acceleration. PiD (PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion) is a third — distillation (DMD2) applied to the decode stage rather than the denoise loop, with the decode-side adapter enabling early LDM exit. The remaining frontier composition would be PiD’s decoder stacked behind a distilled generator, all served through SGLang Diffusion.
  • The headline “up to 5.9×” in the SGLang Diffusion announcement is undisclosed-baseline. The same critique applies to TurboDiffusion’s 97-199× headlines: the comparison is to the unaccelerated Wan reference implementation, not to a strong consumer-GPU serving baseline, and quality is qualitative. The page’s standing recommendation (cf. Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments‘s symmetric-baseline critique) is to wait for an independent reproduction with disclosed hardware, model, resolution, step count, and reference stack — and quantitative quality metrics — before treating any of these numbers as decisive. PiD’s “5.9× faster than SeedVR2” is a single-baseline disclosed comparison on the 512²→2048² decode task; better than aggregate “Nx faster” but still a single-baseline number with project-page-only quality side-by-sides.
  • First Block Caching’s quality–speed tradeoff is measured qualitatively (animations, “images looked better”). At what threshold does it start to materially distort FID / human-eval / video-temporal-consistency, and is the curve smooth as Modal claims or does it cliff somewhere? No filed paper quantifies this against quantization or step-reduction.
  • The video-DiT analog of FBC has higher headroom than the image case in principle, because adjacent video timesteps are more correlated than adjacent image timesteps. No filed paper measures the video upper bound — Hao AI Lab’s FastVideo demo specifically avoids step-caching to preserve quality at the audio synchronization boundary.
  • Approximate caching (FBC), NVFP4 linears, and attention-kernel SM100 tuning all touch different fault lines in output quality. There is no published study isolating each contribution’s quality impact for video DiTs — only the aggregate “preserved” claim. TurboDiffusion’s Fig. 4 ablation isolates latency contributions per lever (rCM 33.3×, SageSLA 3.45×, W8A8 1.14×) but does not isolate quality contributions, leaving the per-component quality story open.
  • How does the SGLang Diffusion / FastVideo runtime compare on serving primitives — continuous batching, paged-cache analog for the denoising loop, request preemption — against bespoke per-model serving (Modal’s recipe, Replicate, fal)? The framework-level pitch only pays off if multi-tenant batched diffusion serving recovers GPU utilization the bespoke recipes can’t.
  • What does the Apple-Silicon / unified-memory side of the cost-of-serving curve look like compared to the NVIDIA datacenter side? MFLUX is the first concrete entry point but there are no head-to-head latency numbers — FLUX.2 [klein] at 1024² on an M-series Mac vs a B200 server has no published comparison.
  • Neodragon’s mobile-NPU regime opens a new question: which of the H100/B200-side techniques transfer to Hexagon-class NPUs? NVFP4 has no direct mobile-NPU analog (Hexagon’s quantization story is INT8/INT16 plus selective FP16); First Block Caching is in principle hardware-agnostic but no filed paper combines it with mobile deployment; full-pipeline graph fusion is the one lever that should transfer cleanly. TurboDiffusion’s W8A8 INT8 design point is closer to the mobile-NPU precision regime than NVFP4 is — does the recipe partially transfer to Hexagon?
  • What is the decode-side serving frontier for video LDMs? PiD is image-only. The video analog would be a single pixel-diffusion model that decodes a Wan/CogVideoX/HunyuanVideo latent and upsamples temporally + spatially in one model. None of the filed papers do this — Vivid-VR / FlashVSR / DUO-VSR are bolted on after the VAE decoder, not in place of it. Whether the early-LDM-exit trick survives the temporal axis is open.

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-09

Section titled “2025-09”

2025-05

Section titled “2025-05”

Unknown date

Section titled “Unknown date”