vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models

TL;DR

vLLM-Omni is an open-source serving system for any-to-any multimodal models — pipelines that combine multiple autoregressive LLMs with diffusion transformers (DiTs) and other specialized generators to produce mixed text / image / audio / video outputs. The contribution is a stage graph abstraction: developers decompose a model (e.g. Qwen3-Omni’s Thinker→Talker→Vocoder, GLM-Image’s AR→DiT, BAGEL’s MoT experts) into nodes served by independent execution engines, with user-defined transfer functions on the edges. Each stage gets its own batching, KV-cache management, parallelism, and accelerator budget; a unified connector (shared-memory single-node, Mooncake RDMA multi-node) moves intermediate data between stages, and streaming stage output lets downstream stages start before upstream ones finish. Reported up to 91.4% reduction in job completion time vs the Hugging Face Transformers reference for Qwen3-Omni, with 12.97× Thinker TPS and 7.98× Talker TPS.

Key claims

Existing LLM serving frameworks (vLLM, SGLang) and diffusion frameworks (Diffusers) are step-centric — each implements one forward pass of one paradigm (AR decode or DiT denoise), and cannot natively express multi-stage any-to-any pipelines; developers must hand-orchestrate stages outside the framework, losing continuous batching, chunked prefill, and per-stage resource control [§2.2].
vLLM-Omni’s frontend exposes a stage graph where nodes are model components (AR LLM, DiT, CNN module) and edges are stage-transfer functions transforming intermediate state between stages [§3.2, Fig. 3].
Each AR stage is served by an independent vLLM engine with its own scheduler, KV manager, and model runner; each DiT stage is served by a dedicated diffusion engine; a per-request preprocess function runs each iteration to combine outputs from prior stages with the current forward input (necessary for e.g. Qwen-Omni Talker, which concatenates Thinker hidden states at each decoding step) [§3.3].
The diffusion engine integrates flash-attention, SAGE-attention, TurboAttention, TEACache, cache-dit, RingAttention context parallelism, and Ulysses sequence parallelism, and serves Z-Image, Qwen-Image, Flux, Qwen-Image-Edit, LongCat-Image-Edit, Wan2.2 variants, and HunyuanVideo [§3.3].
Streaming stage output: downstream stages that don’t require complete upstream output start incrementally — the Vocoder begins audio synthesis from initial Talker tokens rather than waiting for the full codec sequence; reduces final TTFT [§3.3].
A unified connector moves data between stages: shared-memory queues for single-node, Mooncake-based TCP/RDMA for distributed; same put/get interface for KV-cache, hidden states, embeddings, image tensors, and audio tensors. Intra-stage prefill→decode KV transfer and encoder→prefill multimodal-cache transfer are handled by the same mechanism, remaining compatible with existing EPD disaggregation [§3.4].
On Qwen3-Omni with two 80GB accelerators (Thinker tensor-parallel across both, Talker on device-1, Vocoder on device-0), vLLM-Omni reduces RTF 90.7% and JCT 91.4% vs the Hugging Face Transformers reference; Thinker TPS 12.97×, Talker TPS 7.98× [§4.2, Fig. 6].
On Qwen2.5-Omni same testbed: RTF −61.4%, JCT −61.6%, Thinker TPS 1.29×, Talker TPS 1.97× — the smaller speedup vs Qwen3-Omni reflects the Qwen2.5 baseline being closer to its ceiling [§4.2, Fig. 6].

Method

The system has two interfaces. The frontend asks the developer to decompose an any-to-any model into a stage graph: each node has a forward function (step-centric, identical to a normal vLLM model forward) and a preprocess function that runs every decode iteration to build the input from preceding-stage state. Edges are stage-transfer functions that fire once at stage boundaries to transform query state for the next stage. The example in §3.2 implements Qwen2.5-Omni with three node forwards (thinker_forward, talker_forward, dit_decode), two preprocesses (mm_encode, process_input), and two edges (Thinker2Talker, Talker2Vocoder).

The backend runs an orchestrator process that instantiates one execution engine per stage with its configured parallelism and memory budget. AR stages run on vLLM engines with independent schedulers, PagedAttention KV managers, and continuous batching. DiT stages run on a dedicated diffusion engine with the optimizations listed in §3.3. The unified connector decouples transport from model logic — small payloads flow through control queues, larger payloads through system shared memory single-node or Mooncake RDMA multi-node, and the same interface also handles vLLM’s existing prefill↔decode KV transfer. Streaming stage output asynchronously forwards partial outputs (newly generated tokens, embeddings) to downstream stages as they’re produced.

Results

Reported on two 80GB accelerators with 24 CPU cores and 192 GB system memory, vLLM 0.12.0. Inputs: librispeech_asr (audio), food101 (image), ucf101-subset (video), first 100 queries each. Baselines: Hugging Face Transformers for Qwen-Omni (vLLM and SGLang support only the Thinker half), original implementations for BAGEL and MiMo-Audio, Diffusers for diffusion-only models.

Qwen3-Omni: RTF −90.7%, JCT −91.4%, Thinker TPS 12.97×, Talker TPS 7.98× vs Transformers [§4.2, Fig. 6].
Qwen2.5-Omni: RTF −61.4%, JCT −61.6%, Thinker TPS 1.29×, Talker TPS 1.97× vs Transformers [§4.2, Fig. 6].
Headline number across the whole benchmark suite: up to 91.4% JCT reduction [Abstract, §4.2].

Why it’s interesting

Pulls together two threads the wiki tracks separately. On the LLM Inference Efficiency side, this is the first filed paper that treats any-to-any serving as a first-class problem rather than a collection of per-model hacks — the stage graph is the natural abstraction the wiki’s diffusion-serving and LLM-serving notes have been gesturing at, generalizing vLLM’s EPD disaggregation to arbitrary graphs of AR + DiT + neural stages. On the Diffusion serving optimization side, it’s the framework-level companion to FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU and the SGLang Diffusion announcement (SGLang Diffusion — bringing SGLang's high-performance serving to diffusion models (LMSYS announcement)): the bespoke per-deployment recipes are consolidating into a single runtime that can serve unified Unified Multimodal Models like Qwen3-Omni, LongCat-Flash-Omni (LongCat-Flash-Omni Technical Report), Ming-Omni (Ming-Omni: A Unified Multimodal Model for Perception and Generation), and BAGEL. The 12.97× Thinker speedup vs Transformers on Qwen3-Omni is large enough to suggest most existing Omni benchmark wall-clock numbers should be treated as overestimates of inference cost; the missing comparison the page-level open questions still want is vLLM-Omni vs SGLang Diffusion + a hand-written Thinker-Talker bridge, under matched hardware.