M*: A Modular, Extensible, Serving System for Multimodal Models

TL;DR

M* is a unified serving runtime from Stanford for composite multimodal models — UMMs (BAGEL), SpeechLMs (Orpheus), Omni models (Qwen3-Omni), VLAs (π0.5), and world models (V-JEPA 2) — that breaks the single-AR-loop assumption baked into vLLM and SGLang. The core abstraction is the Walk Graph: each model is declared as a graph of component nodes connected by tensor edges, with named Walks (labeled subgraphs) selected per request by a small state machine. Sequential, Parallel (CFG fan-out), Loop (diffusion / world-model rollout / AR decode), and streaming edges are first-class primitives, so continuous batching, CUDA-graph replay, paged-attention KV cache, and tensor transport apply uniformly to every loop type. Against vLLM-Omni and SGLang-Omni — both of which model requests as flat stage pipelines — M* matches or beats the strongest specialized baseline on five real models, with up to 2.7× higher TTS throughput on Qwen3-Omni vs vLLM-Omni, 4× vs SGLang-Omni, and 12.5× vs Meta’s native V-JEPA 2 rollout.

Key claims

vLLM and SGLang are modality-locked: built for AR text decode, image inputs are encoder add-ons at prefill, and the decode loop’s output is always text — there is no first-class composition of heterogeneous components into loops or parallel branches and no cross-component streaming [§“Why today’s serving stacks fall short”].
vLLM-Omni and SGLang-Omni model a request as a flat DAG of stages wired by explicit data-transfer functions; iteration stays inside a stage and stages cannot be composed in parallel, so patterns like diffusion loops and classifier-free guidance (CFG) fan-out must be added per-model as glue code (e.g., vLLM-Omni runs BAGEL’s CFG through a bespoke plugin built on torch.distributed) [§“Why today’s serving stacks fall short”].
The Walk Graph generalizes prior abstractions: graph nodes are model components (not stage instances), composition supports Sequential / Parallel / Loop / Stream, loops can span any subgraph (not just a single stage), and placement is per-component with optional per-Walk overrides [Table 1].
BAGEL is declared with four components (vit_encoder, vae_encoder, LLM, vae_decoder) and a per-request state machine that picks among Walks prefill_text, prefill_vit, prefill_vae, image_gen (+ CFG variant), and decode — image-understanding requests never touch the diffusion loop or VAE decoder; image-generation requests never run the ViT path [§“The Walk Graph, by example”].
The Loop primitive covers diffusion steps, world-model rollouts, and AR token decode uniformly — continuous batching and CUDA-graph replay apply to flow steps exactly as to token decode; a rollout’s KV cache persists across steps instead of being recomputed [§“Add a loop”, §“What the Walk Graph unlocks”].
Three-way CFG runs as a Parallel block of three LLM “views” (one unconditional + two conditioned) sharing the same underlying language model under three names; each branch can sit on its own GPU with no per-model glue code [§“Add parallelism”].
Placement is a YAML mapping from logical nodes to GPU ranks with optional per-Walk scoping (a node can be placed differently for different Walks, e.g., prefill on one GPU, decode on another); the same API covers component disaggregation, prefill/decode disaggregation, and tensor-parallel sharding [§“Placement”].
Streaming is a first-class edge type with a small library of chunk policies (FixedChunkPolicy, LeftContextChunkPolicy, sliding-window) declared once in a model’s topology; Qwen3-Omni’s Thinker → Talker → Code2Wav chain runs as a streaming graph with chunk_size=1 from Thinker to Talker and a 25-frame chunk with 25-frame left context from Talker to Code2Wav, replacing bespoke per-model streaming code [§“Streaming, by example”].
A Conductor tracks each request’s Walk and dispatches to per-GPU Workers that route tensors directly to one another over a pluggable data plane (shared memory / RDMA / TCP via Mooncake); overlapped scheduling prepares the next batch and attention plan on a separate stream and defers loop stop-checks by one iteration to keep the GPU from stalling on CPU scheduling [§“Under the hood”].
On Qwen3-Omni TTS (2×H200), M* sustains ≈2.7× the throughput of vLLM-Omni and ≈4× that of SGLang-Omni at batch=16, and stays real-time through batch=32 where SGLang-Omni’s tail latency runs past the real-time threshold [Table “Does it work?”].
On Qwen3-Omni TTS with the Thinker tensor-parallel-sharded across two GPUs, M* keeps a ≈3.8× throughput lead over SGLang-Omni at batch=16 — sharding and disaggregation working together [Table “Does it work?”].
On BAGEL image editing (3×H100, CFG-parallel, B=1), M* runs ≈2.6× lower latency than vLLM-Omni’s default two-stage pipeline and ≈1.2× lower than vLLM-Omni’s best-tuned single-stage configuration; text-to-image margin is ≈1.3× [Table “Does it work?”, §“What is vLLM-Omni’s single-stage config?”].
vLLM-Omni’s “single-stage” config collapses LLM, ViT, VAE, and DiT into one diffusion process to eliminate cross-stage transfer for editing — but loses continuous batching, token streaming, and paged-attention KV management for the text/understanding path; M* avoids the tradeoff because a Walk names exactly the components a request uses [§“What is vLLM-Omni’s single-stage config?”].
On BAGEL image-to-text (1×H100, B≤16), M* returns the first token ≈1.6× faster than vLLM-Omni and holds a throughput lead that grows with batch (≈46% at short outputs), at the cost of 1–3 ms higher median inter-token latency [Table; §“Image understanding is more nuanced”].
On Orpheus TTS (1×H200), M* posts a lower real-time factor and ≈1.3× higher throughput than VoxServe at batch=8 [Table “Does it work?”].
On V-JEPA 2 rollout (1×H100), expressing the rollout as a Loop with a persistent KV cache instead of recomputing it each step yields up to 12.5× over Meta’s native rollout [Table “Does it work?”].

Method

A model author writes three things: (1) a graph of GraphNodes with named input edges and named output edges (the wiring); (2) a NodeSubmodule per node (a torch.nn.Module with prepare_inputs and a pure-tensor forward — batching, KV caching, CUDA graphs, tensor transport are the runtime’s job); (3) a state machine next_walk(state) that picks the next Walk per request. Walks are subgraphs built from four primitives — Sequential, Parallel, Loop (with max_iters or EOS-style stop), and streaming StreamingGraphEdges tagged with a target_partition and matched to chunk policies declared once in a PartitionTopology. A placement YAML maps logical nodes to GPU ranks with optional graph_walks scoping so a node can be placed differently for different Walks.

The runtime is a Conductor (tracks requests, dispatches Walks) + per-GPU Workers. Worker engines come in two flavors: a modality-agnostic AR engine with FlashInfer paged-attention KV cache, and a stateless engine for encoders / decoders / audio codecs; both support continuous batching and CUDA-graph replay. Tensor transport is pluggable (shared memory / RDMA / TCP via Mooncake), chosen by where the producer and consumer live. Overlapped scheduling prepares the next batch and its attention plan on a separate stream while the current step runs on the GPU, and defers each Loop stop check by one iteration to keep the GPU off the critical path. Tensor-parallel sharding (parallel linears, vocab-parallel embeddings, sharded MoE and KV cache, NCCL collectives) is set with a tp_size in the placement file.

Results

Five-model benchmark suite, each compared to the strongest specialized baseline:

BAGEL text→image, 3×H100, CFG-parallel, B=1: ≈1.3× lower latency vs vLLM-Omni [Table].
BAGEL image editing, 3×H100, CFG-parallel, B=1: up to 2.6× lower latency vs vLLM-Omni default; ≈1.2× vs vLLM-Omni single-stage [Table, §“What is vLLM-Omni’s single-stage config?”].
BAGEL image→text, 1×H100, B≤16: ≈1.6× faster first token, ≈46% higher throughput at short outputs, +1–3 ms median ITL [Table, §“Image understanding is more nuanced”].
Qwen3-Omni TTS, 2×H200: ≈2.7× throughput vs vLLM-Omni at B=16, ≈4× vs SGLang-Omni; real-time through B=32 [Table].
Qwen3-Omni TTS, TP-2 Thinker, 2×H200: ≈3.8× throughput vs SGLang-Omni at B=16 [Table].
Orpheus TTS, 1×H200: ≈1.3× throughput vs VoxServe at B=8, lower RTF at every benchmarked batch size [Table].
V-JEPA 2 rollout, 1×H100: up to 12.5× faster than Meta’s native rollout [Table].

The V-JEPA 2 number is the largest single multiplier reported and isolates the value of treating the rollout as a Loop with a persistent KV cache rather than recomputing per step — i.e. the contribution is structural, not kernel-level.

Why it’s interesting

Directly extends and improves on vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models — vLLM-Omni’s stage-graph abstraction is exactly the “flat DAG” subset of the Walk Graph that M* explicitly identifies as too restrictive (Table 1). The two systems agree on the diagnosis (existing LLM serving stacks are step-centric and modality-locked, you need first-class graph-of-models) but disagree on the graph algebra: vLLM-Omni’s edges fire once at stage boundaries with iteration trapped inside a stage; M*‘s Loop is a generic subgraph primitive that wraps diffusion, AR decode, and world-model rollout under the same machinery — and Parallel is a first-class operator rather than per-model glue. The empirical consequence is the 2.7× headline vs vLLM-Omni on the model that vLLM-Omni was specifically designed for (Qwen3-Omni).

Connects two wiki threads that have been parallel: LLM Inference Efficiency (where verification-dominated SD, prefix cacheability, and streaming-session inference all assume an AR loop) and Diffusion serving optimization (where the dominant tools — FastVideo, SGLang Diffusion, TurboDiffusion — are per-deployment recipes for the denoise loop). M* is the framework-level recipe that runs both under one runtime, and the V-JEPA 2 12.5× number flags World Foundation Models rollouts as a third serving regime that benefits from the same primitives — the wiki hadn’t tracked world-model serving as a serving-optimization frontier before this.