MiMo-V2.5-Pro-UltraSpeed: Pushing 1T-Parameter Model Generation Speed to 1000 TPS

TL;DR

Xiaomi MiMo and the TileRT systems team announce MiMo-V2.5-Pro-UltraSpeed, the first claim of >1000 tokens/s decode on a 1-trillion-parameter MoE — on a single 8×B200 commodity node, without Cerebras/Groq-style custom silicon. The recipe is full model–system codesign: FP4 (MXFP4) QAT applied selectively to MoE Experts (rest of model kept at higher precision), block-level masked parallel-prediction speculative decoding (DFlash) with block size 8, and the TileRT tile-level runtime that fuses operator boundaries so per-op microsecond gaps don’t fracture the decode pipeline. Headline numbers: 6.30 average accepted length on coding, 5.56 on math/reasoning, 4.29 on agent — and an API priced at 3× MiMo-V2.5-Pro for ~10× generation speed during a two-week trial window (Jun 9–Jun 23 2026).

Key claims

Trillion-parameter MoE decode at >1000 tok/s is achievable on commodity 8×B200 hardware via model–system codesign, not specialized chips [§Intro].
FP4 (MXFP4) quantization applied only to MoE Experts — keeping non-Expert modules at original precision — preserves capability under QAT while shrinking the bandwidth-bound parameters that dominate decode-time memory traffic; whole-model FP4 degrades reasoning/code [§Model: FP4 quantization].
Speculative decoding with a parallel-prediction draft (DFlash, block size 8) achieves average accepted lengths of 6.30 (coding), 5.56 (math/reasoning), 4.29 (agent), with peak 7.14 on some coding samples — i.e. 6–7 of 8 draft tokens accepted per verification round [§Speculative decoding, results table].
Open-ended conversational scenarios show notably lower acceptance rates than coding/reasoning — flagged as the team’s current generalization frontier rather than a solved problem [§Speculative decoding].
TileRT replaces the operator-boundary execution model with persistent kernels + tile pipelines that “eliminate execution gaps” at microsecond granularity, on the diagnosis that at 1000 tok/s every operator launch and global-memory round-trip is a visible cost [§System].
The DFlash draft is trained with Muon second-order optimizer + self-distillation so the lightweight draft still hits the acceptance rates that turn block-size-8 prediction into wall-clock throughput, not just per-step token count [§Speculative decoding].
Block size 8 is a deliberate choice: it caps verification overhead and allows the gains from acceptance length to translate into concurrency, instead of being eaten by verification cost [§Speculative decoding].

Method

The recipe has three tightly coupled layers. Model layer — MXFP4 quantization-aware training is applied selectively to MoE Expert weights (which are the bulk of the 1T parameters and the most quantization-tolerant), while non-Expert modules (attention, embeddings, LM head, shared FFN) retain higher precision. Draft model — instead of standard autoregressive speculative decoding, the draft is DFlash: a block-level masked parallel-prediction head that fills a block of 8 masked positions in one forward pass, breaking the serial bottleneck where each draft token requires its own forward. Trained with Muon + self-distillation to keep acceptance rates high at small parameter count. System layer — TileRT decomposes operators into tile-level tasks via a TileLang/TileScale compiler stack, dynamically reschedules compute/I/O/communication across the 8 B200s, and runs as a persistent kernel that doesn’t return control between layers. The MoE Experts’ FP4 layout and the DFlash block-decode geometry are co-designed with TileRT’s compute kernels — neither the quantization nor the speculative scheme is layered on top of an off-the-shelf runtime.

The DFlash method itself is from a research paper cited as arxiv:2602.06036.

Results

1000+ tokens/s decode on a 1T-parameter MoE on a single 8×B200 node (vs. prior open-systems best around 600 tok/s on GLM-5 / DSv3.2 from TileRT v0.1.4) [§Intro].
DFlash acceptance length table — Coding 6.30, Math/Reasoning 5.56, Agent 4.29 — with peak coding acceptance 7.14 out of 8 [§Speculative decoding, results table].
API pricing: 3× MiMo-V2.5-Pro cost for ~10× generation speed during the trial window; capacity-limited (10 sessions/day, 30 min/session, 5 min idle timeout) [§Trial access].
Whole-model capability “essentially on par with the original” under selective-Expert FP4 QAT; no breakdown numbers per benchmark are provided in the blog [§Model].

Notable absences: no end-to-end wall-clock vs. prior-state-of-the-art table, no TTFT numbers, no batch-size sweep, no comparison against vLLM/SGLang/TRT-LLM with EAGLE-3 + FP8 + MTP on the same model. The 1000 tok/s headline appears to be best-case decode under DFlash with high coding-style acceptance; the conversational regime is explicitly flagged as lower.

Why it’s interesting

This is the clearest published datapoint yet that the LLM Inference Efficiency page’s “verification dominates 42–95% of SD wall-clock” diagnosis (Speculative Decoding: Performance or Illusion?) can be flipped by parallel-prediction draft heads of the kind tracked in TiDAR: Think in Diffusion, Talk in Autoregression and TorchSpec: Speculative Decoding Training at Scale — when the draft itself stops being serial, acceptance length 6–7 becomes the binding constraint rather than draft latency. It is also the first time the wiki has seen MoE-Expert-only FP4 (vs. uniform NVFP4 across NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 / Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling) explicitly chosen because Experts are bandwidth-bound and quantization-tolerant while the rest of the model is not — a sharper read of the FP4 design space than “quantize everything you can.” And the system-side argument — that at microsecond-per-token granularity the dominant cost is operator-boundary gaps, not arithmetic — is the same diagnosis as the megakernel work catalogued under Loads and Loads of Fluffy Kittens: Compute-Communication Kernels with Multi-GPU ThunderKittens and Kittens virtual machine fuses entire training runs into a single GPU kernel (Ben Spector / Hazy Research), now arriving in a production trillion-parameter setting via TileRT: Tile-Based Runtime for Ultra-Low-Latency LLM Inference. Continues the Xiaomi MiMo program tracked across Xiaomi MiMo-V2-Flash — 309B/15B MoE with 5:1 SWA hybrid attention, 73.4 SWE-Bench Verified / Fuli Luo on MiMo-V2-Flash engineering — Hybrid SWA, MTP, MOPD post-training / MiMo-V2-Pro (a.k.a. Hunter Alpha) — Artificial Analysis model card, where the model team consistently leans on hybrid attention + sparse MoE + aggressive inference codesign instead of raw scaling.