Skip to content
Bud's Wiki

TileRT: Tile-Based Runtime for Ultra-Low-Latency LLM Inference

TL;DR

Section titled “TL;DR”

TileRT is an open-source LLM serving runtime built explicitly around per-request latency (TPOT) instead of aggregate throughput — the opposite tradeoff from vLLM/SGLang’s continuous-batching design. Its core mechanism is a compiler-driven tile-level execution model: LLM operators are decomposed into fine-grained tiles, then dynamically rescheduled at runtime so compute, I/O, and communication across multiple GPUs overlap end-to-end without the per-operator launch/synchronization gaps that dominate latency at microsecond-per-token decode rates. v0.1.4 supports DeepSeek-V3.2 and GLM-5 on 8×B200 with MTP (Multi-Token Prediction) speculative decoding, with a separate Xiaomi MiMo-V2.5-Pro-UltraSpeed deployment pushing the same runtime past 1000 tok/s on a 1T-parameter MoE via FP4 + block-parallel speculative decoding.

Key claims

Section titled “Key claims”
  • TileRT targets per-request latency for trading, interactive AI, real-time decision-making, and AI-assisted coding — explicitly a different point in the design space from throughput-optimized batch engines [README §Overview].
  • A tile-level runtime engine decomposes operators into fine-grained tasks and dynamically reschedules computation, I/O, and cross-device communication to minimize idle time and maximize hardware utilization [README §Overview].
  • The compiler stack used to produce these kernels (TileLang, TileScale) is upstreamed to the community as work progresses, but TileRT itself ships as pre-built wheels pinned to exact PyTorch / CUDA / glibc versions [README §Installation].
  • GLM-5.1-FP8 with MTP at average accepted length 3.2 is benchmarked at 1K input / 1K–192K output on 8×B200; chart shows MTP-on substantially exceeds MTP-off at the same model, with a further “peak under best-case MTP acceptance” envelope [README Fig. glm5_tilert_mtp].
  • Production deployment: GLM-5.1-highspeed on Z.ai is reported as running on TileRT — i.e. the runtime is past “research demo” into a live production product [README News §2026-05-22].
  • In collaboration with Xiaomi MiMo, TileRT v0.1.4 pushes MiMo-V2.5-Pro-UltraSpeed past 1000 tok/s on a 1T model on a single 8-GPU node without custom silicon [README News §2026-06-08].
  • v0.1.3 (Feb 2026) reports up to 500 tok/s on GLM-5-FP8 and up to 600 tok/s on DeepSeek-V3.2 with MTP; v0.1.4 (Jun 2026) is described as a “major performance upgrade” with model quality unchanged [README News §2026-02-14, §2026-06-01].

Method

Section titled “Method”

The architecture is built on three layered claims. (a) At ultra-low latency, the visible cost is operator-boundary overhead — kernel launches, hardware syncs, global-memory round-trips — and these fracture the execution flow into microsecond “execution gaps”. (b) A compiler that decomposes ops into tile-level tasks can re-schedule those tasks across SMs, NVLink, and HBM so that compute, communication, and memory ops overlap continuously rather than serializing at operator boundaries. (c) The result is a persistent kernel execution model where the GPU does not return control between layers; the runtime loads tiles, schedules them, and routes outputs without per-op CPU dispatch.

v0.1.4 ships two independent backend libraries (libtilert_dsv32.so, libtilert_glm5.so); a Python process loads exactly one via tilert.load_backend(...) and runs the matching DSAv32Generator or GLM5Generator. The weight pipeline is a one-time HF-checkpoint → 8-shard conversion via tilert.models.preprocess.weight_converter, after which the runtime reads pre-sharded *_dev_{0..7} weights directly. MTP is enabled per-generator with with_mtp=True and reports per-step acceptance-length statistics during decode.

Notably tight environment pinning: B200 GPUs, CUDA 13.2 runtime, Python 3.12, torch==2.11.0+cu130, glibc≥2.28, transformers==4.46.3 — distributed only as a manylinux_2_28 wheel against this exact stack. An official Docker image (ghcr.io/tile-ai/tilert:cu132-latest) is the recommended path.

Results

Section titled “Results”
  • v0.1.4 chart: GLM-5.1-FP8 on 8×B200, output 1K, input 1K–192K. Three bars per length — TileRT without MTP, TileRT with MTP at avg acceptance length 3.2, and a “peak under best-case MTP acceptance” envelope. The MTP-on bar materially exceeds the MTP-off bar; numbers in the chart but not in the README text [README Fig. glm5_tilert_mtp].
  • v0.1.1 (Dec 2025): ~35% further latency reduction (3–4× speedup over baseline) on 8×B200 [README News §2025-12-23].
  • v0.1.2-alpha.1 (Jan 2026): MTP with mtp=3 reaches ~590 tok/s on synthetic workloads [README News §2026-01-26].
  • v0.1.3 (Feb 2026): up to 500 tok/s GLM-5-FP8, up to 600 tok/s DeepSeek-V3.2 [README News §2026-02-14].
  • v0.1.4 (Jun 2026) + Xiaomi MiMo codesign: >1000 tok/s on 1T-parameter MiMo-V2.5-Pro-UltraSpeed [README News §2026-06-08].
  • Sample MTP run reports “Accepted length: mean=2.77, min=1, max=4” on DeepSeek-V3.2 for a 10-jokes prompt — an indicative single-prompt acceptance number, not a benchmark sweep [README §Running with MTP].

Why it’s interesting

Section titled “Why it’s interesting”

This is the open-source runtime that produced the 1000 tok/s headline in the sibling artifact MiMo-V2.5-Pro-UltraSpeed: Pushing 1T-Parameter Model Generation Speed to 1000 TPS — and unlike that closed-API system, the GLM-5 and DeepSeek-V3.2 backends are installable and re-runnable. It sits at the intersection of three threads the wiki has been tracking: (a) LLM Inference Efficiency‘s diagnosis that production speculative decoding (Speculative Decoding: Performance or Illusion?) is verification-bound rather than draft-bound — TileRT lands MTP into a runtime where the verification path is itself tile-overlapped, which is a different lever than Speculative Speculative Decoding‘s draft/verify parallelization; (b) IO-Aware Kernel Design‘s thesis that operator-boundary overhead is the real binding constraint at modern decode rates, closely related to the megakernel diagnosis in Loads and Loads of Fluffy Kittens: Compute-Communication Kernels with Multi-GPU ThunderKittens and Kittens virtual machine fuses entire training runs into a single GPU kernel (Ben Spector / Hazy Research) — TileRT productizes that diagnosis with a compiler stack and persistent kernel design; (c) the tight pinning to a single hardware/CUDA/PyTorch matrix is the cost of going past the abstraction boundaries vLLM and SGLang operate at. The fact that this runtime ships as a Z.ai production product (GLM-5.1-highspeed) and as a Xiaomi production product (MiMo-V2.5-Pro-UltraSpeed) within months is a strong signal that ultra-low-latency LLM serving is becoming a separate market segment from throughput-optimized serving — distinct from the Interaction Models: A Scalable Approach to Human-AI Collaboration streaming-session regime, which is small-prefill chunk-driven, but adjacent in motivation.

See also

Section titled “See also”