Mercury: Ultra-Fast Language Models Based on Diffusion

TL;DR

Mercury is the first commercial-scale family of diffusion-based LLMs from Inception Labs, parameterized as a standard Transformer but trained to predict multiple tokens in parallel via a noise→denoise process over discrete tokens. The first products are Mercury Coder Mini and Mercury Coder Small, two coding-focused dLLMs that report 1109 and 737 output tokens/sec respectively on H100s — up to ~10× the throughput of speed-optimized AR frontier models — while matching open-weights and frontier-speed AR baselines on HumanEval, MBPP, EvalPlus, MultiPL-E, LiveCodeBench, BigCodeBench, and FIM. On Copilot Arena, Mercury Coder Mini ties for second on Elo and is the fastest model overall (25 ms median latency, ~4× faster than GPT-4o Mini). The report also frames Mercury as a drop-in OpenAI-API-compatible replacement and details the inference-engine work (dynamic batching, paging, custom kernels for parallel diffusion sampling) required to actually realize wall-clock wins.

Key claims

Mercury Coder Mini reaches 1109 output tokens/sec and Mercury Coder Small 737 tokens/sec on H100s under speed-optimized serving — “speeds previously possible only using custom chips” — while open-weights AR baselines and frontier speed-optimized AR APIs sit at 59-201 tokens/sec [Table 1, §1.1].
Quality vs speed-optimized AR baselines, Mercury Coder Mini scores: HumanEval 88.0, MBPP 77.1, EvalPlus 78.6, MultiPL-E 74.1, LiveCodeBench 17.0, BigCodeBench 42.0 — matching or exceeding all open-weights 7–8B AR coder baselines [Table 1].
Mercury Coder Small scores 90.0 HumanEval / 80.4 EvalPlus / 76.2 MultiPL-E / 45.5 BigCodeBench — comparable to Claude 3.5 Haiku, GPT-4o Mini, Codestral 2501, and Gemini 2.0 Flash Lite in the same speed-optimized class [Table 1].
Fill-in-the-middle: Mercury Coder Small 84.8 average (93.1 single-line, 76.5 random-span-light), beating every baseline including Codestral 2501 (82.5) — the strongest FIM result in the report [Table 3].
On Copilot Arena (paired human preference on real code completions), Mercury Coder Mini ties for 2nd in Elo at 993, ahead of GPT-4o Mini (939), GPT-4o (980), Gemini 1.5 Flash (977), and ties Codestral (992) — while being the fastest model with 0.25 s median latency, ~3× faster than the next model (Codestral at 0.31 s) [Table 4].
Architecture is a standard Transformer; “diffusion” refers to the training and generation algorithms (parallel coarse-to-fine denoising), not the network — so existing Transformer-side optimizations (FlashAttention, paged KV, instruction-tuning recipes) carry over [§2.1 Architecture].
Training extends [Lou et al., 2024] (SEDD-style discrete diffusion) with scale-up modifications and curated real+synthetic data; total training is “on the order of trillions of tokens” on H100 clusters [§2.1].
The inference engine implements dynamically batched diffusion sampling with paging and custom kernels for parallel workloads; the report frames the speed advantage as coming from arithmetic-intensity maximization on existing GPUs, not from custom hardware [§2.2 Serving].
The API is OpenAI-compatible and supports zero-shot, few-shot, and chain-of-thought prompting — Mercury can be a drop-in replacement for AR models in existing stacks [§2.2 Prompting, Serving].
“Scaling properties of diffusion LLMs are less well understood”; Mercury Small consistently beats Mini across all benchmarks, suggesting standard scaling holds for dLLMs but no formal scaling-law fit is reported [§3.2.2 Scaling].

Method

Mercury is parameterized as a Transformer denoiser f_θ(x_t, t) and trained on a discrete masked-diffusion objective: the forward Markov process q(x_t | x_{t-1}) progressively corrupts a clean token sequence x_0 toward a known prior q_T, and f_θ is optimized to minimize a weighted denoising loss Σ_t w(t) · E_{q(x_t|x_0)} [log p_θ(x_0 | x_t)] over noise levels (Eq. 1). At inference, sampling starts from x_T ~ q_T and iteratively denoises in parallel; multiple tokens are revised per forward pass, producing a coarse-to-fine refinement of the full output sequence rather than the strict left-to-right factorization of AR models. The report explicitly notes the diffusion-vs-architecture orthogonality: the same recipe could be parameterized by a U-Net or recurrent backbone, but Transformer is chosen for tooling and optimization compatibility.

Training proceeds in pretrain → fine-tuning → alignment stages over a mix of web crawls and proprietary real+synthetic data. Context length and fine-tuning specifics are mentioned but not quantified in the report. Inference is served by a proprietary engine with dynamic batching, paging, and custom kernels for parallel diffusion sampling, exposing an OpenAI-compatible API; the report attributes the wall-clock gains to maximizing utilization of existing NVIDIA H100 hardware rather than to algorithmic step reduction below what prior small-scale dLLMs achieved.

Results

Speed (output tokens/sec, third-party measured by Artificial Analysis on coding-focused 1k-input/1k-output prompts): Mercury Coder Mini 1109, Mercury Coder Small 737, vs Gemini 2.0 Flash Lite 201, Qwen 2.5 Coder 7B Instruct 195, Codestral 2501 171, Llama 3.1 8B Instruct 153, Nova Micro 148, Mistral Small 3 126, DeepSeek Coder V2 Lite 93, Claude 3.5 Sonnet 76, Claude 3.5 Haiku 61, GPT-4o 61, GPT-4o Mini 59, DeepSeek V3 27 [Table 1, §3.2.2 Speed].

Quality on standard coding benchmarks (Mercury Coder Mini / Mercury Coder Small): HumanEval 88.0 / 90.0; MBPP 77.1 / 76.6; EvalPlus 78.6 / 80.4; MultiPL-E 74.1 / 76.2; LiveCodeBench 17.0 / 25.0; BigCodeBench 42.0 / 45.5 [Table 1]. On MultiPL-E across C++/Java/JS/PHP/Bash/TS the Mercury Small averages 76.2, with notable strength on Bash (50.1 — second only to DeepSeek V3 at 59.0) [Table 2]. On FIM (single-line / random-span-light): Mercury Mini 92.9 / 71.5, Mercury Small 93.1 / 76.5 — the latter is the strongest FIM result reported [Table 3].

On Copilot Arena (n implicit, paired-preference Elo), Mercury Coder Mini ties for 2nd with Codestral at Elo 993/992 behind DeepSeek V2.5 (FIM) and Claude 3.5 Sonnet, while being the fastest model at 0.25 s median latency vs Codestral 0.31 s, Gemini 1.5 Flash 0.59 s, GPT-4o Mini 0.84 s [Table 4].

Why it’s interesting

Mercury is the upstream artifact for the whole “production diffusion LM” thread the wiki has been accumulating. The papers Bud has filed since — DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation (Apple’s masked-diffusion code model with coupled-GRPO), WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference (Tencent’s topological-reordering recipe to make a dLLM serve under standard causal-attention vLLM/PagedAttention infra), Introspective Diffusion Language Models (I-DLM’s introspective consistency + strided decoding), Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding (NVIDIA’s tri-mode AR+diffusion+self-speculation joint training), and Stabilizing Reinforcement Learning for Diffusion Language Models (StableDRL fixes GRPO for dLLMs) — are all downstream answers to Mercury’s open questions: how to RL them, how to serve them on existing AR infra, and how to scale them. Compared to Introspective Diffusion Language Models and WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference, Mercury is the data point with the cleanest commercial wall-clock claim (third-party measured 1100+ tok/s and #1 latency on Copilot Arena), but the report is thin on architecture/training detail — a deliberate trade-off versus the more reproducible academic recipes. For the wiki’s LLM Inference Efficiency thread, Mercury establishes the target (≈10× speedup over speed-optimized AR APIs at matched code quality) that subsequent diffusion-LM serving work has tried to recover under more deployable constraints.