Kimi Linear: An Expressive, Efficient Attention Architecture

TL;DR

Kimi Linear is Moonshot AI’s hybrid linear-attention architecture pitched as a drop-in replacement for full-attention LLM backbones, with a 48B-total / 3B-active MoE checkpoint trained on 5.7T tokens and released openly (Base + Instruct, 1M context). The core primitive is Kimi Delta Attention (KDA) — a refined Gated DeltaNet that uses a specialized Diagonal-Plus-Low-Rank chunkwise algorithm with finer-grained gating — and the deployment recipe is a 3:1 layerwise hybrid of KDA and Multi-Head Latent Attention (MLA). Under fair, identical-recipe comparisons, Kimi Linear claims to outperform full-MLA on short-context, long-context, and RL-scaling regimes, while reducing KV-cache usage by up to 75% and reaching ~6× faster TPOT at 1M context. The KDA kernel lands in fla-org/flash-linear-attention and vLLM gets dedicated serving support.

Key claims

KDA refines Gated DeltaNet with a finer-grained gating mechanism, enabling more effective use of limited finite-state RNN memory [Abstract; README “Key Features”].
A bespoke chunkwise algorithm using a specialized Diagonal-Plus-Low-Rank (DPLR) variant of the transition matrices substantially reduces computation vs the general DPLR formulation while staying consistent with the classical delta rule [Abstract].
The deployed architecture is a layerwise hybrid at a 3:1 KDA-to-global-MLA ratio, which the team reports reduces memory usage while matching or surpassing full-attention quality [README “Key Features”].
The released checkpoint is 48B total / 3B activated parameters, 1M context, trained on 5.7T tokens; both Base and Instruct variants are open on Hugging Face [README model table].
On MMLU-Pro at 4k context, Kimi Linear reaches 51.0 at “similar speed” to full attention; on RULER at 128k context it reports 84.3 with a 3.98× speedup — claimed Pareto-optimal on that frontier [README Fig. (a)].
TPOT speedup vs full-MLA reaches up to 6× at 1M context length [README Fig. (b); Abstract].
KV-cache usage is reduced by up to 75% relative to the full-MLA baseline [README “Overview”; Abstract].
Under “fair comparisons” on 1.4T-token training runs, Kimi Linear outperforms full attention on long-context, short-context, and RL-style benchmarks [README “Key Features”].
The KDA kernel is open-sourced into fla-org/flash-linear-attention (path fla/ops/kda) and vLLM serving is provided as a first-class deployment path [README “Inference” / “Deployment”; Abstract].

Method

The architecture is a layerwise hybrid: most blocks use Kimi Delta Attention (KDA) and roughly one in four uses Multi-Head Latent Attention (MLA), giving the 3:1 ratio the team identifies as the sweet spot. KDA itself is an evolution of Gated DeltaNet — keep the delta-rule recurrence, but route through a finer-grained gating mechanism so the fixed-size linear-attention state is used more selectively. The chunkwise kernel exploits a specialized Diagonal-Plus-Low-Rank parametrization of the per-chunk transition matrix, which the team frames as the right structural restriction of general DPLR: cheap enough to be a serious replacement for softmax attention’s per-step compute, yet expressive enough to keep delta-rule semantics on the inter-chunk path. MLA layers carry the global / retrieval-heavy work that linear attention is known to lose on, mirroring the recipe in other hybrid releases.

Released artifacts are the 48B-A3B Base and Instruct checkpoints, both with 1M-context support, the KDA kernel as an fla-core ≥ 0.4.0 operator, and a vLLM serving recipe with tensor-parallel-4 / max-model-len 1048576.

Results

MMLU-Pro at 4k context: 51.0 at full-attention-comparable speed [README Fig. (a)].
RULER at 128k context: 84.3 with 3.98× speedup over the full-attention baseline; claimed Pareto-optimal on this frontier [README Fig. (a)].
TPOT: up to 6× faster than MLA at 1M context [README Fig. (b); Abstract].
KV-cache footprint: up to 75% smaller than full MLA [README “Overview”; Abstract].
The team reports wins for Kimi Linear over full MLA on short-context, long-context, and reinforcement-learning regimes “across all evaluated tasks” under identical training recipes [Abstract].
Two open checkpoints (Kimi-Linear-48B-A3B-Base, Kimi-Linear-48B-A3B-Instruct), both 1M context [README model table].

Why it’s interesting

This is now the strongest existence proof yet for the hybrid-linear-attention thesis at production scale, and it lands inside an actively converging cluster. MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention pushed Lightning Attention through a frontier RL run with a 1:7 softmax:linear ratio; Qwen3-Next-80B-A3B — hybrid Gated DeltaNet + Gated Attention, ultra-sparse 512-expert MoE, MTP (Alibaba Qwen announcement) shipped Qwen3-Next as an extremely sparse MoE on Gated DeltaNet + Gated Attention; Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning used Mamba-2 + global attention at 120B-12A; Mamba-3: Improved Sequence Modeling using State Space Principles rebuilt the SSM block for the inference-bound regime and explicitly recommends “linear core + sparse global attention” as the deployment pattern. Kimi Linear is the first to ground that pattern in the delta-rule + finer gating branch of the design space (vs Lightning’s transnormer or Mamba’s SSM) and to claim a fair, identical-recipe win over MLA across all three regimes — which, if it holds up, is exactly the comparison the wiki has been waiting for since the LLM Inference Efficiency discussion congealed around “memory traffic, not FLOPs, is binding at long generation”. The Jiaming framing in the Slack note — “minimax walked back, kimi became the proponent” — is worth registering: MiniMax-M1 was the previous flag-bearer for linear attention scaling, and watching which labs commit to which branch (KDA / Lightning / Gated DeltaNet / Mamba-2 / Mamba-3) is now the cleanest readout available on whether the hybrid pattern is becoming consensus or fragmenting by lab. The 3:1 hybrid ratio is also a much higher linear-fraction than MiniMax-M1’s 7:1 (linear:full) — they disagree by a factor of two on how much softmax you actually need, and that disagreement is the next thing the cluster needs to resolve.