Kimi K3 Technical Report — Open Frontier Intelligence

TL;DR

The formal technical report of Kimi K3 — a 2.8T-parameter native-multimodal MoE with 104B activated parameters, 93 layers (69 KDA + 24 Gated MLA), 896 routed experts with 16 activated per token, and a 1M-token context window. Compared with the launch blog, the report gives the mechanistic detail: the exact KDA lower-bounded log-decay reparameterization that lets diagonal tiles use Tensor Cores; Block Attention Residuals with 8 blocks of 12 layers plus embedding source; the SiTU-GLU activation with soft-cap constants β₁=4, β₂=25; the Quantile Balancing update rule based on Top-(k+1) routing quantiles; and the KDA / MoonEP / million-token agentic RL / prefix-cache infrastructure stack. Post-training uses RL across coding, agentic, knowledge, and reasoning environments at multiple effort levels, consolidated via multi-teacher on-policy distillation. Kimi K3 trails Claude Fable 5 and GPT-5.6 Sol overall but consistently leads the other evaluated open and proprietary models across long-horizon coding, agentic, knowledge, reasoning, and vision suites.

Key claims

2.8T total / 104B activated parameters, 93 layers, 896 routed experts activated 16-of-896 (sparsity 56), 1M context, MoonViT-V2 vision encoder (401M), MXFP4 weights + MXFP8 activations under QAT, vocabulary 160K [§Model Summary].
Kimi Delta Attention is used in a 3:1 KDA:Gated-MLA hybrid, with an additional Gated MLA at the end of the backbone to guarantee a final global-attention layer [§2.1].
Kimi K3 replaces Kimi Linear’s unbounded negative-Softplus log-decay with a scaled sigmoid bounded below at gmin=−5, keeping the reciprocal rescaling factor inside BF16 dynamic range so both diagonal and off-diagonal tiles run as dense Tensor Core matmuls (eliminating the position-pair diagonal path) [§2.1.1, Fig. 3].
All MLA layers use No Position Encoding (NoPE); the KDA layers supply position-sensitivity and recency, and MLA supplies unrestricted global content interaction — this removes RoPE-frequency retuning / YaRN when extending context length [§2.1.2].
MLA gets an input-dependent, channel-wise full-rank output gate (Sigmoid(Wg x) ⊙ o), matching KDA’s new gating; the attention output is kept in FP32 during training to correct flash-attention’s biased rounding error, with the training kernel redesigned to overlap the FP32 tile with KV staging [§2.1.2].
Attention Residuals uses learnable per-layer pseudo-queries with RMSNorm-normalized keys; Kimi K3 uses Block AttnRes (8 blocks of 12 layers plus embedding source = 9 sources) to reduce memory / cross-stage-communication overhead from O(Ld) to O(Nd), while online-softmax merges parallel inter-block and sequential intra-block partial sums to reduce inference cost [§2.2, Eq. 8-10].
Stable LatentMoE inserts RMSNorm between routed-expert aggregation u and the up-projection W↑, stabilizing the routed branch’s scale before merging with the shared branch and consistently improving validation loss [§2.3.1, Eq. 11].
SiTU-GLU applies soft-cap β·tanh(x/β) to both branches of the SwiGLU with β₁=4 (gate) and β₂=25 (up), producing a bounded gate/up product |f(x)| ≤ β₁β₂=100 while preserving SwiGLU’s near-origin response [§2.3.2, Fig. 4, Eq. 12].
Quantile Balancing sets each expert’s bias by running Top-(k+1) routing on the biased score and taking the (k+1)-th entry per token as a cutoff α_i^(t); the next bias for expert j is the (q+1)-th largest margin s_{i,j}+b_j^(t)−α_i^(t), giving hyperparameter-free load balancing at 896-expert scale [§2.3.3, Fig. 5, Eq. 13].
Reported ≈2.5× overall scaling efficiency vs Kimi K2 from the combined architecture + data + training recipe changes [Abstract, §1, §2].
Infrastructure: MoonEP for perfectly balanced expert-parallel training with static shapes and zero-copy communication; KDA-specific fused kernels, KDA Context Parallelism, and state-aware prefix caching; million-token agentic RL with partial rollouts, external KV-cache retention, adaptive throttling, and resumable microVM sandboxes [§1, Contributions].
Post-training runs RL across long-horizon coding, general agents, general reasoning and knowledge, and multimodal reasoning with vision-in-the-loop tool use, then consolidates domain- and effort-specialized policies through multi-teacher on-policy distillation [§1].
Benchmark headlines: BrowseComp 91.2, DeepSearchQA F1 95.0, MCPMark-Verified 94.5, SWE-Marathon 42.0, ProgramBench 77.8, FrontierSWE 81.2, Terminal-Bench 2.1 88.3, HLE-Full 43.5 / 56.0 (single / with tools), GPQA Diamond 93.5, Kimi Code Bench 2.0 72.9; Kimi K3 trails Fable 5 / GPT-5.6 Sol overall but leads Opus 4.8 / GPT-5.5 / GLM-5.2 [Fig. 1, §3].

Method

The report is organized around the three-dimensional information-flow argument: sequence (KDA + Gated MLA hybrid), depth (Attention Residuals), and width (Stable LatentMoE). Section 2.1 develops KDA’s chunkwise-parallel form, then shows the lower-bounded log-decay parameterization (g = gmin · Sigmoid(e^A · z) with gmin = −5, learnable per-head A) that keeps the reciprocal rescaling factor within BF16 range and lets all causal tiles run as dense Tensor Core matmuls. Gated MLA gets NoPE (positional information rides on KDA) and a full-rank data-dependent output gate. Section 2.2 introduces Block Attention Residuals: each of 8 blocks of 12 layers aggregates its layer outputs by summation b_n = Σ_{j∈B_n} f_j(h_j) with the embedding as b_0; attention over block representations uses learned per-layer pseudo-queries with RMSNorm-normalized keys via softmax kernel exp(q · RMSNorm(k)), reducing memory/communication from O(Ld) to O(Nd). Section 2.3 details Stable LatentMoE: shared experts run at full width, routed experts operate in latent width ℓ=3584 with W↓ → gated multi-branch FFN → W↑; RMSNorm before W↑ controls activation scale; SiTU-GLU soft-caps both gate and up branches; Quantile Balancing derives expert biases from Top-(k+1) routing quantiles in a single forward pass, giving hyperparameter-free load balancing at extreme sparsity. Section 2.4 covers MoonViT-V2 + a lightweight projector into the shared embedding space; §2.5 covers Per-Head Muon. Infrastructure sections cover MoonEP (perfectly balanced EP with static shapes, zero-copy comms), KDA systems co-design (fused kernels, Context Parallelism, state-aware prefix caching in vLLM), a co-located million-token agentic RL system with resumable microVM sandboxes and partial rollouts, and cache-/budget-aware fleet scheduling for serving.

Results

Reasoning & Knowledge: GPQA Diamond 93.5 (leading K3, tied with GPT-5.5), CritPt 23.4 (trails Fable/GPT-5.6/GPT-5.5), AA-LCR 74.7 (best), HLE-Full 43.5 (raw) / 56.0 (with tools) [§3 table].
Coding: DeepSWE 67.5 (K3) vs Fable 70.0, GPT-5.6 Sol 73.0, Opus 4.8 59.0, GPT-5.5 67.0, GLM-5.2 46.2; ProgramBench 77.8 (best); Terminal-Bench 2.1 88.3 (2nd behind GPT-5.6); FrontierSWE 81.2 (2nd behind Fable 86.6); SWE-Marathon 42.0 (best); Kimi Code Bench 2.0 72.9 (2nd behind Fable 76.9) [§3, Fig. 1].
Agentic: BrowseComp 91.2 (best), DeepSearchQA F1 95.0 (best), ResearchRubrics 76.2 (best), GDPval-AA v2 Elo 1686 (3rd behind Fable 1747, GPT-5.6 1736), MCPMark-Verified 94.5 (best), MCP-Atlas 84.2 (2nd behind Fable 84.7), AutomationBench 30.8 (best), τ³-Banking 33.4 (best), Harvey Lab-AA 94.6 (best), OSWorld-Verified 84.8 (2nd behind Fable 85.0), OSWorld 2.0 58.3 (trails Fable 66.1) [§3 table].
Vision: WorldVQA ForceAnswer 51.0 (2nd behind Fable 56.7); the full report table extends beyond the fetch truncation.
Moonshot’s summary: K3 trails Fable 5 and GPT-5.6 Sol overall but “consistently outperforms other open and proprietary models evaluated in our suite” [Abstract].

Why it’s interesting

This is the mechanistic backup for the K3 launch blogs — it turns the launch claims into recipes the wiki can actually cite. Three points that matter for the team:

KDA lower-bounded decay is a hardware trick, not a modeling trick. The scaled-sigmoid reparameterization vs Kimi Linear’s Softplus is what promotes the diagonal tiles from position-pair scalar work to Tensor Core matmuls, and it’s the same class of numerical-precision trade-off argued in Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention and Defeating the Training-Inference Mismatch via FP16 — bounded-range architecture choices unlock the FP4/FP8 QAT stack. This is the trillion-scale companion evidence to Kimi Linear: An Expressive, Efficient Attention Architecture under the Hybrid Linear Attention cluster.
Block AttnRes is a real architectural component now, not a research demo. The report specifies exactly 8 blocks × 12 layers with embedding-as-source and RMSNorm-in-the-key of the pseudo-query attention. This is the first frontier-scale deployment of the depth-selective retrieval framing from Attention Residuals (AttnRes / Block AttnRes) — Technical Report, and the O(Ld) → O(Nd) memory/comm argument makes it interesting alongside distributed-training papers under Distributed training parallelism.
Quantile Balancing at 896 experts. The Top-(k+1)-routing quantile trick from Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method is now stress-tested at the deepest sparsity ratio in the MoE Routing Design cluster (1:56), pairing with the kernel-side story of SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations. Combined with SiTU-GLU’s soft-cap-and-sigmoid gating, it also connects the training-stability threads in Training stability at scale — bounded activations plus bounded router scores are the recipe here.