Kimi K3 — Open Frontier Intelligence (2.8T MoE with KDA + AttnRes)

TL;DR

Moonshot’s Kimi K3 is a 2.8T-parameter open MoE (16-of-896 experts effectively activated) that composes the wiki’s separately-filed Moonshot architecture threads into a single frontier release: Kimi Delta Attention (KDA) as the linear-attention backbone, Attention Residuals (AttnRes) for depth-aware routing of layer outputs, Quantile Balancing for hyperparameter-free MoE load balancing, and Per-Head Muon as an optimizer tweak. Kimi K3 uses MXFP4 weights + MXFP8 activations under quantization-aware training from SFT onward, and native vision + 1M-token context. Moonshot reports ≈2.5× overall scaling efficiency vs Kimi K2 and positions K3 behind Claude Fable 5 / GPT 5.6 Sol but ahead of all other tested models, with weights promised by 2026-07-27.

Key claims

Kimi K3 is a 2.8T-parameter MoE that effectively activates 16 of 896 experts inside a “Stable LatentMoE” framework, claimed as the first open 3T-class model [§An Open 3T-Class Model].
The architectural stack is Kimi Delta Attention (KDA) + Attention Residuals (AttnRes) + Stable LatentMoE + Quantile Balancing + Per-Head Muon + Sigmoid Tanh Unit (SiTU) + Gated MLA, yielding ~2.5× overall scaling efficiency vs Kimi K2 [§An Open 3T-Class Model, §Architecture and Infrastructure].
Quantization-aware training is applied from the SFT stage onward using MXFP4 weights with MXFP8 activations for broad hardware compatibility [§Architecture and Infrastructure].
A fully balanced expert-parallel training method with static shapes and no host-synchronization on the critical path is used to prevent expert imbalance from degrading throughput at large EP scales; deployment is recommended on supernodes of ≥64 accelerators [§Architecture and Infrastructure].
KDA-compatible prefill caching has been contributed to vLLM to preserve prefix-cache economics under the linear-attention backbone [§Architecture and Infrastructure].
In a 24-hour sandboxed GPU-kernel-optimization eval spanning AttnRes, KDA, and a 512-head-dimension MLA kernel across H200 and a non-NVIDIA GPGPU, K3 is reported competitive with Claude Fable 5 (with fallback) and substantially ahead of Opus 4.8, GPT 5.6 Sol, and GPT 5.5 [§Kernel Optimization].
In a from-scratch GPU-compiler build task, K3 produced “MiniTriton” — a Triton-like tile-IR compiler on MLIR with PTX codegen — that matches or beats Triton / torch.compile on the reported roofline benchmarks and sustains end-to-end nanoGPT training [§GPU Compiler Development].
In a 48-hour autonomous chip-design run on open EDA + Nangate 45nm, K3 built a 4 mm² chip that closes timing at 100 MHz and simulates >8,700 tokens/s decode for a nano model built on K3’s own architecture (1.46M cells, 0.277 MB SRAM, INT4 MAC with fused dequant) [§Chip Design].
Reported research case: reproducing the I–Love–Q universal relations in computational astrophysics — reviewing 20+ papers, evaluating 300+ EOS, generating 3,000+ lines of Python, producing an interactive HTML dashboard — in ~2 hours vs a reported 1–2 weeks of expert time [§Coding for Research].
BrowseComp is evaluated with 300K-token context-compaction (matching Claude model cards); a 1M-token no-compaction run scores 90.4 [§Footnotes].
Recommended runtime posture: preserved-thinking-history mode, KimiCode / Claude Code compatible harnesses, and no mid-session model swaps — quality is reported “highly unstable” if historical thinking content isn’t passed back [§Limitations].

Method

Kimi K3 is a decoder-only MoE. The attention stack is Kimi Delta Attention (KDA) — the hybrid linear-attention primitive introduced in Kimi Linear: An Expressive, Efficient Attention Architecture — combined with Attention Residuals (Attention Residuals (AttnRes / Block AttnRes) — Technical Report), which selectively retrieves representations across depth rather than accumulating them with unit weight as standard PreNorm does. The MoE layer is “Stable LatentMoE” activating 16 of 896 experts per token, with load-balancing driven by Quantile Balancing (Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method) — router-quantile-derived expert allocation with no auxiliary loss and no balancing hyperparameter. Optimization is Per-Head Muon (Muon with per-attention-head decoupling). Activation and attention-selectivity ingredients are Sigmoid Tanh Unit (SiTU) and Gated MLA respectively.

Quantization: MXFP4 weights + MXFP8 activations, QAT from SFT onward — extending the Attn-QAT recipe (Attn-QAT: 4-Bit Attention With Quantization-Aware Training) from attention-only to the full pipeline. Training infra: fully balanced expert-parallel with static shapes and no host-sync on the critical path. Serving: KDA prefix caching landed in vLLM, and Moonshot recommends supernode deployments ≥64 accelerators; API pricing is $0.30 /$ 3.00 / $15.00 per MTok (cache-hit input / cache-miss input / output) with a reported >90% cache-hit rate on coding workloads under Mooncake disaggregated inference. Context window is 1M tokens; the model is natively multimodal (text + image + video).

Results

Reported ~2.5× overall scaling efficiency vs Kimi K2 for the same compute [§An Open 3T-Class Model].
On the DeepSWE leaderboard, K3 with the KimiCode harness ranks alongside frontier models; with the mini-SWE-agent harness K3 scores 67.3 [§Footnotes / DeepSWE].
Terminal-Bench 2.1: KimiCode harness is used for K3; Moonshot’s blog compares against GLM-5.2 (Claude Code), Opus 4.8 / Fable 5 (Terminus 2), and GPT 5.5 / 5.6 Sol (Codex) [§Footnotes / Terminal-Bench 2.1].
BrowseComp: 90.4 with 1M context, no compaction; matches Claude-card 300K compaction otherwise [§Footnotes / BrowseComp].
Kernel-optimization sandbox: competitive with Claude Fable 5 (with fallback) and substantially ahead of Opus 4.8 / GPT 5.6 Sol / GPT 5.5 across 4 tasks on H200 + a non-NVIDIA GPGPU [§Kernel Optimization].
Chip-design demo: 4 mm² @ 100 MHz, >8,700 tokens/s decode in simulation for a nano model matching K3’s architecture, in a single 48h autonomous run on open EDA + Nangate 45nm [§Chip Design].
Moonshot explicitly says overall UX still trails Claude Fable 5 and GPT 5.6 Sol [§Limitations].

Why it’s interesting

Kimi K3 is the first release where the individual Moonshot architecture papers filed on the wiki over the last months compose into a single production model, so it is a useful integration checkpoint. KDA (from Kimi Linear: An Expressive, Efficient Attention Architecture) is now scaled from a 48B/3B research MoE to a 2.8T frontier MoE inside the same lab, and AttnRes (Attention Residuals (AttnRes / Block AttnRes) — Technical Report) makes its first appearance in a flagship deployment — this is one of the first data points on how the “depth-selective retrieval” story survives at trillion-parameter scale under linear attention. On the routing side, Quantile Balancing (Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method) is now stress-tested at 16/896 sparsity, which is deeper into the sparsity regime than any of the papers already tracked under MoE Routing Design — a natural pairing with the SonicMoE (SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations) kernel-side story. Finally, K3 is the reference point against which the next round of open frontier releases tracked under Open foundation-model releases (LongCat-Flash-Omni, DeepSeek-V4, GLM-5.2, Trinity Large, MiniMax M3) will be measured on the agentic-SWE / long-horizon-coding axes captured in Agentic Software Engineering.