Skip to content
Bud's Wiki

Kimi K2: Open Agentic Intelligence

TL;DR

Section titled “TL;DR”

The full Kimi K2 technical report — a 1.04T-parameter / 32B-active Mixture-of-Experts open-weight LLM pre-trained on 15.5T tokens with the MuonClip optimizer (Muon + QK-Clip), reporting zero loss spikes across the full run. K2’s pre-training thesis is that token-efficiency is the binding constraint, addressed via a Muon-class optimizer plus domain-specialized synthetic rephrasing pipelines for knowledge and math. Post-training is agentic-first: large-scale synthetic tool-use trajectory synthesis followed by a joint RL stage over real and synthetic environments. Headline numbers: 65.8 SWE-Bench Verified pass@1 (71.6 with parallel TTC), 47.3 SWE-Bench Multilingual, 66.1 Tau2-Bench, 76.5 ACEBench-En, positioning K2 as SOTA among open-source non-thinking models on coding and agentic axes at release.

Key claims

Section titled “Key claims”
  • MuonClip = Muon + QK-Clip: addresses Muon’s known large-scale instability (unbounded QK-projection logit growth) by clipping the Muon updates on QK projections, enabling a 15.5T-token pre-training of a 1T MoE with zero loss spikes [§Pre-training].
  • Architecture follows DeepSeek-V3: ultra-sparse MoE with multi-head latent attention (MLA), hidden dim 7168, MoE expert hidden dim 2048, 384 experts at 1T total / 32B active per token [§2, Fig. comparing K2 vs DeepSeek-V3].
  • Token-efficiency thesis: given limited high-quality text supply, the scaling axis the team optimizes is intelligence-per-token rather than raw token count — motivating both the Muon-family optimizer choice and the synthetic-rephrasing pipeline [§Pre-training].
  • Synthetic data generation via domain-specialized rephrasing is used to amplify high-quality tokens without inducing overfitting; two pipelines are detailed, targeting the Knowledge and Mathematics domains [§Pre-training, rephrasing strategy].
  • Post-training is multi-stage agentic-first: (1) a large-scale synthetic tool-use trajectory generation pipeline producing structured multi-step reasoning + tool-invocation data, and (2) a joint RL stage over real and synthetic environments incorporating preferences and self-critique [§Post-training].
  • SWE-Bench Verified 65.8 pass@1 with bash/editor tools, single-attempt patches, no test-time compute, 8K output budget (16K for Agentless); 71.6 with parallel TTC sampling and an internal scoring model [§4, Table 3].
  • SWE-Bench Multilingual 47.3 pass@1 under the same single-attempt, no-TTC conditions [§4, Table 3].
  • Tau2-Bench 66.1, ACEBench-En 76.5 — SOTA among open non-thinking models at release on the tool-use / function-calling axis [§4, Table 3].
  • Math / knowledge despite non-thinking setting: AIME 2025 49.5, GPQA-Diamond 75.1, OJBench 27.1, LiveCodeBench v6 53.7; evaluation uses avg@k on AIME / HMMT / CNMO / PolyMath-en / GPQA-Diamond / EvalPlus / Tau2 to stabilize variance [§4, Table 3].
  • Two checkpoints: Kimi-K2-Base (foundation) and Kimi-K2-Instruct (post-trained), both released open-weight on Hugging Face under the MIT-derived modified license [§5, Release].

Method

Section titled “Method”

K2 is a transformer MoE with 1.04T total parameters / 32B activated / 384 experts, architecturally close to DeepSeek-V3 (MLA, hidden 7168, expert hidden 2048). The training-recipe contributions of the tech report live in pre-training and post-training.

MuonClip optimizer (pre-training). Muon (Newton-Schulz-orthogonalized momentum updates on 2D weights) is the base. Muon’s known failure mode at large scale is unbounded growth of attention-logit magnitudes through QK projections, manifesting as loss spikes. K2 introduces QK-Clip, a stability mechanism applied specifically to Muon updates of the QK projections, that bounds the per-step update magnitude in a way that does not break Muon’s orthogonalization guarantees. The result is the headline robustness datapoint: 15.5T tokens, 1T MoE, zero spikes.

Synthetic data rephrasing (pre-training). Two domain-specialized pipelines — Knowledge and Mathematics — rephrase high-quality source documents to amplify their effective token count. The framing positions multi-epoch repetition as suffering from diminishing returns and overfitting, while a single epoch over original-only data leaves the available knowledge under-absorbed; rephrasing is the middle path.

Agentic post-training stack. Two stages: (1) Large-scale agentic data synthesis — structured tool-use trajectories that include multi-step reasoning, long-horizon planning, and tool invocation; the synthesizer is itself an LLM-in-the-loop pipeline with environment simulators. (2) Joint RL over real and synthetic environments, combining preference signals and self-critique to refine the resulting policy. The two-stage decomposition (synthetic SFT trajectories → RL refinement) is the recipe template that K2.5 and K2.6 continue building on.

What is out of scope. Multimodal input, thinking-mode (explicit reasoning chains over hidden CoT), and the Agent Swarm orchestration. K2 is the text-only non-thinking foundation that subsequent K2.5 (multimodal continual pre-training) and K2.6 (agent swarm + improved coding) scale on.

Results

Section titled “Results”

Non-thinking-cohort comparison (Table 3, “Bold denotes global SOTA, underlined bold = best open-source”):

  • Agentic / tool use: Tau2-Bench 66.1, ACEBench-En 76.5 — open-source SOTA at release.
  • Coding: SWE-Bench Verified 65.8 single-attempt → 71.6 with parallel TTC + internal scoring; SWE-Bench Multilingual 47.3; LiveCodeBench v6 53.7.
  • Math: AIME 2025 49.5 (avg@k), OJBench 27.1.
  • Knowledge / reasoning: GPQA-Diamond 75.1.

All metrics except SWE-Bench Verified (Agentless) use an 8K output budget; Agentless uses 16K. Stability is established via avg@k on the volatility-prone benchmarks (AIME / HMMT / CNMO / PolyMath-en / GPQA-Diamond / EvalPlus / Tau2). The headline four-number summary the paper leans on is Tau2-Bench 66.1, ACEBench-En 76.5, SWE-Bench Verified 65.8, SWE-Bench Multilingual 47.3.

Why it’s interesting

Section titled “Why it’s interesting”

This is the tech-report companion to the already-filed Kimi K2: Open Agentic Intelligence — same artifact lineage, deeper-content form, posted by a different team member with the GitHub-PDF link rather than the blog URL. It is the production-scale stability proof for the Training stability at scale cluster: MuonClip’s 15.5T-token / 1T-MoE zero-spike run is the strongest known empirical datapoint that Muon-family optimizers work end-to-end at frontier scale, and is the deployment that Addition of Muon optimizer to torch.optim — PyTorch team welcomes PR (Issue #148819) cites as social proof for adding Muon to PyTorch core. For MoE Routing Design K2 sits at the upper-scale end of filed open-MoE releases (1T total / 32B active / 384 experts) — comparable to DeepSeek-V4 collection release (Flash + Pro, up to 1.6T) (1.6T) and ERNIE 4.5 Technical Report (300B-A47B). The agentic-first post-training framing (synthetic tool-use trajectories → joint RL over real + synthetic environments) is the recipe that Kimi K2.6: Advancing Open-Source Coding continues scaling, and the SWE-Bench Verified 65.8 / 71.6 numbers are the baseline K2.6’s 80.2 builds off.

See also

Section titled “See also”