GLM-5: from Vibe Coding to Agentic Engineering

TL;DR

GLM-5 is Z.ai’s next-generation open-weights agentic-engineering foundation model: a 744B-total / 40B-active MoE scaled up from GLM-4.5’s 355B/32B, trained on 28.5T tokens, with DeepSeek Sparse Attention (DSA) introduced via continued pre-training from a dense MLA base to roughly halve long-context inference cost without quality loss. Post-training is a sequential RL pipeline (Reasoning RL → Agentic RL → General RL) glued together by On-Policy Cross-Stage Distillation to prevent forgetting, running on a new asynchronous slime-based RL infrastructure with novel asynchronous agent-RL algorithms. Headline results: state-of-the-art among open-weights on Artificial Analysis Intelligence Index v4.0 (50, first open model at this level, +8 over GLM-4.7), #1 open on LMArena Text and Code, comparable to Claude Opus 4.5 / GPT-5.2 (xhigh) and better than Gemini 3 Pro on an 8-benchmark average, and #1 open on Vending-Bench 2 with a $4,432 final balance over a year-long simulated business. Released MIT-licensed with full-stack adaptation to seven Chinese domestic GPU platforms (Huawei Ascend, Moore Threads, Hygon, Cambricon, Kunlunxin, MetaX, Enflame) from day 0.

Key claims

GLM-5 is a 744B-total / 40B-active MoE with 256 experts and 80 layers (reduced layer count to minimize expert-parallel communication overhead), doubling GLM-4.5’s 355B/32B size [§2.1].
DeepSeek Sparse Attention (DSA) is adopted via continued pre-training from a dense MLA base: a 1000-step warmup at 14×202,752-token sequences with LR 5e-3, followed by a 20B-token sparse-adaptation stage that matches the original MLA model in SFT loss and benchmark evaluation — 47× cheaper than DeepSeek-V3.2’s 943.7B-token DSA conversion [§2.1.1, Table 3].
DSA is framed as lossless by construction — its lightning indexer achieves token-level sparsity without discarding long-range dependencies, applied across all layers with no quality degradation; ablation against SWA-Interleave, search-based SWA-Pattern, GDN, and SimpleGDN shows DSA avoids the 5.69-point RULER@128K drop the alternative efficient-attention methods incur [§2.1.2, Tables 4–6].
Muon Split: matrix orthogonalization is applied per-attention-head rather than per up-projection matrix, lifting MLA performance to match GQA-8 under the Muon optimizer and stabilizing attention-logit scale during pre-training without clipping [§2.1, Table 1].
MLA-256: head dimension increased from 192 to 256 with attention-head count reduced by 1/3, keeping training computation and parameter count constant while cutting decoding computation — recalibrating MLA from H800’s roofline to GLM-5’s target hardware [§2.1].
3-MTP-layer parameter sharing: shared parameters across 3 Multi-Token Prediction layers during training (vs DeepSeek-V3’s single shared layer predicting 2 tokens), giving 2.76 accept length at 4 speculative steps vs DeepSeek-V3.2’s 2.55 [§2.1, Table 2].
Context window is extended progressively: 32K (1T tokens) → 128K (500B tokens) → 200K (50B tokens) in mid-training; the additional 200K stage on top of 128K is reported to further improve performance even inside the 128K window [§2.3].
Software-engineering mid-training corpus is built by concatenating repo-level code files, commit diffs, GitHub issues, PRs, and relevant source files into unified sequences; ~10M issue–PR pairs after relaxed repo-level filtering + stricter per-issue quality filtering, yielding ~160B unique tokens [§2.3].
Post-training pipeline: multi-task SFT with interleaved thinking modes → Reasoning RL → Agentic RL → General RL → On-Policy Cross-Stage Distillation as final refinement to mitigate capability regression across stages [§3].
INT4 QAT in the SFT stage is used to provide low-precision accuracy; a custom quantization kernel ensures bitwise-identical behavior between training and inference [§2.4.3].
New asynchronous RL infrastructure decouples generation from training (extending the slime framework and GLM-4.5’s decoupled rollout engines) to maximize GPU utilization on long-horizon agent trajectories; paired with novel asynchronous agent-RL algorithms targeting planning and self-correction [§1, §3].
Headline result on Artificial Analysis Intelligence Index v4.0: 50, +8 over GLM-4.7’s 42 and the first open-weights model to reach 50, driven by agentic-performance and knowledge/hallucination gains [§1, Fig. 2].
Vending-Bench 2 (one-year simulated vending-machine business): GLM-5 ranks #1 among open-source with a $4,432 final account balance, approaching Claude Opus 4.5 [§1, Fig. 4 left].
CC-Bench-V2 (internal eval): GLM-5 significantly outperforms GLM-4.7 across frontend, backend, and long-horizon tasks, narrowing the gap with Claude Opus 4.5 [§1, Fig. 4 right].
On an 8-benchmark average over HLE, SWE-bench Verified, SWE-bench Multilingual, Terminal-Bench 2.0, BrowseComp, MCP-Atlas, τ²-Bench, Vending-Bench 2, GLM-5 averages ≈+20% over GLM-4.7 and is comparable to Claude Opus 4.5 / GPT-5.2 (xhigh), better than Gemini 3 Pro [§1, Fig. 1].
#1 open model on both LMArena Text and Code, overall on par with Claude Opus 4.5 and Gemini-3-pro [§1, Fig. 3].
Day-0 full-stack adaptation across seven domestic Chinese GPU ecosystems (Huawei Ascend, Moore Threads, Hygon, Cambricon, Kunlunxin, MetaX, Enflame) with deep optimization from kernels to inference frameworks [§1].
Multi-backend serving on release: vLLM, SGLang, xLLM, and Ktransformers all support local deployment of the GLM-5 series [GitHub README].

Method

GLM-5’s architecture starts from GLM-4.5’s MoE backbone and re-engineers the attention and MoE layouts for the 744B / 40B-active scale. The choice between GQA-8 and MLA is settled by Muon Split — orthogonalizing per-head rather than per-projection-matrix — which makes MLA match GQA-8 on six pre-training benchmarks while letting the Muon optimizer train without attention-logit clipping. To make MLA cheaper at decode time, the head dimension is raised from 192 to 256 and head count is reduced by 1/3 (MLA-256), keeping training cost flat while halving decoding cost. The MTP module uses 3 parameter-shared layers during training, matching DeepSeek-V3’s draft-model memory footprint while improving the per-token accept rate. Pre-training totals 28.5T tokens across web, code (with a 28% larger deduplicated unique-token pool and per-language classifiers), and curated math/science (LLM-scored, with chunk-and-aggregate scoring on long docs). Mid-training stretches context from 32K → 128K → 200K, with ~160B software-engineering tokens from issue–PR concatenation.

After mid-training, the model converts to DeepSeek Sparse Attention via a two-stage continued-pre-training recipe — 1000-step warmup at 5e-3 LR on long sequences, then 20B tokens of sparse-adaptation training that follows mid-training hyperparameters. The DSA model matches the MLA model in SFT loss and downstream evals at ~47× lower DSA-conversion budget than DeepSeek-V3.2, and avoids the 5.69-point RULER@128K hit that SWA-Interleave / SWA-Pattern / GDN / SimpleGDN incur in a controlled GLM-9B ablation. DSA reduces long-sequence attention compute by 1.5–2× and is the load-bearing piece that makes 128K-context agent rollouts affordable at this parameter scale.

Post-training takes the base model through multi-task SFT (with interleaved thinking modes and significantly expanded agent + coding corpora vs GLM-4.5) and then a sequential RL pipeline: Reasoning RL → Agentic RL → General RL, with On-Policy Cross-Stage Distillation as the connective tissue to prevent capability regression as later stages overwrite earlier ones. The RL infrastructure extends slime + decoupled rollout engines from GLM-4.5 to decouple generation from training fully, enabling massive-scale exploration of long-horizon agent trajectories without synchronization bottlenecks. Novel asynchronous agent-RL algorithms target planning and self-correction quality in dynamic environments — the paper frames these as the direct cause of the real-world coding gains. INT4 QAT in the SFT stage with a bitwise-identical train-vs-inference kernel produces a low-precision-ready checkpoint without a post-hoc quantization step.

Training-infrastructure contributions include flexible MTP placement under interleaved pipeline parallelism, Pipeline ZeRO2 gradient sharding with double-buffered accumulation, zero-redundant Muon-distributed-optimizer communication (only sharded all-gathers, overlapped with compute), pipeline activation offloading + fine-grained recomputation, and sequence-chunked output projection that bounds peak loss-layer memory. Long-sequence training uses workload-aware sequence reordering, dynamic context-parallel partitioning, and hierarchical intra/inter-node all-to-all overlap for QKV.

Results

Intelligence Index v4.0: 50 (first open-weights model at this level, +8 over GLM-4.7) [§1, Fig. 2].
LMArena: #1 open on both Text and Code, on par overall with Claude Opus 4.5 and Gemini-3-pro [§1, Fig. 3].
8-benchmark average (HLE, SWE-bench Verified, SWE-bench Multilingual, Terminal-Bench 2.0, BrowseComp, MCP-Atlas, τ²-Bench, Vending-Bench 2): ≈+20% over GLM-4.7, comparable to Claude Opus 4.5 / GPT-5.2 (xhigh), better than Gemini 3 Pro [§1, Fig. 1].
Vending-Bench 2: $4,432 final balance, #1 open-source, approaching Claude Opus 4.5 [§1, Fig. 4 left].
CC-Bench-V2 (internal): significant improvement over GLM-4.7 across frontend/backend/long-horizon, narrowing the gap to Claude Opus 4.5 [§1, Fig. 4 right].
DSA conversion: 20B-token sparse-adaptation stage (vs DeepSeek-V3.2’s 943.7B) matches MLA-baseline SFT loss and benchmarks [§2.1.1, Table 3].
MTP: accept length 2.76 at 4 speculative steps, vs DeepSeek-V3.2’s 2.55 on the same prompt set [§2.1, Table 2].
Efficient-attention ablation (GLM-9B baseline, 128K context): SWA-Interleave catastrophically collapses (RULER@128K 44.93), SWA-Pattern preserves most performance (69.59), GDN/SimpleGDN intermediate; DSA on GLM-4.7-Flash closes nearly the entire gap (RULER@128K 78.86 vs 79.21 baseline) [§2.1.2, Tables 4–6].
Note: Per the public release artifacts, GLM-5 also reports SWE-Bench Verified 77.8% and a 200K input / 128K output context window; quantitative SWE-Bench-Pro / Terminal-Bench 2.0 numbers are reported as figures in the technical report and were not extracted verbatim into this filing.

Why it’s interesting

GLM-5 is the first filed example of DSA being adopted by a non-DeepSeek frontier model via a deliberately cheap continued-pre-training conversion from MLA — 20B tokens vs DeepSeek-V3.2’s ~944B. That’s a strong commoditization signal for sparse attention as a primitive, and it directly answers a structural question the wiki had open: whether DSA is a DeepSeek-specific trick or a portable architectural piece. The MLA → DSA conversion recipe (warmup the indexer only → joint train) is small enough to be reusable for other frontier MoE backbones.

Beyond DSA, the model lands squarely on the low-active-MoE agentic-SWE frontier tracked in Agentic Software Engineering: alongside Qwen3-Coder-Next (3B-active / 80B-total), DeepSeek-V3.2 (37B-active), GLM-4.7 (32B-active), MiniMax M2.1 (10B-active), Step 3.5 Flash (11B-active / 196B-total), and Kimi K2.5, GLM-5 sits at 40B-active / 744B-total — the largest total in the cluster but a moderate active ratio. The sequential Reasoning RL → Agentic RL → General RL pipeline glued by On-Policy Cross-Stage Distillation is a new entry in the Reasoning RL design space: a forgetting-prevention mechanism specifically tuned to the multi-stage RL composition that Step 3.5 Flash (MIS-PO) and Kimi K2.5: Visual Agentic Intelligence (PARL) both also navigate, but from different angles.

The release-packaging story is also notable: MIT-licensed, full-stack adaptation to seven domestic Chinese GPU ecosystems on day 0, multi-backend serving (vLLM / SGLang / xLLM / Ktransformers) — extending the open-foundation-release packaging convention in a hardware direction (cf. Open foundation-model releases) that no previously-filed open model has covered at this depth.