GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models

TL;DR

The GLM-4.5 tech report from Z.ai, the underlying paper behind the model card / blog already filed at GLM-4.5: Agentic Foundation Model from Z.ai. Documents an open-weights 355B-total / 32B-active MoE (and a 106B / 12B Air variant) trained on 23T tokens via a three-phase pipeline — base pretraining, agentic / reasoning mid-training, and a two-stage post-training where domain-expert models are first trained separately with RL and then self-distilled into a single hybrid-reasoning model. Architectural unusual choices: 96 attention heads at hidden 5120 (“deeper-narrower” with more heads improves reasoning benchmarks without lowering training loss), QK-Norm, loss-free balance routing with sigmoid gates, and Muon for everything except embeddings / norms / biases. Single-stage RL at the full 64K-token target length, dynamic sampling temperature, and an iterative self-distillation loop on agentic tasks are the post-training centerpieces.

Key claims

355B total / 32B active MoE; 106B / 12B for Air. 89 MoE layers + 3 dense + 1 MTP (Air: 45 + 1 + 1). Hidden 5120, 96 attention heads, 8 KV heads, 160 routed experts + 1 shared, 8 active per token [Table in §2.1].
“Deeper-narrower with more attention heads” is the architectural bet — 96 heads at hidden 5120 (2.5× the head count of comparable models). The increased head count does not improve training loss versus a fewer-heads baseline but does improve MMLU and BBH reasoning scores [§2.1].
QK-Norm is added to stabilize the range of attention logits [§2.1].
Routing uses loss-free balance routing with sigmoid gates; auxiliary sequence-level balance loss carries a 0.0001 weight. The loss-free bias update rate is 0.001 for the first 15T tokens and 0.0 afterward [§2.1, §2.4].
Muon is the default optimizer for all parameters except embeddings, biases, and RMSNorm weights; 5 Newton–Schulz steps, momentum 0.95, update RMS scaled to 0.2. Authors report Muon both speeds convergence and tolerates larger batch sizes [§2.4].
WSD schedule underfits the stable stage versus cosine decay on SimpleQA / MMLU in their early experiments — cosine wins [§2.4].
“Mid-training” is positioned as a distinct phase between base pretraining and post-training, with three sub-stages: (a) repo-level code training with cross-file context extended from 4K → 32K, (b) synthetic reasoning data for math / science / coding competitions, (c) long-context + agent training extending 32K → 128K with synthetic agent trajectories [§2.3].
Post-training is expert iteration + unified training: first train three specialist models (Reasoning, Agent, General chat), then self-distill them into one hybrid-reasoning model that supports both thinking and direct-response modes via an enable_thinking flag [§3].
Function-call template uses XML-like special tokens to avoid JSON character escaping inside code-segment parameters — substantially reduces the escaping burden for agentic SFT data without degrading function-call performance [§3.1].
Single-stage RL at the full 64K output length beats the prior multi-stage progressive-length recipe — stage-wise shorter-length RL causes the model to “unlearn” long-context capability and never fully recovers in the final 64K stage [§3.2].
Dynamic sampling temperature, raised when rollout reward converges and capped by a held-out validation drop of ≤1%, replaces the standard fixed-temperature recipe [§3.2].
Code RL: token-weighted mean loss (not sequence-mean) is critical — converges faster, suppresses repetitive “base case” samples [§3.2, Fig. 7 left].
Science RL: expert-verified multiple-choice questions only — data quality dominates volume on GPQA-Diamond [§3.2, Fig. 7 right].
Agentic RL drops the KL term and uses outcome supervision with a process-format penalty: zero reward for trajectories with malformed tool calls, otherwise final-answer accuracy [§3.3.2].
Iterative self-distillation on agentic tasks: train RL until plateau, replace the SFT cold-start data with RL-trained-model rollouts, re-run SFT, then resume RL — pushes performance limits of agent training within fixed compute [§3.3.2].
Test-time scaling on agents is via interaction turns with the environment (search depth, self-verification rounds), not via more output tokens — accuracy scales smoothly with browsing effort [§3.3.2, Fig. 8].
Headline benchmarks: 70.1% TAU-Bench, 91.0% AIME 24, 64.2% SWE-bench Verified; ranked 3rd overall and 2nd on agentic across 12 evaluated benchmarks despite having ~half DeepSeek-R1’s and ~one-third Kimi K2’s parameters [Abstract, §1, Fig. 1, Fig. 2].

Method

The architecture inverts the recent “wider-shallower” frontier-MoE trend: GLM-4.5 keeps hidden dim 5120 (smaller than DeepSeek-V3’s 7168 or Kimi K2’s 7168) but stretches to 89 MoE layers + 3 dense + 1 MTP, with 96 attention heads (vs DeepSeek-V3’s 128 at hidden 7168 and Kimi K2’s 64). The MTP layer is itself an MoE layer added for speculative decoding at inference. Routing is sigmoid-gated, loss-free-balanced (per the DeepSeek-V3 recipe) with an auxiliary sequence-level balance term at 1e-4. QK-Norm + GQA with partial RoPE makes up the attention block.

Training is split across three named phases — pretraining (4K context, web + code + math + multilingual, with Nemotron-CC-style quality bucketing and SemDedup), mid-training (32K → 128K, repo-level code + synthetic reasoning + agent trajectories, applying best-fit packing to avoid truncating long reasoning traces), and post-training. Total training corpus: 23T tokens. Muon optimizer for all non-norm/bias/embedding parameters, batch-size warmup from 16M → 64M tokens over the first 500B tokens, learning-rate warmup to 2.5e-4 and cosine decay to 2.5e-5, RoPE base frequency bumped from 10K → 1M when extending to 32K.

Post-training has two stages. Expert Training (Stage 1) trains three specialist models — Reasoning, Agent, General chat — each via cold-start SFT then RL. Unified Training (Stage 2) collects outputs from all three experts and SFT-distills them into a single base model with maximum context 128K, then runs a General RL phase that mixes rule-based, RLHF, and RLAIF rewards over a balanced 5K-prompt dataset spanning 7 primary × 33 secondary × 139 tertiary categories. The reasoning-RL recipe uses GRPO (no KL term), difficulty-based two-stage curriculum, single-stage 64K-length training, and dynamic temperature; the agentic-RL recipe uses outcome supervision + process-format penalty + iterative self-distillation, with environment-feedback tokens excluded from the loss.

Results

TAU-Bench (agentic) 70.1%; AIME 24 91.0%; SWE-bench Verified 64.2% [Abstract].
Average across 12 ARC benchmarks: GLM-4.5 ranks 3rd overall, 2nd on agentic, 3rd on coding (close to Claude Sonnet 4) [§1, Fig. 1].
GLM-4.5 and GLM-4.5-Air both sit on the SWE-bench-Verified-vs-parameters Pareto frontier among open-source models [Fig. 2].
The 96-head ablation shows MMLU and BBH improve with more heads while training loss is unchanged versus the fewer-heads baseline [§2.1, prose].
Single-stage 64K RL beats multi-stage progressive-length RL — multi-stage causes irreversible average-output-length drop [§3.2, Fig. 6].
Token-weighted code-RL loss substantially speeds convergence over sequence-mean [§3.2, Fig. 7 left].
Removing the bottom 50% of prompts by response length and applying 4× response scaling on the remaining hard prompts yields +2–4% on math/science with half the data + a further +1–2% [§3.1].
The paper reports parameter-efficiency framing: GLM-4.5 has ~half DeepSeek-R1’s params and ~one-third Kimi K2’s while ranking ahead on agentic [§1].

Why it’s interesting

The paper formalizes the “mid-training” stage as a first-class phase between pretraining and RL post-training — exactly the framing the Luma omni memo argues for, and a corroborating datapoint for the trend that frontier-open recipes increasingly load capability acquisition into pre-RL stages with longer, mixed-objective sequences rather than relying on RL alone (see also Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs and OpenThoughts: Data Recipes for Reasoning Models). The “deeper-narrower with more heads” architectural bet — same training loss, better reasoning — is a counterintuitive datapoint for MoE Routing Design alongside Margaret Li’s Slicing and Dicing MoEs — Margaret Li announcing a >2000-MoE-LM design-space study claim that expert size + count dominate; here the claim is that attention shape, not just expert shape, also moves the reasoning Pareto. The Muon-as-default + QK-Norm + loss-free balancing combination provides a frontier-scale validation of training-stability recipes that the wiki has only seen in smaller-scale ablations (QK-Clip: Taking Muon Further on the Scaleup Journey, Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models). For Reasoning RL, the single-stage 64K-length RL finding directly contradicts the multi-stage progressive-length recipe most prior open papers ship, and the no-KL-term + outcome-supervision + process-format-penalty + iterative-self-distillation agentic-RL recipe is a tight, replicable specification — closer to the MiroThinker pipeline (MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling) than to DanceGRPO. This page is the canonical training-recipe artifact for the GLM-4.5 series; the existing GLM-4.5: Agentic Foundation Model from Z.ai page captures the model-card / release-shape side.