Skip to content
Bud's Wiki

GLM-5.3-Flash — 320B/18B natively multimodal hybrid sparse+linear attention MoE with mHC

TL;DR

Section titled “TL;DR”

GLM-5.3-Flash is Z.ai’s first natively multimodal model in the GLM-5 series: a 320B-total / 18B-active MoE trained from scratch on a 30T-token multimodal corpus, positioned as a one-tenth-of-GLM-5.2-price flagship that approaches Claude Opus 4.8 on coding and agentic benchmarks. The architectural news is a hybrid attention stack combining sparse and linear attention (a first for the GLM series, targeted at long-context serving cost) paired with Manifold-Constrained Hyper-Connections (mHC) to widen the residual stream as an independent scaling axis. It ships as an open-weights HF release with SGLang / vLLM / TokenSpeed / KTransformers deployment recipes and points at the GLM-5 tech report (GLM-5: from Vibe Coding to Agentic Engineering) for the underlying training methodology.

Key claims

Section titled “Key claims”
  • Total 320B parameters with 18B active per token, positioned as outperforming GLM-5.2 across benchmarks while serving at one-tenth the price [§Introduction].
  • Starts from a newly trained base model rather than continued-pretraining on top of GLM-5.2; the pretraining corpus is 30T multimodal tokens [§Introduction].
  • The attention stack is a first-for-GLM-series hybrid combining sparse attention and linear attention, motivated by cutting long-context serving cost while preserving precise long-context capability [§Introduction].
  • Residual-stream architecture uses Manifold-Constrained Hyper-Connections (mHC), following DeepSeek’s mHC: Manifold-Constrained Hyper-Connections recipe of Birkhoff-polytope-constrained mixing matrices [§Introduction].
  • Evaluations on HLE-with-tools use a 300K context management strategy with temperature=1.0, top_p=0.95, and max_new_tokens=163,840, judged by GPT-5.6-luna (medium) [§Footnotes].
  • Agentic benchmarks include DeepSWE (mini-swe-agent harness, 400K context, 6h timeout), Terminal-Bench 2.1 (Claude Code 2.1.207 harness with 6h timeout), Toolathlon Verified (pass@1 averaged over 3 runs), and AutomationBench v1.0.6 [§Footnotes].
  • BabyVision multimodal eval resizes inputs so the shorter side is ≥1.5K pixels, run at 164K max context [§Footnotes].
  • Deployment shipped with SGLang, vLLM, TokenSpeed, and KTransformers recipes as of release [§Serve GLM-5.3-Flash Locally].

Method

Section titled “Method”

The model card itself is a deployment page rather than a technical report, so architectural detail is compressed to a paragraph, but the specific claims are load-bearing. Three levers are named: (1) a hybrid sparse + linear attention stack, the first time the GLM series has adopted the 2025-2026 hybrid-attention pattern documented in Hybrid Linear Attention (Kimi Linear, MiniMax-M1, Qwen3-Next, Jet-Nemotron) — motivated by long-context serving cost, which GLM-5’s DeepSeek Sparse Attention (GLM-5: from Vibe Coding to Agentic Engineering) already targeted from the sparse side; (2) Manifold-Constrained Hyper-Connections (mHC), the Birkhoff-polytope residual-stream widening primitive from mHC: Manifold-Constrained Hyper-Connections — first filed frontier-scale deployment of that primitive outside DeepSeek’s own reports; (3) a 30T-token multimodal pretraining corpus, up from GLM-5’s 28.5T text-only corpus. Post-training and RL details are deferred to the GLM-5 tech report and the accompanying Z.ai blog post; no numbers are on the HF card.

Results

Section titled “Results”

The HF card claims GLM-5.3-Flash outperforms GLM-5.2 across benchmarks and real-world workloads at one-tenth the price, and approaches Claude Opus 4.8 on coding and agentic benchmarks — headline claim, no per-benchmark table on the card itself (the Z.ai blog and forthcoming tech report host the numbers). The specific benchmarks named in the footnote block are HLE-with-tools (full set), NL2Repo, DeepSWE (400K context), Terminal-Bench 2.1, Agent’s Last Exam, Toolathlon Verified, AutomationBench v1.0.6, GDPval-AA v2, and BabyVision — a coding/agentic + multimodal battery consistent with the “GLM-5’s agentic flagship extended to vision” positioning. Serving-cost comparison to GLM-5.2 (~10× cheaper) implies the hybrid sparse+linear stack materially cuts prefill+decode cost at 18B active vs. GLM-5.2’s 40B active, but the card does not break out the attention-vs-MoE cost split.

Why it’s interesting

Section titled “Why it’s interesting”

This is the first filed frontier open-weights release that combines both the hybrid sparse+linear attention pattern (the Hybrid Linear Attention cluster’s central prescription) and the mHC residual-stream widening primitive from mHC: Manifold-Constrained Hyper-Connections — a datapoint that DeepSeek’s V4 bet on mHC is being read and adopted by peer labs, not just talked about. It also completes the GLM-5 → GLM-5.1 → GLM-5V-Turbo → GLM-5.2 → GLM-5.3-Flash cadence with the family’s first native multimodal model (contrast with GLM-5.2-Vision-NVFP4 — bolting MoonViT onto frozen GLM-5.2 with a 49.5M projector, a frozen-tower fine-tune bolting MoonViT onto GLM-5.2). Sits in the same 2026 open-weights MoE frontier cohort as Kimi K3 (2.8T total, Kimi K3 Technical Report — Open Frontier Intelligence), Qwen3.8 (2.4T-A95B, Qwen3.8 open weights release — 27B dense multimodal + 2.4T-A95B MoE (Alibaba Qwen)), and Nemotron 3 Ultra (550B-A55B), but at a much lower active-parameter fraction (18/320 ≈ 5.6%) — the closest active-fraction neighbor is Kimi K3’s ultra-sparse 16-of-896 recipe. The one-tenth-price-vs-GLM-5.2 headline, if it holds, is the first concrete production evidence that hybrid linear attention pays off on the cost axis at frontier scale, not just the throughput axis.

See also

Section titled “See also”