GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning

TL;DR

The GLM-V series tech report from Z.ai documents two open-weights vision-language models with explicit Thinking Mode: GLM-4.1V-9B-Thinking (built on GLM-4-9B-0414) and GLM-4.5V (built on GLM-4.5-Air, 106B-total / 12B-active MoE). The architecture pairs an AIMv2-Huge vision encoder with the GLM LLM via an MLP adapter, swapping the encoder’s 2D conv for 3D conv to enable 2× temporal downsampling on video, adding 2D-RoPE plus bicubic-interpolated absolute position embeddings to support arbitrary aspect ratios (>200:1) and resolutions (>4K), and extending RoPE to 3D-RoPE on the LLM side. The training recipe is reasoning-centric: large interleaved-image-text + OCR + grounding + video pretraining → cold-start SFT on standardized <think>...</think><answer>...</answer> long-CoT traces → multi-domain reinforcement learning with the proposed Reinforcement Learning with Curriculum Sampling (RLCS), which combines curriculum learning with difficulty-aware rollout selection. GLM-4.1V-9B-Thinking matches or exceeds Qwen2.5-VL-72B on 18 of 28 benchmarks despite being 8× smaller, and is competitive with GPT-4o on long-document and STEM tasks.

Key claims

Architecture: AIMv2-Huge vision encoder + MLP adapter + GLM LLM decoder, with 2D→3D conv in the encoder for 2× temporal video downsampling, 2D-RoPE in ViT, bicubic-interpolated absolute position embeddings for arbitrary resolutions/aspect ratios, and 3D-RoPE in the LLM [§2].
Video time-index tokens (each frame’s timestamp encoded as a string token inserted after frame tokens) explicitly inform the model of inter-frame temporal distances, distinguishing video from multi-image input [§2].
Pretraining pool starts from >10B image-text pairs (LAION, DataComp, DFN, Wukong + web search) and refines through heuristic filtering, CLIP-Score ≥0.3 relevance filtering, MetaCLIP-style concept-balanced resampling, and an iteratively-trained factual-centered recaptioning model [§3.1].
OCR pretraining dataset is 220M images across three sources: synthetic document images (text rendered over LAION backgrounds), Paddle-OCR-processed natural scene text, and Nougat-style PDF↔HTML academic document pairs from arXiv [§3.1].
Grounding pretraining: 40M natural-image annotations from LAION-115M auto-labeled with GLIPv2 (≥2 boxes per image required); plus a from-scratch GUI grounding dataset built by Playwright-crawled CommonCrawl webpages with DOM-element bounding boxes, totaling 140M Referring Expression Generation/Comprehension QA pairs [§3.1].
Training is two-stage: 120K steps multimodal pretraining (8,192 seq len, 1,536 batch, 2-way TP) then 10K steps long-context continual training (32,768 seq len, 2-way TP + 4-way context parallelism) — sequence length and parallelism are increased only in the second stage to amortize compute [§3.2].
SFT explicitly uses long-CoT data only (no short-CoT step), framed as alignment rather than knowledge injection — its role is to prime the cold start for stable RL rather than to teach new content [§4].
Response format standardizes <think>{reasoning}</think><answer>{answer}</answer> with optional <|begin_of_box|>...<|end_of_box|> around the final answer; all six tags are added to the tokenizer vocabulary as special tokens to enable accurate online parsing during RL [§4.1].
Iterative SFT enhancement: high-quality reasoning traces sampled from RL checkpoints are merged back into the cold-start SFT dataset to expose the next round to harder reasoning patterns [§4.1].
RL combines RLVR (Reinforcement Learning with Verifiable Rewards) and RLHF across STEM, grounding, OCR, video understanding, GUI agents, chart/document understanding, logical reasoning, and instruction following — a single unified reward system with shared evaluation logic and subdomain-specific verifiers [§5].
Reinforcement Learning with Curriculum Sampling (RLCS) selects rollout problems based on a fine-grained offline difficulty grade (from pass@k evaluations across multiple existing/prior RL models combined with human labels) [§5.1, §5.3].
Dynamic sampling expansion via ratio-based Exponential Moving Average (EMA) is required to keep RL informative — the proportion of “informative” rollouts is tracked with EMA and the sampling budget scales accordingly [§5.3].
Reward-system precision is critical: when training a unified VLM across diverse skills, even a slight weakness in the reward signal for one capability can collapse the entire RL process [§5.2, “Key findings”].
Multi-domain RL shows cross-domain generalization and mutual facilitation — training on one domain improves performance on others, and joint training across domains yields greater improvement in each domain than per-domain training alone [§6.3].
GLM-4.1V-9B-Thinking outperforms Qwen2.5-VL-7B on nearly all 28 benchmarks and matches/exceeds Qwen2.5-VL-72B on 18 benchmarks [Abstract, §1].
Specific frontier results for GLM-4.1V-9B-Thinking: MMBenchV11-EN 85.8, MMStar 72.9, AI2D 87.9, MMMU-Pro +6.0 over Qwen2.5-VL-72B, ChartMuseum +9.2 over Qwen2.5-VL-72B, long-document understanding 42.4 vs Qwen2.5-VL-72B’s 35.2, WebQuest-SingleQA 72.1 vs 60.5, Flame-VLM-Code 72.5 vs 46.3 [§1].
Closed-frontier comparison: GLM-4.1V-9B-Thinking exceeds GPT-4o on MMStar (72.9 vs 66.2), MUIRBENCH (74.7 vs 69.7), and MathVista (80.7 vs 64.0) [§1].
GLM-4.5V scales the same recipe to the GLM-4.5-Air base (106B / 12B active) and claims SOTA on 42 public vision-language benchmarks at its scale, with grounding via normalized [0,1000] bounding boxes wrapped in <|begin_of_box|>...<|end_of_box|> [HF model card; arXiv abstract revision].

Method

The pipeline has four stages — pretraining, long-context continual training, supervised fine-tuning, and reinforcement learning — and the framing throughout is that pretraining sets the upper bound on post-RL performance, so the pretraining mix and the SFT cold-start data quality together determine how high RL can climb. Pretraining mixes seven data types (image caption, interleaved image-text from web/books, OCR, grounding on natural images and GUIs, video, instruction tuning, plus pure text) with explicit per-source pipelines. The interleaved image-text pipeline is notable: web data (MINT + MMC4 + OmniCorpus) is filtered by CLIP-Score image-context relevance, ad/QR-code removal via a purpose-trained classifier, and a “high-knowledge-density” classifier iteratively trained to prioritize scientific illustrations / engineering schematics / instructional diagrams over decorative imagery. Academic books (>100M digitized) are filtered to STEM and deeply parsed for interleaved image-text segments. The 50M instruction-tuning set is organized by a fine-grained semantic taxonomy with balanced sampling and an explicit data-contamination check against public benchmarks.

SFT then aligns the pretrained base with the long-CoT format on a curated reasoning dataset, with strict format enforcement (think/answer tags, single boxed final answer, no language mixing, no redundant thought patterns). Crucially, SFT is positioned as a style alignment step rather than knowledge injection — the authors find that even noisy long-CoT data leads to successful RL, but clean cold-start data gives more stable convergence. They iteratively re-mix high-quality RL-discovered traces back into the SFT dataset across rounds, treating SFT and RL as a coupled loop rather than a one-shot pipeline. RL itself spans all multimodal subdomains under a single unified reward system, uses RLCS (curriculum + difficulty-aware sampling) plus ratio-EMA dynamic sampling expansion to keep rollouts informative, drops the KL and entropy losses, uses larger batch sizes, and runs at the full target context length rather than progressive-length stages. Three “key findings” the report highlights: cross-domain mutual facilitation in RL (§6.3), the necessity of informative-rollout selection (§5.3), and the brittleness of multi-domain RL to even small reward-signal weaknesses (§5.2).

Results

28-benchmark evaluation: GLM-4.1V-9B-Thinking outperforms Qwen2.5-VL-7B on nearly all tasks, leads on 23 of 28 at the 10B scale, and matches/exceeds Qwen2.5-VL-72B on 18 tasks despite being 8× smaller [§1, Abstract].
MMBenchV11-EN 85.8 / MMStar 72.9 / AI2D 87.9 — exceeds both Qwen2.5-VL-72B and other baselines at 10B scale [§1].
MMMU-Pro: +6.0 points and ChartMuseum +9.2 points over Qwen2.5-VL-72B [§1].
Long-document understanding: 42.4 vs Qwen2.5-VL-72B’s 35.2 [§1].
WebQuest-SingleQA (multimodal agent): 72.1 vs Qwen2.5-VL-72B’s 60.5 [§1].
Flame-VLM-Code (coding): 72.5 vs Qwen2.5-VL-72B’s 46.3 [§1].
Closed-frontier head-to-head: MMStar 72.9 vs GPT-4o 66.2; MUIRBENCH 74.7 vs 69.7; MathVista 80.7 vs 64.0 [§1].
GLM-4.1V-9B-Base achieves significantly better pass@k than state-of-the-art base models of similar scale on the same benchmarks, supporting the “base bounds RL” framing [§3, Fig. 3].
GLM-4.5V (the 2025-08-11 release, built on GLM-4.5-Air at 106B / 12B active) claims SOTA on 42 public vision-language benchmarks among models of comparable scale [HF model card].

Why it’s interesting

This is the vision-language counterpart to GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models — same Z.ai team, same reasoning-centric RLVR + RLHF design, same <think>/<answer> format, same “single-stage RL at the full target length” choice. The two papers together specify the GLM family’s full multimodal training recipe at frontier-open scale, and GLM-4.5V explicitly inherits the GLM-4.5-Air backbone — so this paper is the missing link explaining how the vision tower, video time-index tokens, 3D-RoPE adaptation, and grounding output format ride on top of the LLM tech report’s findings. For Reasoning RL, RLCS (curriculum sampling + ratio-EMA dynamic-sampling expansion) is a sharper specification than the page’s current GRPO-default characterization — it’s GRPO-family, but the sample-selection layer is non-trivial and the report makes a specific claim that informative-rollout selection (not just gradient updates) is what determines RL efficiency. For Thinking with Modalities, this is the first filed page where the substrate is a tagged VLM-native CoT (<grounding>/<caption>-style tags as in Video-Thinker) but trained at 9B–106B scale with explicit cross-domain RL — the closest comparison is Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning, which uses the same tagged-CoT substrate at 7B but only for video reasoning, while GLM-4.5V applies it across image / video / document / GUI / grounding simultaneously. For Open foundation-model releases, this fits the “single-model-series + multi-backend serving + joint tech report” pattern: HF model cards for both 9B and 106B variants, day-0 Transformers / vLLM / SGLang snippets, a single arXiv report covering both series, a desktop assistant reference app, and an open-sourced VLM reward system.