GLM-5.2-Vision-NVFP4 — bolting MoonViT onto frozen GLM-5.2 with a 49.5M projector

TL;DR

Baseten released a community fine-tune that gives Z.ai’s text-only GLM-5.2 reasoning model image inputs without touching a single GLM weight. The recipe freezes both towers — GLM-5.2 (744B total / 40B active) as the text backbone and the MoonViT-3d vision encoder from Kimi-K2.6 as the image encoder — and trains only a 49.5M-parameter PatchMerger MLP projector that maps MoonViT’s 1152-dim patch embeddings into GLM’s 6144-dim token space. Text weights ship pre-quantized to NVFP4 (from nvidia/GLM-5.2-NVFP4), so a Blackwell-only serving path fits the full 1M-token context on 8×B200 or 256k on 4×B200 via SGLang with an out-of-tree Glm5vForConditionalGeneration plugin. It’s a concrete demonstration that the “bolt a frozen ViT onto a frozen frontier LLM through a small trained connector” recipe now works at the 744B-active-MoE scale on consumer-accessible open weights.

Key claims

Both parent towers are byte-identical to their upstream releases (GLM-5.2 and Kimi-K2.6 MoonViT); the only newly-trained parameters are the 49.5M projector [component table].
The projector is a PatchMerger MLP with pre_norm → linear_1 → GELU → linear_2 mapping 1152 → 4608 → 6144 dims — a two-layer expansion-then-projection into GLM’s token space [component table].
Text weights ship as NVFP4 (from nvidia/GLM-5.2-NVFP4); serving requires Blackwell-class GPUs (--quantization modelopt_fp4, --attention-backend dsa) [Quickstart].
Full 1M-token context runs on 8×B200 with tp-size 8, --kv-cache-dtype fp8_e4m3, --page-size 64, and --mem-fraction-static 0.85; a 256k variant fits on 4×B200 [8×B200 / 4×B200 sections].
Glm5vForConditionalGeneration is not yet an upstream SGLang architecture, so the release bundles its own out-of-tree plugin (sglang_glm5v) that must be installed and activated via SGLANG_EXTERNAL_MODEL_PACKAGE / SGLANG_EXTERNAL_MM_PROCESSOR_PACKAGE / SGLANG_EXTERNAL_MM_MODEL_ARCH env vars plus a patch entry point [SGLang section].
The reasoning parser is glm45 and the tool-call parser is glm47 — GLM-5.2’s CoT and tool-call formats are inherited unchanged, with chain-of-thought arriving in message.reasoning_content and the answer in message.content [Query it section].
First deployment downloads ~466 GB of weights [Deploy on Baseten section].
Licensing: MIT for the projector; redistributed upstream weights keep their original licenses (GLM-5.2 MIT, Kimi-K2.6 Modified MIT) [License section].

Method

Rather than train a new VLM from scratch or finetune the GLM-5.2 backbone, Baseten wires the MoonViT-3d encoder from a different frontier open-weights model (Kimi-K2.6) into GLM-5.2 through a learned connector, keeping every backbone parameter frozen. MoonViT produces 1152-dim patch embeddings; the PatchMerger projector applies pre-norm, expands to a 4608-dim hidden layer with GELU, then projects to GLM’s 6144-dim token embedding space, and those tokens are inserted into GLM-5.2’s sequence in the same slot a text image-placeholder would occupy. Because neither backbone is updated, only 49.5M new parameters need to be trained — a tiny fraction of the ~744B-parameter surface — and both parents remain byte-identical to their published checkpoints.

Serving is Blackwell-only. Text weights arrive already in NVFP4 (via nvidia/GLM-5.2-NVFP4), and the SGLang launch config combines --quantization modelopt_fp4 for the linears, --attention-backend dsa for GLM-5.2’s DeepSeek Sparse Attention, --mm-attention-backend sdpa for the multimodal path, and --kv-cache-dtype fp8_e4m3 for the KV cache. Ready-to-push Truss configs handle deploy on Baseten. Because upstream SGLang doesn’t yet know about Glm5vForConditionalGeneration, the repo bundles a sglang_glm5v plugin that patches the model registry at startup.

Results

No benchmarks are reported in the release notes. The artifact is engineering-oriented: reproducible checkpoints, working SGLang launch commands for two hardware footprints (8×B200 full-1M-context and 4×B200 256k-context), and Truss configs for one-shot deployment. Downloads count (73 as of filing) is a proxy for uptake at release day. The claim being demonstrated is that the “add vision by training only a small projector on top of two frozen frontier towers” recipe works at 744B-active-MoE scale end-to-end — not that it matches purpose-built VLMs like Qwen3-VL Technical Report Qwen3-VL or Kimi-K2.6’s own native vision on benchmark tables.

Why it’s interesting

This sits at the intersection of three threads the wiki already tracks. It’s a concrete instance of the frozen-projector recipe that InstructX: Towards Unified Visual Editing with MLLM Guidance used to bolt Qwen2.5-VL-3B onto Wan2.1-14B via an MLP connector — same shape (frozen source encoder + trained MLP connector + frozen target backbone) applied to a much larger asymmetric pair (a ViT from one model family bolted onto a 744B MoE LLM from another). It also extends the community-fine-tune surface around GLM-5: from Vibe Coding to Agentic Engineering GLM-5 and GLM-5.1 — Next Level of Open Source (Z.ai announcement) GLM-5.1 in a direction Z.ai themselves have not shipped: the parallel first-party path is GLM-5V-Turbo — Vision Coding Model from Z.ai (announcement) GLM-5V-Turbo, which is a separate multimodal model rather than a frozen-tower graft. And it’s an early open-weights datapoint for the NVFP4 serving stack that NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 Nemotron 3 Ultra and The 4-bitter Lesson: Balancing Stability and Performance in NVFP4 RL 4-bitter Lesson have been staking claims in — this time on the inference side of a community-produced multimodal checkpoint, not a first-party trainer.