Qwen3.5: Towards Native Multimodal Agents

TL;DR

Qwen3.5-397B-A17B is the first open-weight release in the Qwen3.5 series — a natively multimodal vision-language model (text + image + video) built on a hybrid Gated DeltaNet + sparse Mixture-of-Experts backbone. 397B total parameters with 17B activated per forward pass, 250k vocabulary, support for 201 languages, Apache 2.0 licensed. Headline architectural moves: alternating linear-attention (Gated DeltaNet) and full-attention layers in a 3:1 ratio to cut KV-cache cost ~4×, MoE scaled to 512 fine-grained experts (10 routed + 1 shared per token, 4× more than Qwen3’s 128), early-fusion vision-language pretraining, and RL scaled across million-agent environments. The launch claims 8.6×–19.0× decoding throughput over Qwen3-Max and benchmark parity-or-better with closed frontier VLMs (85.0 MMMU, 88.6 MathVision, 90.8 OmniDocBench). Of interest because it operationalizes at production scale the hybrid-linear-attention + ultra-sparse-MoE recipe that Qwen3-Next previewed in Sep 2025.

Key claims

397B total / 17B active MoE; fewer active parameters than Qwen3 235B (22B active) despite 69% more total parameters [§Efficient Hybrid Architecture].
Architecture is 15 × (3× Gated DeltaNet + 1× global attention) for 60 layers total, with hidden dim 4096 and a 248,320-token vocabulary [model card / NVIDIA NIM card].
MoE routing: 10 routed + 1 shared expert active out of 512 total experts — ~4× wider expert pool than Qwen3 (128 experts) [§Efficient Hybrid Architecture].
Linear-attention layers (Gated DeltaNet) alternate with full-attention layers in a 3:1 ratio, reducing KV-cache memory by ~4× and enabling 8.6×–19.0× decoding throughput over Qwen3-Max [launch tweet].
Native multimodal: vision encoder is a ViT with early-fusion training on trillions of image+video+text tokens; outperforms the prior dedicated Qwen3-VL line across vision-language reasoning and agent benchmarks [§Unified Vision-Language Foundation].
Single model spans reasoning and non-reasoning modes (reversal vs Qwen3, which shipped separate Instruct and Thinking variants); the hosted Qwen3.5-Plus serves with a 1M-token context window [HF model card].
Multi-token prediction enables 2-3× speculative-decoding speedup at inference [Lambda model brief].
RL is scaled across “million-agent environments” with progressively complex task distributions; training infrastructure achieves near-100% multimodal-training efficiency relative to text-only training [§Scalable RL Generalization, §Next-Generation Training Infrastructure].
Reported benchmarks (open-weights, day-0): 88.4 GPQA, 87.8 MMLU-Pro, 85.0 MMMU, 88.6 MathVision, 90.8 OmniDocBench, 87.5 VideoMME, 62.2 OSWorld-Verified, 66.8 AndroidWorld [HF/Together model cards].
Apache 2.0 weights on Hugging Face and ModelScope at release; day-0 serving in HF Transformers, vLLM, SGLang, llama.cpp (text+vision), and MLX [GitHub README, §Quickstart].

Method

The model is a decoder-only causal LM with an interleaved Vision Transformer encoder, trained with early fusion on multimodal tokens. The text/vision backbone is 60 transformer layers organized into 15 repeating blocks of 3 × GatedDeltaNet + 1 × global-attention — a hybrid that biases most layers toward linear-cost recurrence and reserves quadratic full-attention only for every fourth layer. The 3:1 ratio is the central efficiency lever: linear attention via Gated DeltaNet has no KV cache, so the aggregate KV footprint shrinks ~4× and decoding throughput rises 8.6×–19.0× over the dense Qwen3-Max baseline.

The FFN is replaced by a fine-grained MoE: 512 experts per layer, of which 10 are routed per token plus 1 is shared (always active). At 4× the expert count of Qwen3, this is a much sparser routing regime — only ~2% of experts active per token. Total params 397B; active per token 17B. Multi-token prediction is added on top, enabling 2-3× speculative decoding at inference.

Training is described as “early fusion” across trillions of multimodal tokens (text, image, video) with a 248,320-token vocabulary, followed by RL post-training “scaled across million-agent environments with progressively complex task distributions.” The release does not (yet) include a tech report, so reconciliation with Qwen3-Next is left to the GitHub README’s architecture summary and third-party model-card recaps.

Results

Reported by Qwen and aggregated from third-party model cards (HF, NVIDIA NIM, Together, Lambda, Artificial Analysis). All numbers are open-weights, day-0 self-reports, not independent reproductions.

Reasoning / knowledge. GPQA 88.4, MMLU-Pro 87.8, HLE 28.7.
Vision-language. MMMU 85.0 (up from Qwen3-VL’s 80.6), MathVision 88.6 (claimed ahead of Gemini 3 Pro’s 86.6), OmniDocBench 90.8, VideoMME 87.5.
Agent. OSWorld-Verified 62.2, AndroidWorld 66.8.
Multilingual. MMMLU 88.5 (88 languages), WMT24++ 78.9 (55 languages); coverage 201 languages and dialects (up from 119 in Qwen3).
Inference throughput. 8.6×–19.0× decoding throughput vs Qwen3-Max — directly attributable to the 3:1 linear/full attention layer mix and 4× KV-cache reduction.
Token efficiency. ~86M output tokens (including ~80M reasoning tokens) to run the Artificial Analysis Intelligence Index — more than Qwen3 235B (63M) but less than Kimi K2.5 (89M) and GLM-5 (110M).
Release cadence. 397B-A17B first (Feb 16, 2026), followed by 122B-A10B / 35B-A3B / 27B (Feb 24), then 9B / 4B / 2B / 0.8B (Mar 2). Qwen3.6 (35B-A3B, then 27B) followed in April with the same MoE-with-hybrid-attention backbone.

Why it’s interesting

Two threads to pull. First, this is the production-scale validation of the architectural bet that Qwen3-Next (80B-A3B, Sep 2025) made: hybrid Gated DeltaNet + ultra-sparse MoE is now the default Qwen recipe for frontier-scale open weights. If linear-attention layers at the 3:1 ratio hold up under independent inference benchmarking, the open-source frontier is converging on a non-quadratic backbone — relevant for any Luma serving stack pushing long-context multimodal contexts. Second, the move to “native multimodal” with early-fusion training that replaces the separate Qwen3-VL line is the same trajectory Kimi K2.5 took (continued-pretraining a text MoE on ~15T vision tokens); two open frontier-VLM datapoints in three months both abandoning the separate-VL-model pattern is worth tracking. Architecture details (60 layers, 15 blocks, 512 experts, 10+1 active) are reported by third-party model cards but not yet by a Qwen tech report — confidence in the exact numbers should scale with whether a paper drops.