Qwen3-Next-80B-A3B — hybrid Gated DeltaNet + Gated Attention, ultra-sparse 512-expert MoE, MTP (Alibaba Qwen announcement)

TL;DR

Alibaba Qwen announces Qwen3-Next-80B-A3B, the first model in a new “Qwen3-Next” series engineered for efficient long-context inference: 80B total parameters but only ~3B activated per token via an ultra-sparse 512-expert MoE (10 routed + 1 shared), interleaved Gated DeltaNet (linear attention) and Gated Attention (full attention) layers, and native Multi-Token Prediction. The team claims ~10× cheaper training and ~10× faster inference than Qwen3-32B, especially past 32K context, while matching or exceeding Qwen3-235B in reasoning and long-context benchmarks. Two variants ship at launch: an Instruct model approaching the team’s 235B flagship, and a Thinking model that outperforms Gemini-2.5-Flash-Thinking on reasoning tasks. Open-weights release with day-0 HF / ModelScope / Kaggle / Alibaba Cloud API surface.

Key claims

80B total / ~3B activated per token via a 512-expert MoE with 10 routed + 1 shared expert (1:50 active ratio) — pushing MoE sparsity well past DeepSeek-V3-class designs [tweet body].
Hybrid attention stack: Gated DeltaNet (linear attention) interleaved with Gated Attention (standard full attention) for long-context efficiency without giving up high-fidelity recall [tweet body].
Multi-Token Prediction (MTP) is built into pretraining and inference — boosts pretraining signal and acts as turbo-charged speculative decoding at decode time [tweet body].
Author claim: ~10× cheaper training and ~10× faster inference than Qwen3-32B, with the speed gap widening past 32K context [tweet body].
Two variants released day-0: Qwen3-Next-80B-A3B-Instruct (claimed to approach Qwen3-235B flagship), and Qwen3-Next-80B-A3B-Thinking (claimed to outperform Gemini-2.5-Flash-Thinking) [tweet body].
Open release surface: chat.qwen.ai for interactive use, HuggingFace + ModelScope + Kaggle for weights, Alibaba Cloud API for hosted inference, plus the qwen.ai blog [tweet body, embedded links].

Method

A tweet, not a paper — the announcement names the architectural ingredients without ablations. Composition is: (a) interleaved hybrid attention with Gated DeltaNet handling the bulk of layers (the linear-attention path for long-context cheapness) and Gated Attention handling a minority (the full-attention path for recall); (b) a 512-expert ultra-sparse MoE with 10 routed + 1 shared expert per layer, giving a 3B / 80B = ~3.7% activation ratio that is at the extreme of the open-MoE sparsity frontier as of September 2025; (c) native Multi-Token Prediction so the model both trains with an MTP objective and exposes the MTP heads to inference runtimes for fast decoding without a separate draft model. The Qwen HF model cards (subsequently posted) confirm a fourth ingredient: zero-centered and weight-decayed LayerNorm plus additional stability fixes for the hybrid + ultra-sparse combination. Linked blog qwen.ai/blog?id=4074cc was not retrievable as plain text at filing time (JS-rendered); the announcement text plus the HF model cards are the artifact for this page.

Results

The tweet itself names two headline comparisons: (a) Qwen3-Next-80B-A3B-Instruct “approaches” Qwen3-235B-A22B-Instruct-2507 — the team’s own dense-vs-sparse parity claim at the flagship level despite running ~7× fewer active params; (b) Qwen3-Next-80B-A3B-Thinking “outperforms Gemini-2.5-Flash-Thinking”, positioning the Thinking variant against a closed reasoning-optimized peer. No numbers in the tweet itself; subsequent third-party coverage and the Qwen team’s own follow-on materials report figures like 87.8% AIME25 / 76.6% LiveBench for the Thinking variant and RULER long-context wins over Qwen3-235B-A22B-Instruct-2507 within 256K context, but those aren’t in this artifact. Inference-throughput claim is qualitative (“10× faster than Qwen3-32B, esp. at 32K+ context”) [tweet body].

Why it’s interesting

Qwen3-Next sits at the intersection of three threads the wiki already tracks. (1) It’s the most extreme datapoint yet for the design-space charted in MoE Routing Design — 512 experts and a 1:50 activation ratio test the “expert size and count dominate” hypothesis from Slicing and Dicing MoEs — Margaret Li announcing a >2000-MoE-LM design-space study at production scale, and the 1 shared + 10 routed split echoes the modality-isolated-plus-shared pattern from ERNIE 4.5 Technical Report §2.1 without the modality conditioning. (2) It’s the canonical reference for the MTP variant of speculative decoding called out as workload-best on GLM-4.5-Air in Speculative Decoding: Performance or Illusion? §3.2 — Qwen3-Next ships full MTP heads (not just the first) so the open question of whether full-MTP categorically beats EAGLE-3 on reasoning has a credible test target. (3) It’s the second hybrid linear-attention + full-attention production datapoint on the wiki after Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning, and the comparison is informative: Nemotron-3-Super uses Mamba-2 (state-space) blocks interleaved with attention; Qwen3-Next uses Gated DeltaNet (linear attention) interleaved with Gated Attention — same hybrid recipe, different long-context primitive.