Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

TL;DR

Nemotron 3 Super is NVIDIA’s open 120B-total / 12B-active MoE LLM that combines three independent bets in one model: a hybrid Mamba-2 + Attention backbone, the new LatentMoE architecture (route through a compressed latent dimension, then expand top-K and total experts to compensate), and end-to-end NVFP4 pre-training over 25T tokens. Headline: comparable benchmark accuracy to GPT-OSS-120B and Qwen3.5-122B with up to 2.2× and 7.5× higher throughput respectively on 8k-input/64k-output workloads, plus 1M context support and competitive RULER@1M. The release is fully open — pre-trained, post-trained, NVFP4 and FP8 quantized checkpoints, plus the pre-training (Nemotron-Pretraining-Specialized-v1.1) and post-training (Nemotron-Super-Post-Training-Data) datasets — and the agentic post-training pipeline is the explicit centerpiece.

Key claims

The model is 88 layers of hybrid Mamba-2 + Latent MoE with a few global-attention “anchors”, at 120.6B total / 12.7B active parameters, top-22 routing over 512 experts per layer, head dim 128, model dim 4096, MoE latent size 1024, 1M context [§2.1, Table 1, Fig. 2].
LatentMoE projects tokens from hidden dim $d$ to a smaller latent dim $\ell$ via a learnable down-projection, runs all routed expert computation and all-to-all in the latent space, then up-projects; the savings are spent increasing total experts $N \to N \cdot d/\ell$ and active experts $K \to K \cdot d/\ell$ , raising accuracy at constant inference cost [§2.1.1, Fig. 3].
The LatentMoE design is justified by five principles distilled from systems analysis: (1)/(2) per-expert memory and routing communication both scale with $d$ , (3) effective non-linear budget $K \cdot m$ must be preserved, (4) a task-specific feature rank $r_{\text{eff}}$ lower-bounds how far $d$ can be cut, (5) jointly scaling $N$ and $K$ improves quality by expanding expert-combination space [§2.1.1].
Multi-Token Prediction with shared-weight heads (rather than $N$ independent offset-trained heads) regularizes the prediction head across draft positions and produces more stable acceptance under longer recursive drafts; achieves average acceptance length 3.45 (vs DeepSeek-R1 2.70, Qwen3-Next 3.33) at draft length 7 on SPEED-Bench [§2.1.2, Table 2, Fig. 4].
Throughput-vs-latency Pareto frontier improves substantially with MTP at draft depth 3 over MTP-off on B300 NVFP4 + TRT-LLM [Fig. 5].
End-to-end NVFP4 pretraining at 25T tokens is shown to be stable, using 2D block scaling for weights, 1D-block scaling for activations/gradients along the GEMM reduction axis, Random Hadamard Transforms on wgrad inputs, stochastic rounding on gradients, E2M1 elements in 16-element micro-blocks with E4M3 micro-block scales and an FP32 global scale [§2.2, Table 3].
Selected layers are held in higher precision to preserve quality: final 15% of network in BF16 (stability), latent projections / MTP / QKV-attention / embeddings in BF16, Mamba output projection in MXFP8 (NVFP4 caused high underflow rates there) [Table 3].
NVFP4 pretraining produces ≈3× more zero-valued weight gradients than BF16 at matched token count, traced to underflow in dgrad of FC2 (creates wgrad-FC1 zeros via backprop) and fprop of FC1 (creates wgrad-FC2 zeros) inside routed expert layers; this correlates with a magnitude-pattern channel collapse in expert FC1/FC2 matrices [§2.2, Figs. 6–8].
Switching all tensors to MXFP8 at 19T tokens (1T pre-anneal) improved the loss trajectory but produced no sustained downstream-accuracy improvement, so the released model stays in NVFP4 the full horizon [§2.2, Fig. 9].
Pre-training is split into a 20T-token “diversity / broad coverage” Phase 1 (80%) and a 5T-token “high-quality / benchmark” Phase 2 (20%), with Phase 2 explicitly upweighting synthetic data targeted at code, formal logic, economics, and multiple-choice QA [§1, §2.3].
The released Nemotron-Pretraining-Specialized-v1.1 includes ~23M problem–solution pairs of synthetic Python from a 91-concept taxonomy extracted from HumanEval, generated open-ended by GPT-OSS-20B and solved by GPT-OSS-120B with strict cleaning [§2.3.2].
The base model “achieves significantly better accuracy than similarly sized state-of-the-art base models” — GLM-4.5-Air-Base and Ling-flash-Base-2.0 — across a comprehensive benchmark suite [§1, §2].
On agentic benchmarks the post-trained NVFP4 checkpoint reports 73.3 IFBench (inst. following), 95.4 HMMT Feb25 (math), 59.9 SWE-Bench (coding, OpenHands harness), 17.4 HLE (science), 24.5 Terminal Bench Hard, 60.5 Tau Bench v2 (tool use), and 91.6 RULER @ 1M, comparable to or better than GPT-OSS-120B and Qwen3.5-122B [Fig. 1].
Headline throughput claim: up to 2.2× higher than GPT-OSS-120B (MXFP4 weights, MXFP8 activations, FP8 KV-cache) and 7.5× higher than Qwen3.5-122B (BF16), on 8k input / 64k output, measured on B200 with best of vLLM / TRT-LLM [Abstract, Fig. 1].
The agentic training recipe is the explicit focus: substantially scaled RL environment breadth, agentic SFT volume + quality, multi-step tool-use post-training, plus resiliency upgrades to enable large-scale asynchronous RL [§1, §3].

Method

The architecture stacks 88 layers with a periodic interleaving pattern: blocks of Mamba-2 layers are paired with Latent MoE layers, and a small number of global-attention layers (GQA, 32 query heads / 2 KV heads, head dim 128, no positional embeddings, RMSNorm) are inserted as anchors for long-range dependency. Mamba-2 carries most token-mixing work since its constant-size state avoids the quadratic KV-cache cost that limits long-context throughput. Each MoE layer runs LatentMoE — down-project to latent dim 1024, route 22 of 512 latent-space experts via softmax+top-K with a shared expert in full $d$ , expert intermediate dim 2688, then up-project — so per-expert weight loads and all-to-all volumes drop by $d/\ell$ while expert combination capacity grows. MTP layers (2, shared-weight, BF16) sit at the top for native speculative decoding.

Pre-training is 25T tokens in two phases, end-to-end NVFP4 except for the last 15% of network depth, embeddings, latent projections, MTP, QKV/attention projections (BF16), and Mamba output projection (MXFP8). The NVFP4 recipe uses 2D block scaling for weights, 1D block scaling along the GEMM reduction axis for activations/gradients, RHT on wgrad inputs, and stochastic gradient rounding. Long-context extension goes to 1M. Post-training is SFT + RL with a heavy emphasis on agentic capabilities — software engineering (OpenHands harness for SWE-Bench), terminal use, and tool use — with a separately released Nemotron-Super-Post-Training-Data collection of RL environments and SFT datasets. The reward model used for RLHF is Qwen3-Nemotron-235B-A22B-GenRM-2603, also released.

Results

Throughput on B200: up to 2.2× GPT-OSS-120B and 7.5× Qwen3.5-122B at 8k/64k ISL/OSL [Fig. 1].
Accuracy parity or better vs GPT-OSS-120B / Qwen3.5-122B across IFBench, HMMT Feb25, SWE-Bench (OpenHands), HLE, Terminal Bench Hard, Tau Bench v2, RULER@1M [Fig. 1].
MTP acceptance length 3.45 at draft length 7 vs DeepSeek-R1 2.70, Qwen3-Next 3.33 (SPEED-Bench), with the gap widening at draft indices 4–7 where recursive drafting is hardest [Table 2, Fig. 4].
Base model beats GLM-4.5-Air-Base and Ling-flash-Base-2.0 across a comprehensive benchmark suite [§1] (specific tables truncated in the report excerpt).
7% of total parameters end up with zero-valued weight gradients by end of NVFP4 pretraining — a measurable artifact of low-precision quantization underflow, not a stability failure [§2.2].
1T-token NVFP4 model exhibits the same channel-magnitude pattern and zero-gradient prevalence as a 10T-token BF16 model — i.e. NVFP4 accelerates the appearance of these patterns but does not introduce a fundamentally different dynamic [§2.2, Fig. 7].
The MXFP8 healing experiment (switch all tensors to MXFP8 for the final 1T tokens before annealing) improved loss but did not improve downstream accuracy [Fig. 9].
The released Qwen3-Nemotron-235B-A22B-GenRM-2603 generative reward model is shared as part of the package.

Why it’s interesting

Lands at the intersection of four threads the wiki is already tracking. First, this is the clearest current example of Open foundation-model releases at the 120B-active-MoE LLM scale — open weights and pre-training data and post-training data and training recipe, comparable to GPT-OSS-120B but going further on data transparency. Second, Scalable Training of Mixture-of-Experts Models with Megatron Core is the open Megatron-Core MoE systems write-up that names Nemotron-3 explicitly as one of its target models; Nemotron 3 Super is the model that exercises that stack in production, and the FP4 selective-precision scheme described there is implemented here in detail. Third, the NVFP4 pretraining story directly extends Training stability at scale into the low-precision axis — the zero-gradient analysis tying weight-channel magnitude collapse to dgrad/fprop underflow is a stability finding orthogonal to the architectural (QK-Norm, GatedNorm, mHC) and optimizer (SSO) stories the concept already covers. Fourth, the agentic post-training emphasis — SWE-Bench via OpenHands, Terminal Bench Hard, Tau Bench v2 — puts this firmly in the Agentic Software Engineering and Reasoning RL cohort alongside GLM-5: from Vibe Coding to Agentic Engineering and On Data Engineering for Scaling LLM Terminal Capabilities. The Mamba-Attention hybrid + MoE + MTP combination is also the same architectural cocktail showing up across the recent open-MoE wave (DeepSeek-V3, Qwen3-Next), making this useful as a reference for how the three primitives compose at scale.