Skip to content
Bud's Wiki

Nemotron 3 Ultra — 550B/55B-active hybrid Mamba-Transformer MoE open frontier model for long-running agents (NVIDIA AI)

TL;DR

Section titled “TL;DR”

NVIDIA’s launch tweet for Nemotron 3 Ultra — the long-promised 550B-total / 55B-active hybrid Mamba-2 + Attention + LatentMoE flagship of the Nemotron 3 family, pre-trained end-to-end in NVFP4 with MTP layers for native speculative decoding, 1M context, and post-trained with SFT + RL + Multi-teacher On-Policy Distillation (MOPD) for long-horizon agentic workloads. Weights and recipe ship on Hugging Face / NVIDIA NIM / OpenRouter on launch day under the NVIDIA Open Model License. NVIDIA quotes 5.9× / 4.8× / 1.6× higher throughput vs GLM-5.1-754B-A40B, Kimi K2.6-1T-A32B, and Qwen-3.5-397B-17B at 8k input / 64k output, scoring 48 on Artificial Analysis’s Intelligence Index — claimed as the strongest US open-weights model at filing time, though Kimi K2.6 still leads overall.

Key claims

Section titled “Key claims”
  • Nemotron 3 Ultra is a 550B-total / 55B-active hybrid Mamba-Transformer MoE pre-trained in NVFP4 with LatentMoE and MTP layers [tweet, NVIDIA research page].
  • The 1M-token context is supported through the Mamba-2 / Attention hybrid backbone reused from Nemotron 3 Super, avoiding quadratic KV-cache growth at long context [tweet, NVIDIA research page].
  • Reported throughput: 5.9× over GLM-5.1-754B-A40B, 4.8× over Kimi K2.6-1T-A32B, and 1.6× over Qwen-3.5-397B-17B at 8k input / 64k output [NVIDIA research page].
  • Agentic post-training pipeline combines SFT, RL, and Multi-teacher On-Policy Distillation (MOPD); model supports inference-time reasoning budget control [NVIDIA research page].
  • Open release packages weights (NVFP4 + BF16), training recipe, and pretraining-data collections (Code-v3, Legal-v1, Specialized-v1.2) [NVIDIA research page].
  • Available day-zero on Hugging Face, NVIDIA NIM, OpenRouter, ModelScope, and Amazon SageMaker JumpStart [aws blog].

Method

Section titled “Method”

The model extends the Nemotron 3 Super recipe to 4.5× the scale: ~88-layer Mamba-2 + LatentMoE backbone with periodic global-attention anchors, MTP heads for speculative decoding, full NVFP4 pretraining with 2D block scaling on weights / 1D on activations and gradients (selectively keeping the last 15% of layers, embeddings, latent projections, MTP, QKV-attention, and Mamba output projections in higher precision), then 1M long-context extension. Post-training swaps standard distillation for Multi-teacher On-Policy Distillation across multiple domain experts. The tweet itself carries minimal technical content; the underlying artifact is the NVIDIA Nemotron 3 Ultra release page plus the Hugging Face / NIM / OpenRouter checkpoints.

Results

Section titled “Results”
  • Artificial Analysis Intelligence Index 48 — claimed as #1 among US open-weights models, trailing Kimi K2.6 internationally [chatforest, aitoolsrecap].
  • 5x inference throughput and up to 30% cost reduction vs other open frontier models on long-running-agent workloads [tweet, aws blog].
  • 5.9× / 4.8× / 1.6× higher throughput than GLM-5.1-754B-A40B / Kimi K2.6-1T-A32B / Qwen-3.5-397B-17B at 8k/64k ISL/OSL [NVIDIA research page].
  • ~50M cumulative Nemotron 3 family downloads in the year leading to April 2026 [cryptobriefing].

Why it’s interesting

Section titled “Why it’s interesting”

The completion of NVIDIA’s Nemotron 3 cascade — already at Nano (30B / 3B-active, Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence) and Super (120B / 12B-active, Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning) — and the first time the LatentMoE + NVFP4 + hybrid Mamba recipe has been pushed to frontier-active-parameter scale. The reported 5.9× throughput edge over GLM-5.1-754B-A40B and 4.8× over Kimi K2.6 in the long-output regime is the most concrete data point so far that hybrid-Mamba + extreme-sparsity MoE can outprice dense-attention DeepSeek/Kimi-class designs in the long-output, long-horizon agent regime that Context-Bench: Benchmarking LLMs on Agentic Context Engineering argues is the hard part of agent serving. Complements Scalable Training of Mixture-of-Experts Models with Megatron Core (the Megatron-Core MoE systems paper that names Nemotron-3 as its target) and continues the open-weights race tracked by The Trinity Manifesto — Arcee's open-weight MoE family (Mini + Nano + Large roadmap) and Kimi K2 Thinking — 1T MoE thinking model with native INT4 QAT and 200–300-step tool use. Worth contrasting with DeepSeek-V4 collection release (Flash + Pro, up to 1.6T) — both target the same agentic frontier, but Nemotron stakes its bet on Mamba-hybrid + LatentMoE while DeepSeek-V4 stays attention-only with Sparse Attention.

See also

Section titled “See also”