NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4

TL;DR

Nemotron 3 Ultra is the frontier (Ultra) tier of NVIDIA’s Nemotron 3 family — a 550B-total / 55B-active hybrid LatentMoE LLM with interleaved Mamba-2, MoE, and select Attention layers, end-to-end NVFP4 pre-training over ~20T tokens, Multi-Token Prediction (MTP) for native speculative decoding, and a 1M-token context. The model card on Hugging Face pairs the NVFP4 inference checkpoint with full deployment recipes (vLLM, SGLang, TRT-LLM, with Ray for multi-node) targeting 4×B200/GB200/GB300 or 8×H100. Quoted benchmarks place the NVFP4 release near parity with the BF16 reference across agentic (SWE-Bench Verified 69.7 NVFP4 vs 71.9 BF16, Terminal Bench 2.1 53.9 vs 56.4, TauBench v3 Average 70.3 vs 70.9), reasoning/knowledge (GPQA 87.9 vs 87.0, HLE 26.1 vs 26.7), and long-context (RULER 1M 94.0 vs 94.7) tasks. Together with Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning and Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence this completes the Nano/Super/Ultra tier announced in NVIDIA Debuts Nemotron 3 Family of Open Models.

Key claims

• Total/active params are 550B/55B with a hybrid LatentMoE = Mamba-2 + MoE + Attention architecture plus Multi-Token Prediction (MTP), max context up to 1M tokens [§Model Summary].
• Minimum-GPU deployment target is 4×GB200/B200/GB300/B300 or 8×H100; the Hugging Face release is NVFP4-quantized for inference [§Model Summary, §Quick Start Guide].
• LatentMoE projects tokens into a smaller latent dimension for expert routing and computation, improving accuracy per byte while reducing all-to-all traffic [§Model Design].
• Pre-training is ~20T tokens with an NVFP4 recipe; the majority of linear layers use NVFP4 for weights/activations/gradients while latent projections, MTP layers, QKV/attention projections, and embeddings stay in BF16 or MXFP8 for training stability [§Model Design, §Training Methodology Stage 1].
• MTP uses a shared-weight design across prediction heads (the same choice made for Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning), enabling native speculative decoding and more stable longer drafts than independently-trained offset heads [§Model Design].
• Post-training pipeline is SFT → multi-environment async-GRPO RL (math/code/science/tool-use/multi-turn/structured outputs) → RLHF → Multi-Domain On-Policy Distillation (MOPD) using strong teachers on the student’s own rollouts [§Training Methodology Stages 2-4].
• The async RL architecture fully decouples training from inference across separate GPU devices using in-flight weight updates and MTP-accelerated rollout generation [§Training Methodology Stage 3].
• Agentic benchmarks (NVFP4): SWE-Bench Verified 69.7, SWE-Bench Multilingual 65.8, Terminal Bench 2.1 53.9, GDPVal 47.9, ProfBench (Search) 56.4, PinchBench 89.8, BrowseComp 41.4, TauBench v3 Average 70.3 [§Benchmarks].
• Reasoning/Knowledge (NVFP4): GPQA (no tools) 87.9, IOI 2025 564.7, SciCode (subtask) 43.5, HLE (no tools) 26.1, CritPt 3.4, OmniScience Accuracy 24.6 / Non-Hallucination 75.5 [§Benchmarks].
• Long-context: RULER 1M 94.0 (NVFP4) vs 94.7 (BF16); AA-LCR 65.5 vs 65.4 — long-context evaluation survives FP4 quantization at near-parity [§Benchmarks].
• IFBench (prompt) 82.3 NVFP4 vs 81.7 BF16 — NVFP4 quantization does not degrade instruction following [§Benchmarks].
• Governing license is OpenMDW-1.1 (open weights with explicit commercial-use grant) [§License/Terms of Use].
• A separate NVIDIA-Nemotron-3-Ultra-550B-A55B-Base-BF16 base model is also released [§Training Methodology Stage 1].
• The accompanying technical report is published at research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Ultra-Technical-Report.pdf and the RL environments + datasets are released as part of NeMo Gym [§Training Methodology].

Method

The Ultra checkpoint is the largest member of the Nemotron 3 LatentMoE family. Tokens are down-projected from the model hidden dimension into a smaller latent dimension where expert routing and all expert computation happen, then up-projected back — the saved bandwidth and FLOPs are spent on more total and active experts at constant inference cost (this is the same LatentMoE design analyzed in Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning §2.1.1). The backbone interleaves Mamba-2, MoE, and a small number of Attention layers as long-range anchors, matching the hybrid pattern in the Super tier but scaled to 550B total / 55B active. MTP heads (shared-weight) sit at the top for native speculative decoding, which the deployment recipes set to draft length 5.

Pre-training is end-to-end NVFP4 over ~20T tokens, with selective BF16/MXFP8 islands for components where FP4 underflow would compromise stability (latent projections, MTP layers, QKV/attention projections, embeddings). Post-training is a four-stage pipeline: SFT on synthetic code/math/science/tool-use/structured-output data → asynchronous GRPO with in-flight weight updates across multi-environment rollouts → RLHF for conversational quality → Multi-Domain On-Policy Distillation (MOPD) which uses strong teacher models to guide training on the student’s own on-policy rollouts (the same MOPD recipe surfaced as a paper in Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation). The Hugging Face card ships full vLLM, SGLang, and TRT-LLM serving recipes with NVFP4 KV-cache, chunked prefill, mamba SSM cache stochastic rounding (mamba-cache-philox-rounds 5), 5-token MTP speculative decoding, and Ray-based multi-node templates targeting 4×B200 single-node or 2×4×GB300 multi-node deployments.

Results

NVFP4 benchmarks largely match the BF16 reference — the headline finding for an FP4-quantized frontier model:

• Agentic / tool use: SWE-Bench Verified 69.7 NVFP4 vs 71.9 BF16; SWE-Bench Multilingual 65.8 vs 67.7; Terminal Bench 2.1 53.9 vs 56.4; PinchBench 89.8 vs 90.0; ProfBench (Search) 56.4 vs 56.0; TauBench v3 Average 70.3 vs 70.9; BrowseComp 41.4 vs 44.4; GDPVal 47.9 vs 46.7 [§Benchmarks].
• Reasoning / knowledge: GPQA 87.9 vs 87.0; IOI 2025 564.7 vs 570.0; SciCode 43.5 vs 44.6; HLE (no tools) 26.1 vs 26.7; CritPt 3.4 vs 3.1; OmniScience Accuracy 24.6 vs 24.1 [§Benchmarks].
• Long context: RULER 1M 94.0 vs 94.7; AA-LCR 65.5 vs 65.4 — long-context retrieval survives FP4 essentially unchanged [§Benchmarks].
• Instruction following: IFBench (prompt) 82.3 vs 81.7 — slight improvement under NVFP4 [§Benchmarks].
• TauBench v3 per-domain: Airline 80.0/81.5, Retail 88.4/86.4, Telecom 93.6/92.9, Banking 19.2/22.6 — Banking is the only sub-task where NVFP4 loses non-trivially [§Benchmarks].

No head-to-head against GPT-OSS, DeepSeek-V3/V4, Qwen3.5/3.6/3.7-Max, or Kimi K2 Thinking is included on the HF card itself; the wiki should expect those comparisons in the linked technical report.

Why it’s interesting

This is the long-promised top tier of the Nemotron 3 family — the umbrella drop announced in NVIDIA Debuts Nemotron 3 Family of Open Models back in December 2025 listed Ultra (~500B / ~50B-active) as “H1 2026”, and this is that release at 550B/55B-active. With Nano already shipped, Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning for the Super tier landing in April, and Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence handling the multimodal Nano variant, the Nano/Super/Ultra triple is now complete — making the family directly comparable to the Llama-4 Maverick/Scout, DeepSeek-V3/V4, and GLM-4.5/5 open releases that Open foundation-model releases tracks. The architecture is the same LatentMoE + Mamba-2 + Attention hybrid + shared-weight MTP cocktail proven on Super, scaled ~4.5× in total parameters and ~4.3× in active parameters; the close NVFP4-vs-BF16 benchmark deltas (worst case ≈3 points on Banking and SWE-Bench Verified, near-parity or slightly better elsewhere) extend the Training stability at scale thesis that end-to-end FP4 pretraining is viable at frontier scale into a second size point. The MOPD post-training stage (also the centerpiece of Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation) plus the multi-environment async-GRPO RL recipe puts this firmly alongside GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models and GLM-5: from Vibe Coding to Agentic Engineering as a frontier-scale demonstration of agentic post-training. For LLM Inference Efficiency the deployment recipes are unusually detailed — NVFP4 KV-cache, 5-token MTP speculative decoding, chunked prefill, multi-node Ray, both max-throughput and min-latency TRT-LLM presets — making this card a practical reference for serving frontier MoE LLMs on Blackwell.

See also

• NVIDIA Debuts Nemotron 3 Family of Open Models — the December 2025 family announcement that promised Ultra in H1 2026; this is that release.
• Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning — the Super (120B / 12B-active) tier; same LatentMoE + Mamba-2 + Attention + MTP architecture, scaled down.
• Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence — the multimodal Nano variant in the same family.
• Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation — the Multi-Domain On-Policy Distillation (MOPD) post-training recipe used in Stage 4 here.
• Scalable Training of Mixture-of-Experts Models with Megatron Core — the Megatron-Core MoE systems substrate that explicitly targets Nemotron 3.
• Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search — the architecture-search predecessor on the efficient-Nemotron lineage.
• GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models — sibling frontier-MoE LLM with overlapping agentic-RL post-training emphasis.
• GLM-5: from Vibe Coding to Agentic Engineering — open agentic-coding-focused LLM at comparable scale; overlapping SWE-Bench / Terminal Bench targets.
• Kimi K2 Thinking — 1T MoE thinking model with native INT4 QAT and 200–300-step tool use — closest open frontier-MoE competitor (1T total) with INT4 QAT instead of NVFP4.
• Open foundation-model releases — concept page tracking coordinated open-weights drops.
• MoE Routing Design — LatentMoE is a notable point in the routing-design space.
• Hybrid Linear Attention — Mamba-2 + Attention hybrid backbone fits this concept.
• Training stability at scale — extends the NVFP4-end-to-end pretraining stability story onto a 550B-total model.
• LLM Inference Efficiency — the deployment recipes are a practical reference for FP4 + MTP + Mamba serving.
• Agentic Software Engineering, Reasoning RL — SWE-Bench, Terminal Bench, Tau Bench v3 are the post-training targets.