Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence

TL;DR

Nemotron 3 Nano Omni is NVIDIA’s open omni-modal reasoning model that unifies text, image, video, and audio understanding inside a single 30B-total / 3B-active hybrid Mamba-Transformer MoE, with 256K context. It augments the Nemotron 3 Nano 30B-A3B LLM backbone with a C-RADIOv4-H vision encoder and a Parakeet-TDT audio encoder, trained through a multi-stage recipe that progressively introduces modalities and extends context. Headline efficiency: up to 9× higher system throughput than other open omni models with matched interactivity, while topping six leaderboards for document, video, and audio understanding (OCRBenchV2, MMLongBench-Doc, WorldSense, DailyOmni, VoiceBench, MediaPerf). Released as BF16/FP8/FP4 checkpoints alongside a substantial portion of training code and data.

Key claims

The backbone is the Nemotron 3 Nano 30B-A3B hybrid Mamba-2 + Transformer MoE LLM (~3B active parameters per token), augmented with C-RADIOv4-H for vision/video frames and Parakeet-TDT-0.6B-v2 for audio [§1, §2].
Replacing the dense Nemotron Nano V2 12B hybrid backbone with the 30B-A3B MoE hybrid is one of the four explicit advances over Nemotron Nano V2 VL, enabling more efficient long-multimodal-sequence processing and higher inference throughput [§1].
Native audio is added as a new modality alongside image and video, with dynamic image resolution and Conv3D-based temporal video compression in the visual pathway [§1, §2].
256K context length lets the model hold heterogeneous multimodal inputs (long documents, multi-hour audio, multi-minute video) in a single shared perception loop [§2].
Adapter and encoder training scale: ~127B tokens across mixed modalities — text+image, text+video, text+audio, text+video+audio [Training Dataset].
Post-training scale: ~124M curated examples across the same modality combinations, structured for document reasoning, computer use, and long-horizon workflows [Training Dataset].
RL stage: 20 RL datasets across 25 environments covering 5 new multimodal tasks — visual grounding, chart/document understanding, vision-critical STEM, video understanding, and ASR — extending Nemotron’s RL pipeline beyond text [Training Dataset].
Post-SFT RL uses NeMo-RL + NeMo Gym with a Megatron backend, Ray-based distributed setup across B200 and H100 clusters, and multimodal deduplication so repeated rollouts don’t multiply image/video/audio memory [HF blog].
Omni RL uses a diverse verifier suite across multiple-choice / math / GUI grounding / ASR formats, and intentionally includes unanswerable cases to teach abstention [HF blog].
Multimodal token-reduction techniques (efficient video sampling, dynamic resolution) reduce inference latency and increase throughput without sacrificing accuracy [§1, HF blog].
Headline efficiency: up to 9× higher system throughput vs other open omni models with matched interactivity; 7.4× higher for multi-document use cases, 9.2× higher for video, and 2.9× single-stream reasoning speed-up [HF blog Fig. 1].
Top leaderboard placements: best-in-class on OCRBenchV2, MMLongBench-Doc, ChartQA, CharXiv (document); WorldSense, DailyOmni (audio-video); VoiceBench (audio); leading on MediaPerf at the lowest inference cost for video-level tagging [§1, NVIDIA blog].
The release ships BF16, FP8, and NVFP4 checkpoints alongside a large portion of training code and data [Abstract, HF blog].
Training data includes derived labels and reasoning traces generated by Qwen3-VL-30B-A3B-Instruct, Qwen3.5-122B-A10B, Qwen3.5-397B-A17B, Qwen2.5-VL-72B-Instruct, and gpt-oss-120b [HF card §Training Dataset].
The model is positioned as a sub-agent for perception and context maintenance, designed to compose with Nemotron 3 Super (high-frequency execution) and Nemotron 3 Ultra (complex planning) in agent systems [NVIDIA blog].

Method

A unified multimodal architecture: pixel/video frames flow through C-RADIOv4-H with Conv3D-based temporal compression, audio flows through Parakeet-TDT, and both produce token streams interleaved with text into the Nemotron 3 Nano 30B-A3B hybrid backbone. The backbone alternates Mamba-2 layers (for long-context memory efficiency at constant state size) with transformer attention layers (for precise position-dependent reasoning) inside a 30B-total / 3B-active MoE. The 10:1 sparsity ratio + Mamba long-context handling is what enables 256K context on hardware that would struggle with a same-size dense attention model.

Training is multi-stage. Adapter + encoder training (~127B tokens) progressively introduces image, video, and audio modalities to the pretrained LLM, with each stage adding a modality combination. Quality filtering uses model-based judges; CSAM scanning + dedup runs across all image data. Sequences are packed at 32K / 128K / 256K for efficient training. Post-training is SFT (~124M curated examples) followed by RL via NeMo-RL + NeMo Gym on B200 + H100, with multimodal-deduplicated rollouts and a verifier suite that spans multiple-choice, math, GUI grounding, and ASR — and intentionally includes unanswerable cases so the model learns to abstain. The SFT data is constructed in part by re-annotating noisy subsets with Qwen3-VL, with reasoning traces from Qwen3-VL, Qwen3.5, and Kimi-K2.5 families.

Results

Best-in-class on OCRBenchV2 and MMLongBench-Doc (document intelligence), WorldSense and DailyOmni (audio-video), VoiceBench (audio), with leading MediaPerf cost-per-token for video-level tagging [§1, HF blog].
On MediaPerf video tagging, ~9.91 h/h throughput vs ~3.8 h/h for Qwen3-VL — same model class (30B-A3B MoE) [third-party comparison].
Up to 9× higher system throughput than other open omni models at matched interactivity; 2.9× higher single-stream reasoning speed [HF blog].
Surpasses Qwen3-Omni-30B-A3B on every reported benchmark [§5, model card summary].
Computer-use partner H Company reports a “significant leap” on OSWorld at native 1920×1080 input resolution when its agent’s perception loop is powered by Nemotron 3 Nano Omni [NVIDIA blog].
Consistent gains over the prior dense Nemotron Nano V2 VL across document, GUI agent (ScreenSpot/Pro, OSWorld), audio-video (WorldSense, DailyOmni), and voice (VoiceBench) suites [§1, Fig. 3].
Reports zero degradation of the base LLM’s text reasoning performance after multimodal adaptation [§1].

Why it’s interesting

Slots directly into three threads the wiki tracks. First, Unified Multimodal Models — this is the first filed omni-modal unified model in the open cohort (text + image + video + audio + documents) where every modality is in one architecture, contrasting with The Design Space of Tri-Modal Masked Diffusion Models‘s from-scratch tri-modal MDM (text + image + audio, no video) and DuoGen: Towards General Purpose Interleaved Multimodal Generation‘s text-image-only video-DiT recipe. The “unified architecture for perception across modalities” framing also makes it the natural counterpart to NEO-unify (NEO-unify: Building Native Multimodal Unified Models End to End) on the encoder-bearing side of the encoder-free / encoder-bearing axis. Second, Open foundation-model releases — this extends the Nemotron 3 series (Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning is the same-family 120B-A12B text LLM) into multimodal, with the same release packaging convention (BF16 + FP8 + FP4 checkpoints, open data, open code) — directly comparable to Qwen3-Omni-30B-A3B as the next datapoint in the same-class open-omni-MoE cohort. Third, Reasoning RL — the explicit emphasis on RL across 25 environments × 5 multimodal tasks, including unanswerable-case abstention training, is the most concrete filed instance of multimodal verifier-suite RL.