OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM

TL;DR

OmniVinci is an NVIDIA 9B open-weight omni-modal understanding LLM (vision + audio + text) built on the NVILA / VILA codebase. The paper’s contributions are three architectural pieces — OmniAlignNet for aligning vision and audio embeddings in a shared latent space, Temporal Embedding Grouping for relative cross-modal time, and Constrained Rotary Time Embedding for absolute time — plus a 24M-conversation synthesis pipeline. It claims to beat Qwen2.5-Omni by +19.05 on DailyOmni (cross-modal), +1.7 on MMAR (audio), and +3.9 on Video-MME (vision) while training on only 0.2T tokens — 6× less than Qwen2.5-Omni’s 1.2T. Released under NVIDIA’s OneWay Noncommercial License, not Apache/MIT.

Key claims

The model is a 9B omni-modal LLM that jointly understands images, video, audio, and text in one architecture [Abstract].
Three architectural components are introduced for cross-modal fusion: (i) OmniAlignNet, which aligns vision and audio embeddings into a shared omni-modal latent space; (ii) Temporal Embedding Grouping, which encodes the relative time alignment between vision and audio signals; and (iii) Constrained Rotary Time Embedding, which encodes absolute temporal information [Abstract].
A curation and synthesis pipeline produces 24M single-modal and omni-modal conversations used for training [Abstract].
OmniVinci-9B reports +19.05 over Qwen2.5-Omni on DailyOmni cross-modal understanding, +1.7 on MMAR audio understanding, and +3.9 on Video-MME [Abstract].
Training token budget is 0.2T tokens — a claimed 6× reduction vs Qwen2.5-Omni’s 1.2T at comparable benchmark scores [Abstract].
The authors claim “modalities reinforce one another in both perception and reasoning,” i.e. joint audio+vision training is net-positive for each individual modality [Abstract].
Downstream evaluations are reported in robotics, medical AI, and smart factory settings [Abstract].

Method

The model stacks modality-specific encoders (vision and audio) on top of a language-model backbone derived from the NVILA codebase. The novel block sits between encoders and LLM: OmniAlignNet projects vision and audio features into a shared omni-modal latent before they’re handed to the LLM, so the LLM sees aligned representations rather than two disjoint embedding spaces. Temporal handling has two parts — Temporal Embedding Grouping batches vision and audio tokens so their relative time ordering is preserved during cross-modal attention, and Constrained Rotary Time Embedding adds an absolute time index that both modalities share, so a sound at $t = 3.0\,\mathrm{s}$ and a video frame at $t = 3.0\,\mathrm{s}$ inhabit the same temporal coordinate. The data pipeline produces 24M conversations spanning single-modality (image-only, audio-only, video-only) and omni-modal (audio+video together) examples.

Results

Headline numbers vs Qwen2.5-Omni at matched evaluation: DailyOmni +19.05, MMAR +1.7, Video-MME +3.9, all with 0.2T training tokens vs 1.2T for Qwen2.5-Omni (6× less compute on the tokens axis) [Abstract]. The paper additionally reports applied results in robotics, medical AI, and smart factory automation, presented as evidence that cross-modal context improves downstream decision accuracy [Abstract].

Why it’s interesting

OmniVinci is the third independent recipe on this wiki for an understanding-side omni-modal LLM — complementing Qwen3.5-Omni Technical Report (Qwen3.5-Omni’s Thinker-Talker MoE) and Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation (Ming-Flash-Omni’s sparse unified architecture). It is also a direct point of comparison for the audio-side reliability picture in Audio Hallucination Attacks: Probing the Reliability of Large Audio Language Models (AHA) (AHA), which shows that every audio-capable unified model evaluated — Qwen3-Omni at 95.72% ASR, Audio Flamingo 3 at 99.19%, GPT-4 Audio at 95.93% — hallucinates under adversarial implicit-query probes; OmniVinci is not in that evaluation set yet and is the natural next target. The 6×-fewer-tokens claim against Qwen2.5-Omni, if it survives external eval, also makes it interesting alongside Introducing the V-JEPA 2 world model and new benchmarks for physical reasoning and other “modalities reinforce each other” claims, but the burden of proof is high — Qwen2.5-Omni’s 1.2T is not a clean baseline for any specific architectural innovation here.