Skip to content
Bud's Wiki

Toward Native Multimodal Modeling: A Roadmap

TL;DR

Section titled “TL;DR”

NMM-Roadmap is a curated landing page that frames native multimodal modeling (NMM) as four capability axes — Understand, Generate, Interact, Align — and tabulates the standard dataset corpora used to train and evaluate each axis across text, image, video, audio, GUI, and robot modalities. It is not a model release or a survey paper; it is a working reference list that maps the “what data does a from-scratch omni-model actually need” question onto named datasets (LAION-5B, MMC4, OBELICS, OmniCorpus, WebShop, Mind2Web, Open X-Embodiment, RLAIF-V, VBench, etc.) grouped by purpose. Useful as a checklist when planning an NMM data mixture or auditing what an existing model was actually pretrained on.

Key claims

Section titled “Key claims”
  • Native multimodal modeling decomposes into four orthogonal capability axes — Understand, Generate, Interact, Align — and the page organizes datasets accordingly [§Top-level table].
  • The “Understand” axis spans five sub-categories with named corpora: image-text alignment (LAION-5B, COCO Captions, CC3M/CC12M, YFCC100M, DataComp), VQA & instruction tuning (VQA v2, GQA, OK-VQA, ScienceQA, LLaVA-Instruct, InstructBLIP), interleaved & multi-image (MMC4, OBELICS, OmniCorpus, MANTIS, NLVR2, MuirBench, BLINK), document/chart/grounding (DocVQA, InfographicVQA, ChartQA, TextVQA, Flickr30k Entities, RefCOCO), and video/audio understanding (MSR-VTT, ActivityNet, WebVid, AudioSet, LibriSpeech, Common Voice, Clotho) [§Understand].
  • The “Generate” axis covers T2I+editing (LAION-5B, DiffusionDB, InstructPix2Pix, MagicBrush, HQ-Edit, UltraEdit), controllable generation (ControlNet, GLIGEN, T2I-Adapter, Composer as conditioning sources), interleaved image-text gen (VIST, OpenLEAF, CoMM, InterSyn), video generation (WebVid-10M, Panda-70M, OpenVid-1M, VidGen-1M), and audio/speech generation (LibriTTS, VCTK, GigaSpeech, Emilia, AudioCaps, WavCaps, MusicCaps) [§Generate].
  • The “Interact” axis treats web (WebShop, Mind2Web, WebArena, VisualWebArena, WebLINX, WebVoyager), mobile/desktop GUI (AITW, RICO, ScreenAI, SeeClick, OSWorld, Windows Agent Arena), and embodied (ALFWorld, BridgeData V2, Open X-Embodiment, Magma) as the three GUI/world-interaction corpora families [§Interact].
  • The “Align” axis splits along hallucination/faithfulness (LLaVA-RLHF, RLHF-V, VLFeedback, RLAIF-V, HA-DPO, V-DPO), safety (SPA-VL, Safe RLHF-V), generation-quality preference (ImageReward, Pick-a-Pic, HPS v2, VBench, VBench++), and agentic preference (environment rewards + human demos) [§Align].
  • Modalities are tagged per row as T (text), I (image), V (video), A (audio), GUI, robot — making the modality coverage of each corpus explicit in the table [§Top-level table].

Method

Section titled “Method”

The artifact is a single-page HTML table that cross-products the four capability axes against modality-conditioned dataset families. Each row names the standard corpora researchers reach for, lists which modalities the data covers, and gives a one-sentence purpose (“Weakly-aligned web-scale pairs for cross-modal mapping”, “Goal-driven web navigation: search, click, form fill on real/sim sites”, etc.). There is no model, no benchmark numbers, no scaling laws, and no architectural prescription — just a structured inventory of training corpora keyed by capability.

Results

Section titled “Results”

This is a roadmap/catalog page, not an experimental artifact. Its value is the taxonomy itself: a from-scratch NMM team can use it as a checklist for what data buckets to source from, and a reader can use it to audit which capability axis an existing omni-model is data-starved on. No quantitative results are reported on the page.

Why it’s interesting

Section titled “Why it’s interesting”

The wiki already has a deep concept page on unified multimodal models (Unified Multimodal Models) that tracks the architectural side (E2E / Decoupled / Agentic / encoder-free MoT) and a long tail of papers documenting individual NMM model releases (Emu3.5, Ming-Flash-Omni, LongCat-Flash-Omni, NEO-Unify, Qwen3.5-Omni). This roadmap complements all of them by being the data counterpart — none of the filed UMM papers spells out which corpora to source for each capability axis. The Understand-Generate-Interact-Align decomposition also sits adjacent to NEO-unify: Building Native Multimodal Unified Models End to End‘s “build one that was never divided” framing and to 2504.07951-scaling-laws-native-multimodal-models (Shukor et al., Apple)‘s scaling-laws study of from-scratch NMMs — both works leave the data-mixture question implicit and this page makes the corpus side explicit. Less useful as a research artifact in itself; more useful as a sourcing reference when planning data mixtures.

See also

Section titled “See also”