Skip to content
Bud's Wiki

Unified Multimodal Models

TL;DR

Section titled “TL;DR”

Unified Multimodal Models (UMMs) put visual understanding and generation in one architecture, with the ambition that the two capabilities mutually reinforce. Filed papers cluster into three paradigms — AR (Show-o, Janus, Janus-Pro), AR+MAR (Harmon), and AR+Diffusion (BAGEL, OpenUni, BLIP-3-o, MetaQuery, DuoGen) — plus a from-scratch tri-modal masked-diffusion route (Apple’s tri-modal MDM) and an encoder-free Mixture-of-Transformer route (NEO-unify, which removes both the vision encoder and the VAE). Open questions on this page: does generation supervision actually transfer to understanding (and via what targets), what’s the cleanest way to bolt a generator onto a frozen MLLM without destroying it, what training objective and scaling regime the tri-modal/text-image-audio variants want, and whether removing pretrained-encoder priors entirely (NEO-unify) actually pays at scale.

Key claims

Section titled “Key claims”
  • The dominant UMM post-training direction has been understanding → generation (RecA uses understanding-encoder features for reconstruction; SRUM scores generated images with the model’s understanding head; UniCorn distills understanding into generation), and the reverse — generation supervision improving understanding — was unexplored until UniMRG (Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation §1, §2).
  • Auxiliary generation of intrinsic visual representations (depth from Depth Anything V2, segmentation from SAM, pixel reconstruction) on top of standard VQA supervision improves UMM understanding without architectural changes or inference overhead — gains on OpenUni-3.6B: VSR +7.2, MMVP +3.0, HallusionBench +3.7 (Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation Table 1, Table 2).
    • Conflict / scope: UniG2U-Bench evaluates the inference-time G2U variant (Generate-then-Answer: emit a free-form intermediate image, then answer) across >30 UMMs and finds unified models generally underperform their base VLMs, with GtA usually making things worse — gains appear only on spatial / illusion / multi-round subtasks (UniG2U-Bench: Do Unified Models Advance Multimodal Understanding? §1, Findings). The two results together suggest the G2U effect is real but is mediated by training-time intrinsic-representation supervision rather than test-time free-form image emission.
  • Off-the-shelf UMMs fail to emit plausible depth or segmentation maps when prompted, defaulting to RGB-like reconstructions — direct evidence that the standard joint-training objective does not induce intrinsic geometric/structural representations (Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation Fig. 1, Fig. 2).
  • Pixel reconstruction alone (the RecA setting) restores generation quality after understanding-only SFT but provides no understanding gains; depth and segmentation are separately doing work in the ablation (Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation Table 3).
  • Representational capacity ceilings can block UMM post-training: Show-o’s 4,096-token VQ codebook caps how many intrinsic representations can be jointly hosted, yielding only marginal UniMRG gains and the same RecA-era limit (Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation §4.4).
  • A pretrained video DiT can serve as the image-generation head of a unified text+image system at SOTA T2I quality (GenEval 0.88), by treating all prior conversation images as preceding “video frames” stacked along the temporal axis (DuoGen: Towards General Purpose Interleaved Multimodal Generation §4, §4.1).
  • Decoupled two-stage training — instruction-tune the MLLM alone on curated interleaved dialogues, then freeze the MLLM and fine-tune only the connector + DiT on context-alignment data — is the cleanest recipe for “add image generation to an MLLM without breaking the MLLM” (DuoGen: Towards General Purpose Interleaved Multimodal Generation §4.2).
  • For from-scratch tri-modal (text + image + audio) UMMs trained as masked diffusion, compute-optimal token-per-parameter is higher than AR-LLM Chinchilla: a 3B MDM wants ~250B tokens vs Chinchilla’s ~60B, fitted from 262 isoFLOP runs with D∗∝N0.476D^* \propto N^{0.476} (The Design Space of Tri-Modal Masked Diffusion Models §5.3, Fig. 3).
  • Modality-specific tokenizers + a single unified vocabulary + modality-specific boundary/mask/task tokens are sufficient to train a tri-modal MDM from scratch with no per-modality heads (The Design Space of Tri-Modal Masked Diffusion Models §3, §3.1).
  • Inference defaults (noise schedule, CFG, temperature) for a unified MDM are modality-dependent — there is no single “right” setting across text, image, and audio (The Design Space of Tri-Modal Masked Diffusion Models §1 contributions).
  • A useful taxonomy of UMM systems for evaluation purposes: End-to-End (parameter-coupled, jointly trained — Bagel, UniPic2-Metaquery-9B), Decoupled (module-separated, stitched at inference — Bagel-two-stage, OneCAT), and Agentic (tool-use unification — GPT-4o + image tool); only the first two support strict base-VLM pairing for isolating G2U gains (UniG2U-Bench: Do Unified Models Advance Multimodal Understanding? §3.2).
  • G2U effects cluster structurally: perception tasks correlate with perception, logic/reasoning with logic/reasoning; UMMs built on the same base model behave alike, while UMMs sharing only architecture do not — pretraining data dominates over architectural choices (UniG2U-Bench: Do Unified Models Advance Multimodal Understanding? §1).
  • An encoder-free UMM is feasible: NEO-unify removes both the vision encoder and the VAE, taking pixels and text as direct I/O through a Mixture-of-Transformer with separate understanding and generation pathways, trained with AR cross-entropy on text + pixel flow matching on vision. At 2B / 90K pretraining steps it reaches 31.56 PSNR / 0.85 SSIM on MS COCO 2017 vs Flux VAE’s 32.65 / 0.91 — within ~1 PSNR without a dedicated autoencoder (NEO-unify: Building Native Multimodal Unified Models End to End).
  • In an MoT encoder-free design, joint understanding + generation training shows “minimal intrinsic conflict”: even at low data ratios and loss weights, understanding stays stable and generation converges faster, and a frozen understanding branch is sufficient for the generation pathway to perform ImgEdit-style editing at 3.32 (NEO-unify: Building Native Multimodal Unified Models End to End).
  • Floor baseline (frozen-everything-but-connector): A 17M-param MLP adaptor that turns a frozen Gemini Embedding 2 API output into virtual tokens for a frozen Qwen3-4B reaches 97% CIFAR-10 / 89% closed-set sentence STT per-task, but 0% on open-vocabulary LibriSpeech across every chunking and token-count variant — evidence that the shared embedding space encodes semantic categories but not phonetic content, and that closed-set STT numbers from connector-only stacks are cluster recognition rather than transcription (Can Gemini Embeddings Be a Multimodal Encoder for LLMs? (Krafton AI experiment)).
  • Decoupled-then-E2E for video output: OmniWeaving extends the taxonomy by training an MLLM + MMDiT + VAE stack in three stages — Stages 1–2 freeze the MLLM and finetune only MMDiT (Decoupled), Stage 3 unfreezes the MLLM and co-trains it with the MMDiT under a next-token-prediction loss on reasoning traces alongside the diffusion loss (E2E). The MLLM is also elevated from a passive feature extractor to an active reasoner that emits an enhanced prompt under “thinking mode” before generation, with multi-layer hidden states DeepStacked into the first three MMDiT conditioning layers (OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning §4.1, §4.2). First filed unified-generation model where the output modality is video.
  • Audio-side hallucination is the dominant failure mode on the understanding side of audio-capable UMMs: under AHA’s strongest probe (audio-injection + implicit + adversarial), every evaluated unified audio-capable model — Qwen3-Omni 95.72%, Audio Flamingo 3 99.19%, GPT-4 Audio 95.93%, Gemini 3 Pro 79.65%, Gemini 3.1 Pro 50.8% — hallucinates a sound it cannot hear (Audio Hallucination Attacks: Probing the Reliability of Large Audio Language Models (AHA) Table 1). Mean-attention analysis on Qwen2.5-Omni shows that implicit queries cause the model to attend less to audio tokens than explicit ones, giving a mechanistic correlate for the language-prior failure (Audio Hallucination Attacks: Probing the Reliability of Large Audio Language Models (AHA) Investigating the cause of hallucinations). Inference-time Chain-of-Thought can increase implicit-attack ASR; only DPO post-alignment on the AHA-Guard 120K-pair dataset cuts ASR (up to 49 points on implicit-text attacks) — but the audio-injection + implicit-adversarial worst case only drops from 96.71% to 86.63% (Audio Hallucination Attacks: Probing the Reliability of Large Audio Language Models (AHA) Table 2).

Recent contributions

Section titled “Recent contributions”
  • [2026-09-01] Atlas: A World Model for Spatial Intelligence: Atlas is a single omni model that natively operates on text, images, camera poses, and 3D depth maps in one AR-diffusion-transformer sequence; the same weights cover camera-controlled video generation, sparse-view 3D reconstruction, video reframing, robotics sensor simulation, and text-to-image / 360-panorama generation — every “task” is just a different input–output sequence template.
  • [2026-08-31] There and Back Again: Bidirectional Diffusion Bridges for Multimodality Translation: BIT (Ermon lab) proposes an SDE diffusion bridge whose two endpoints are the source and target modalities — one trained model handles text→image and image→text by simulating the same bridge in either direction. Adds a bidirectional-bridge branch to the UMM taxonomy, orthogonal to AR / AR+Diffusion / MoT; the architectural bet is that a source-aware generative path (endpoints are real modality samples, not Gaussian noise) removes the need for a separate captioner and enables sampler flexibility that noise-to-target conditional flows can’t match.
  • [2026-08-27] Gemini Omni 1.1 Flash lets you build with more control: First production iteration of Gemini Omni after the May 2026 launch — closed-flagship UMM competing on controllability (first/last-frame keyframes, video-reference conditioning, 40s scene extension, 360p→4K resolution tiers) rather than raw fidelity, exposed as parameters on a single gemini-omni-1.1-flash API surface with no architectural detail disclosed.
  • [2026-08-26] GLM-5.3-Flash — 320B/18B natively multimodal hybrid sparse+linear attention MoE with mHC: GLM-5.3-Flash is the GLM-5 series’ first natively multimodal model — 320B/18B-active MoE trained from scratch on a 30T-token multimodal corpus, contrast with the earlier bolt-on approach of GLM-5.2-Vision-NVFP4 — bolting MoonViT onto frozen GLM-5.2 with a 49.5M projector that projected MoonViT into frozen GLM-5.2.
  • [2026-08-20] The Multimodal Intelligence of Muse Spark 1.2: Muse Spark 1.2 (MSL) pitches a single multimodal base spanning visual coding (image/video → working web/game code), robotics planning + a Muse Spark VLA policy, and audio-visual understanding — with tool-augmented reasoning (web dev, real-time search, spatial grounding) framed as the load-bearing lift. Open-weights release imminent; no architecture or benchmark numbers disclosed.
  • [2026-08-14] Qwen3.8 open weights release — 27B dense multimodal + 2.4T-A95B MoE (Alibaba Qwen): Qwen3.8-27B ships as a dense native-multimodal model (as opposed to sparse MoE) — Alibaba’s continued push on the “dense mid-scale beats large-MoE on coding + office workflows” thesis, now at 262K native context (1M via YaRN). No benchmark table transcribed in the launch tweet.
  • [2026-08-14] Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction: GAS is the second filed generation-as-auxiliary-supervision recipe after UniMRG, but bets on continuous Next Embedding Prediction (NEP) inside a decoupled Mixture-of-Transformers — shared lower trunk absorbs generation gradients, parallel upper understanding layers are shielded — instead of UniMRG’s discrete depth/segmentation/pixel targets. Same core claim (auxiliary generation improves MLLM understanding on perception/spatial with zero inference overhead, since the generation branch is discarded post-training), different mechanism. Directly addresses the wiki’s open question on whether the effect generalizes beyond depth+segmentation targets: GAS answers “yes, via continuous-embedding targets that match what understanding layers consume natively.”
  • [2026-08-09] ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation: ToolArtist post-trains a UMM to place reasoning + external tool use + native image generation under one agent policy (via SFT that hides a teacher’s external image tool but keeps its images as native-generation targets, then RAD-GRPO with intent + quality rewards). Strictly more agentic than the E2E / Decoupled / Agentic branches in the UniG2U-Bench taxonomy — the “agentic” branch there still keeps tool orchestration external to the UMM, whereas ToolArtist puts the orchestration policy inside the UMM’s weights.
  • [2026-08-07] Scaling Native Multimodal Pre-Training From Scratch: Encoder-free native UMM (single patch-embedding projection into continuous image tokens, no ViT/VAE, MoE decoder-only trunk trained from scratch) with the first filed decoupled language-vs-multimodal isoFLOP scaling fit. Text capability is preserved across all multimodal ratios (<1 pt drift on 16-benchmark average, 71M → 3B active), and multimodal pretraining transfers positively to text-only spatial reasoning on SpatialEval, with the gap widening at scale. Multimodal in-context learning emerges at ≥A874M and continues strengthening with training data. Complements Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes‘s data-mixture allocation finding (L70/U25/G5) from the scaling-exponent side.
  • [2026-08-07] Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes: Meta FAIR + Oxford “physics of multimodal pretraining” — a controlled from-scratch study (same-lab successor to Beyond Language Modeling: An Exploration of Multimodal Pretraining) that names three general mechanisms: (i) knowledge flow is directional — language boosts all vision, understanding→generation is strong, generation→understanding is neutral except as a fine-tuning prior on low-level concepts; (ii) modality synergy vs. competition is governed by data complexity + component sharing (share attention/norm, decouple FFNs); (iii) vision laziness — a named late-alignment pathology where a longer language warm-up progressively shrinks vision-side FFN activation, image-wrapper token norms, and attention on image tokens, diagnosed on four axes. Derives an L70/U25/G5 asymmetric mix that at 13.5B-MoE / 2T tokens beats an L50/U25/G25 balanced recipe on every axis (PPL 11.67 vs 11.97, GenEval 0.482 vs 0.467, DPG 0.689 vs 0.676) while spending 5× fewer generation tokens. First filed direct empirical answer to the “how much of the budget should generation get” question — 5% suffices at frontier scale.
  • [2026-08-06] SenseNova U1.5-8B-MoT-Preview — Native 4K Generation and Editing on NEO-unify: SenseNova U1.5-8B-MoT-Preview scales the encoder-free NEO-unify recipe to 8B and 4K generation. Two deltas over U1: (i) ConvDecoder patch-joint reconstruction head (2D reshape + Pixel Shuffle + 3×3 conv interactions) replaces the per-patch MLP head to kill grid seams at high resolutions; (ii) heavily filtered/synthesized editing corpus with EN–CN balance. Reports Qwen-Image-Bench 48.28 → 49.93 (55.17 w/ PE), ImgEdit-Bench 3.90 → 4.37, GEdit-Bench-en 7.47 → 8.17. Concrete evidence that the “single sequence of reference tokens through frozen understanding pathway” channel is sufficient for identity/structure preservation without a dedicated reference encoder — but WeEdit average still trails Nano-Banana-Pro 6.85 vs 8.84, so encoder-free open UMMs at 8B lag closed-source frontier on composite editing.
  • [2026-08-06] Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding: Qwen-3D unifies referential grounding, instance segmentation, and VQA across both images and videos in one Qwen-backbone model by adding 3D Rotary Positional Embeddings to visual tokens (attention in 3D scene space rather than per-frame) and a query-based dense mask decoder that grounds language directly in the 3D representation. Joint 2D+3D training preserves standard 2D VLM capability — a video/spatial-perception extension of the unified-understanding recipe.
  • [2026-08-03] MiniMax H3 — Open-Weights Omni-Reference Video-Audio Generation Model Card: MiniMax H3 model card details a 33B dense single-stream Omni-Transformer that accepts text + image + video + audio and emits joint video + stereo audio, with modality-specific I/O layers and AdaLN but no modality-specific attention or FFN branches. Uses Qwen3-VL-32B as the frozen text/vision encoder tapped at layer 50 — an example of the increasingly common “frozen large VLM as encoder for a downstream generation trunk” pattern rather than end-to-end joint training.
  • [2026-07-31] Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers: Chimera unifies text, image, and video tokens in one raster-ordered stream with no positional embeddings and no per-modality heads — modality separation is carried by KDA + MLA + modality-aware short convolutions rather than by MoT-style expert routing. Extends the wiki’s UMM taxonomy to a hybrid-attention single-stream design specifically tuned for generation (11B/2B-active MoE diffusion backbone with three-axis compute-optimal scaling).
  • [2026-07-30] Introducing Inkling-Small — Thinking Machines' efficient open-weights MoE reasoning model: Inkling-Small inherits Inkling’s encoder-free multimodal design (dMel audio spectrograms, 40×40 hMLP image patches) at ¼ the parameters and nearly matches its multimodal performance — VoiceBench 90.1%, MMAU 77.0%, MMMU Pro 74.0%, Charxiv RQ 77.4/81.3% (without/with Python tool).
  • [2026-07-28] USP: Unified Self-Supervised Pretraining for Image Generation and Understanding: USP produces one ViT weight-set that initializes both a discriminative backbone (ImageNet linear probe 66.9% B / 74.5% L, ADE20K +0.5 mIoU over MAE) and a diffusion transformer (DiT-XL/2 FID 9.73 at 400K steps without CFG). Provides mechanistic evidence for why understanding→generation transfer is fragile: SFT-classification init gives FID 39.35 vs. self-supervised USP init 28.26 — over-specializing for discrimination erodes generation transfer, complementing UniMRG’s reverse-direction finding.
  • [2026-07-27] Kimi K3 Technical Report — Open Frontier Intelligence: Kimi K3 tech report — native multimodal MoE with MoonViT-V2 (401M) + lightweight projector into the shared embedding space; text/image/video handled by a single 2.8T backbone with 1M context.
  • [2026-07-23] GLM-5.2-Vision-NVFP4 — bolting MoonViT onto frozen GLM-5.2 with a 49.5M projector: Baseten’s GLM-5.2-Vision-NVFP4 demonstrates the “frozen-tower + trained connector” UMM recipe at 744B-active-MoE scale — a 49.5M PatchMerger MLP grafts Kimi-K2.6’s MoonViT onto GLM-5.2 without touching either backbone.
  • [2026-07-23] FLUX 3 — Real World Models: Towards Multimodal Flow Models as the Backbone of Visual Intelligence: FLUX 3 (BFL) is a single flow-matching unified multimodal backbone spanning image, video, audio, and action prediction natively — extending the wiki’s tracked unified-multimodal design space past text+image+video+audio into a fourth “action” modality integrated in the same architecture rather than bolted on. Positioned as the direct scale-up of Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis; released to early access as FLUX 3 Video with FLUX 3 Image, FLUX 3 Action, and open-weights FLUX 3 Dev on the roadmap.
  • [2026-07-22] Han Xiao — turning frozen jina-v5-omni into an open-vocabulary image tagger via test-time scaling: Han Xiao (Jina AI) previews turning frozen jina-v5-omni into an open-vocabulary multi-label n-gram image tagger with no training and no second model — another datapoint on “how much of multimodal capability sits in a single frozen embedding space,” directly opposite the Krafton finding that a frozen Gemini Embedding + trained adaptor hits 0% LibriSpeech open-vocab. Sits outside the AR / AR+Diffusion / MoT training taxonomy since nothing is trained.
  • [2026-07-20] Introducing Cosmos 3 Edge: Cosmos 3 Edge is a 4B unified backbone jointly handling vision+text understanding, video/audio generation, and native action tokens — the on-device variant of the flagship Cosmos 3 recipe, showing the unified-MM+action pattern remains structurally intact at edge scale.
  • [2026-07-19] RynnBrain 1.1 — Open Embodied Foundation Models (Alibaba DAMO Academy): RynnBrain 1.1 uses a single decoder-only Qwen3.5 / Qwen3-VL backbone across 2B / 9B / 122B-A10B scales that emits text, 2D boxes, pointing sequences, 3D bounding boxes (from RGB + intrinsics), and contact-point-plus-orientation tokens over a shared vocabulary — the output modalities are routed by instruction template rather than by separate task heads.
  • [2026-07-19] Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation: Self-SpectraReward generalizes the ABACUS (ABACUS: Adapting Unified Foundation Model for Bridging Image Count Understanding and Generation) cycle-consistent-GRPO recipe — understanding branch scoring its own generation branch — from narrow count-faithfulness to arbitrary T2I RL. The reward is a training-free teacher-forced prompt log-likelihood, so the closed loop needs no reward-model fine-tuning, no external annotators, and no preference labels — a UMM alone plus a diffusion policy is enough. Reports the negative-scaling result that same-model reward can match or beat much larger external reward MLLMs.
  • [2026-07-16] Kimi K3 — Open Frontier Intelligence (2.8T MoE with KDA + AttnRes): Kimi K3 ships as a natively multimodal (text + image + video) 2.8T MoE with 1M-token context, extending Moonshot’s Kimi K2.5 “visual agentic” line to a single unified frontier release.
  • [2026-07-16] RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imagination: RxBrain unifies language understanding/generation and vision understanding/generation in one 6.2B Mixture-of-Transformers via shared vision QKV projections but modality-routed FFN experts, with HY-ViT 2.0 as a single VAE-aligned vision encoder for both sides.
  • [2026-07-15] Introducing Inkling — Thinking Machines' open-weights 975B/41B MoE multimodal reasoning model: Inkling ships an encoder-free multimodal design at frontier scale: audio as discrete dMel spectrograms (Bai et al., 2024), images as 40×40 pixel patches through a 4-layer hMLP (Touvron et al., 2022), both projected via a lightweight embedding and processed jointly with text tokens — no frozen audio/vision encoder, no projector. Second frontier-scale datapoint for encoder-free after Gemma 4 12B’s 35M vision-embedding module, on a much larger backbone (975B total) with three input modalities.
  • [2026-07-14] Xiaomi-Robotics-U0: Unified Embodied Synthesis with World Foundation Model: Xiaomi-Robotics-U0 extends the UMM paradigm to embodied settings — a single 38B AR model over IBQ tokens jointly handling T2I, image editing, multi-view robot scene generation, structured embodied transfer, and embodied video rollout under one next-token loss, continually trained from EMU3.5.
  • [2026-07-13] InternVLA-A1.5: Unifying Understanding, Latent Foresight, and Action for Compositional Generalization: InternVLA-A1.5 (Shanghai AI Lab / InternRobotics) is a robotics-side datapoint for the “keep the VLM training live inside a unified model” position in the UniMRG debate: the native Qwen3.5-2B backbone continues to train on VQA and subtask prediction while a lightweight unified action expert attaches via shared full-attention layers, and the paper reports that this preservation of semantic priors is what delivers the strongest compositional generalization on held-out instruction bindings — where end-to-end unified designs that overwrite the VLM tend to erode compositional structure. Frames “understanding + generation + action” as one unified target rather than the standard “understanding + generation” pair, and adds foresight-token distillation from a frozen video-generation WFM as a UMM training signal alongside the depth/segmentation/pixel auxiliary generation of Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation.
  • [2026-07-13] Video Generation Models are General-Purpose Vision Learners: GenCeption is the video analog of Vision Banana: a pre-trained text-to-video diffusion backbone lightly instruction-tuned into a single feed-forward model that handles depth, surface normals, camera pose, expression-referring segmentation, and 3D keypoints — text-selected across tasks — with SOTA against specialists like DepthAnything3, SAM3, VGGT-Ω, and Sapiens. Extends the “generation is the unified output interface” position from 2D still-image tasks to inherently spatiotemporal ones (camera pose, 4D keypoints), and reports 7×–500× data efficiency vs. specialists on their home benchmarks.
  • [2026-07-10] Context Unrolling in Omni Models: Arxiv version of ByteDance Seed’s Omni — formalizes Context Unrolling as y = argmax p(y | Compose(x, φ_1(x), ...)) (Eq. 1) and provides the inline ablation tables (Table 2 T2I, Table 3 MMSI, Table 4 depth) that the project-page filing summarized. Confirms textual and visual contexts are non-redundant: even oracle Gemini-3-Pro text rewrites gain further from adding self-rollout visual tokens (52.20 → 57.21 on TIFAGM).
  • [2026-07-09] Introducing Muse Spark 1.1: Muse Spark 1.1 pitches native multimodal reasoning grounded in perception+action workflows — visual-to-code artifact generation, ultra-descriptive image/video captioning, and multimodal computer use — as core capabilities alongside coding and tool use.
  • [2026-07-08] Seedream 5.0 Pro — Multimodal Image Generation Model (ByteDance Seed): Seedream 5.0 Pro product page — ByteDance’s mid-2026 continuation of the Seedream cadence, pivoting the marketing axis from “unified gen+edit” toward information-dense multilingual generation with explicit layer separation as a first-class editing primitive. No technical disclosure; product-signal filing only.
  • [2026-07-08] Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models: First filed instance of a UMM (Bagel) used as a world model backbone for embodied VLA finetuning rather than for text-image understanding/generation. Actions are added as integer tokens over the existing vocabulary — no vocabulary or architectural change — and the UMM jointly emits pixel-level next-observations and a discrete reward token. Extends the “what UMMs are for” surface beyond the E2E / Decoupled / Agentic taxonomy on this page to a fourth use: pretrained-UMM-as-off-the-shelf-dynamics+reward-model for embodied RL. Also shows pretraining is load-bearing here — random-init UMM crashes reward F1 from 0.861 to 0.496.
  • [2026-07-07] Muse Video preview — MSL video generation model announcement (Alexandr Wang): Muse Image is an explicit counter-example to native-unified UMMs — a two-model pipeline where Muse Spark handles prompt reasoning / search / planning and Muse Image only generates. The architectural bet is the opposite of Emu3.5 / Ming-UniVision / Bagel and closer to a planner-generator split.
  • [2026-06-29] ABACUS: Adapting Unified Foundation Model for Bridging Image Count Understanding and Generation: ABACUS adapts a frozen 3B unified foundation model into a counting-specialist UMM that handles object/crowd/referring-expression counting and count-faithful generation; introduces cycle-consistent GRPO in which the understanding branch self-critiques generated outputs as the reward signal, closing the understanding-generation gap without external annotations — a UMM-internal alternative to the unified-reward-model pattern of Unified Reward Model for Multimodal Understanding and Generation.
  • [2026-06-23] Seed2.1: A Next-Generation Agent for Real-World Productivity: Seed2.1 closed-source frontier release with strong video numbers (VideoMME 89.2, TOMATO 79.5, Minerva 70.7, OVOBench 80.7, VideoSimpleQA 76.4) leading both Gemini 3.5 Flash and Gemini 3.1 Pro across streaming, motion/perception, reasoning, and knowledge axes; also MMLongBench-128K 78.3 vs Gemini 3.1 Pro 70.7 on multimodal long-context.
  • [2026-06-21] OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains: OmniVideo-100K is an instruction-tuning corpus targeted specifically at the audio-visual understanding leg of omni models (VITA-1.5, Qwen2.5-Omni-7B, Qwen3-Omni-30B), with a paired human-verified OmniVideo-Test benchmark; up to +20.59% on OmniVideo-Test and +12.64% on Daily-Omni / JointAVBench across all three architectures, suggesting AV-QA gains transfer cross-omni.
  • [2026-06-18] M*: A Modular, Extensible, Serving System for Multimodal Models: Stanford’s M* is the serving-side companion to the UMM taxonomy on this page. Where the page tracks UMM architectures (E2E / Decoupled / Agentic / encoder-free MoT), M* observes that all of them are composite — BAGEL has four heterogeneous components and per-request component subsets; Qwen3-Omni runs a Thinker → Talker → Code2Wav streaming pipeline; OmniWeaving stacks an MLLM, MMDiT, and VAE in three training stages — and so a serving framework that hard-codes the AR decode loop (vLLM, SGLang) or a flat stage DAG (vLLM-Omni, SGLang-Omni) loses 2.7–4× throughput vs a graph-of-components runtime with first-class Loop / Parallel / Stream primitives. The BAGEL example is particularly clean: image-generation requests never touch the ViT understanding path, and image-understanding requests never touch the diffusion loop or VAE decoder, because the Walk names exactly the components a request uses. Practical implication for the page: framework choice now shapes what UMM architectures are cheap to deploy — patterns that look expensive under a flat-DAG abstraction (per-Walk component subsets, three-way CFG fan-out, streaming-edge audio pipelines) are first-class under a Walk Graph.
  • [2026-05-26] Toward Native Multimodal Modeling: A Roadmap: NMM-Roadmap — curated dataset-side catalog of native multimodal modeling, decomposing the data question into Understand / Generate / Interact / Align axes with named corpora per modality (LAION-5B, MMC4, OBELICS, Open X-Embodiment, RLAIF-V, VBench, etc.). Complements the architectural taxonomy on this concept page by making the data-mixture question explicit.
  • [2026-05-24] Audio Hallucination Attacks: Probing the Reliability of Large Audio Language Models (AHA): AHA — first filed audio-side reliability probe for unified audio-capable models. 6.6K-QA evaluation suite (AHA-Eval) crosses three attack axes (query structure, language priors, audio injection) over AudioCaps and MusicCaps; documents 79–99% ASR on Qwen3-Omni, Audio Flamingo 3, AudioFlamingo-Next, MOSS-Audio-4B-Think, GPT-4 Audio, and Gemini 3 Pro under the strongest combination. Mitigation evidence: DPO on the paired 120K AHA-Guard corpus cuts Qwen2.5-Omni implicit-text ASR by ~30 points (68.74% → 39.01% on random, 79.19% → 40.24% on adversarial). Methodologically a benchmark + paired alignment dataset; sharper than the existing MMSU/MMAU numbers reported by Audio-Omni and Ming-flash-omni at separating models that are reliable from models that are confident.
  • [2026-05-24] OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning: Tencent Hunyuan’s OmniWeaving — MLLM + MMDiT + VAE built on HunyuanVideo-1.5, unifying T2V / I2V / V2V editing / key-frames-to-V / compositional multi-image-to-V / text-image-video-to-V in one architecture. Two architectural specifics: “thinking mode” — the MLLM autonomously emits an intermediate reasoning trace whose hidden states are forwarded alongside the original features; and DeepStacking — multi-layer MLLM hidden states added into the first three MMDiT conditioning layers. Three-stage training (frozen-MLLM alignment → frozen-MLLM multi-task → unfrozen-MLLM reasoning-augmented with NTP loss on reasoning traces). Adds a Decoupled-then-E2E hybrid to the UniG2U taxonomy and extends the page to video output.
  • [2026-05-23] Can Gemini Embeddings Be a Multimodal Encoder for LLMs? (Krafton AI experiment): Krafton AI / Kangwook Lee’s “Can Gemini Embeddings Be a Multimodal Encoder for LLMs?” — a 17M-param MLP adaptor over a frozen Gemini Embedding API feeding virtual tokens to a frozen Qwen3-4B. Strong on closed-set categorical tasks (CIFAR-10 97%, RAVDESS gender 99%) and shows zero-shot cross-modal transfer (image-trained CIFAR-10 adaptor → ESC-50 audio at 59.8%, 3.6× random); collapses to 0% on open-vocabulary LibriSpeech, exposing the closed-set “STT” success as cluster recognition over 169 known commands. A “frozen everything but the connector” data point outside the E2E / Decoupled / Agentic UMM taxonomy.
  • [2026-05-23] NEO-unify: Building Native Multimodal Unified Models End to End: NEO-unify (SenseTime × NTU) — an encoder-free UMM with no VE and no VAE, native pixel I/O, Mixture-of-Transformer backbone, AR cross-entropy on text + pixel flow matching on vision. 2B preview hits ~31.6 PSNR reconstruction (within 1 PSNR of Flux VAE) and 3.32 ImgEdit with frozen understanding branch. Reports better data-scaling efficiency than BAGEL. Adds an encoder-free MoT branch to the UMM taxonomy.
  • [2026-05-23] UniG2U-Bench: Do Unified Models Advance Multimodal Understanding?: 3000-sample / 30-subtask / 7-regime G2U benchmark that strictly pairs each of >30 UMMs against its base VLM under matched budgets; finds that unified training and Generate-then-Answer inference generally hurt understanding, with consistent gains only on spatial / illusion / multi-round subtasks. Introduces RA / AL metrics for intermediate-image quality and a three-class UMM taxonomy (E2E / Decoupled / Agentic).
  • [2026-05-23] Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation: introduces UniMRG, the first UMM post-training recipe that uses generation to improve understanding — auxiliary depth + segmentation + pixel generation gives +7.2 VSR / +3.7 Hallusion / +3.0 MMVP on OpenUni-3.6B without architectural changes.
  • [2026-05-23] The Design Space of Tri-Modal Masked Diffusion Models: Apple’s tri-modal (text + image + audio) masked-diffusion UMM trained from scratch; the paper’s main contribution is empirical — fitted multiplicative scaling laws (R2R^2=99.3%) showing tri-modal MDMs want more tokens per parameter than AR LLMs.
  • [2026-05-23] DuoGen: Towards General Purpose Interleaved Multimodal Generation: NVIDIA Cosmos Lab’s interleaved image-text UMM; bets that a video DiT is the right image-generation head and trains with a decoupled MLLM-first / DiT-second strategy that preserves the pretrained MLLM.

Open questions

Section titled “Open questions”
  • Does the “generation improves understanding” effect generalize beyond depth + segmentation? UniMRG explicitly leaves pose, sketches, normals for future work — and whether the effect scales to video UMMs is untested. OmniWeaving’s reasoning loss (NTP on reasoning traces alongside the diffusion loss in Stage 3) is the closest filed analogue in the video direction, but it’s an understanding-augments-generation recipe rather than generation-augments-understanding — the reverse arrow remains open for video UMMs.
  • What is the right capacity floor for a UMM to host multiple intrinsic-representation generation tasks? Show-o’s 4,096-token codebook fails; what is the codebook / hidden-width threshold above which the recipe pays?
  • Is the MLLM+DiT decoupled architecture (DuoGen, OpenUni, MetaQuery, BLIP-3-o) strictly better than fully joint AR (Janus-Pro, Show-o) for understanding metrics at matched scale, or is it confounded with backbone quality? OmniWeaving’s three-stage Decoupled→E2E recipe is a partial empirical answer in the video direction: full E2E is only used in Stage 3 for the reasoning-augmented loss, suggesting that as of mid-2026 even teams committed to E2E find Decoupled to be the cheaper recipe for the bulk of training compute.
  • For tri-modal/four-modal UMMs, does the masked-diffusion scaling law from The Design Space of Tri-Modal Masked Diffusion Models hold across architectures (MDM vs AR+Diffusion vs MoT), or is it specific to MDM?
  • Cross-modal scaling: does adding audio to a text-image UMM systematically harm or help text understanding? No filed paper directly tests this.
  • Why does inference-time GtA fail when training-time auxiliary generation succeeds? UniG2U attributes inference-time failure to inaccurate intermediate artifacts via the RA/AL metrics (UniG2U-Bench: Do Unified Models Advance Multimodal Understanding? §3.3), but the link to UniMRG’s training-time gains is not explicitly studied — is there a UMM that does well on both?
  • Does the encoder-free MoT approach (NEO-unify: Building Native Multimodal Unified Models End to End) scale beyond 2B preview, and does the data-scaling-efficiency advantage over BAGEL hold up under matched-compute eval suites (GenEval / MME / MMMU) — currently unpublished?
  • Does an encoder-free model evaluated under UniG2U-Bench show the same “unified hurts understanding” pattern, or does removing the pretrained-encoder prior change the conclusion?
  • What does the Gemini-Embedding floor baseline say about the embedding space? Can Gemini Embeddings Be a Multimodal Encoder for LLMs? (Krafton AI experiment) reports 0% LibriSpeech transcription through a frozen embedding API + frozen LLM, suggesting the embedding space exposes semantic categories but not phonetic content. Does the same hold for an open multimodal embedding stack like Qwen3-VL-Embedding, or is it a Gemini-Embedding-specific artifact of the retrieval-tuned training objective?
  • Should closed-set STT numbers in connector-only multimodal recipes be reported alongside open-vocabulary baselines? The 89% sentence STT → 0% LibriSpeech gap is a strong argument for yes, and would change how readers should interpret several recent “frozen LLM + projector” results.
  • Does OmniWeaving’s “thinking mode” reasoning trace actually pay off vs an enhanced-prompt-without-trace ablation? The architecture ties the reasoning trace into Stage 3 training jointly, so the trace’s contribution is not isolated from the unfreezing of the MLLM in the filed material.
  • Are audio-capable UMMs reliable on the understanding side, or do they hallucinate from language priors? AHA’s ~80–99% ASR on every evaluated model under the strongest probe says the standard MMSU/MMAU benchmarks dramatically over-report reliability. DPO on AHA-Guard halves implicit-attack ASR but doesn’t close it. Open: does the same vulnerability profile transfer to the generation side (e.g., does an audio-capable UMM emit audio content matching a hallucinated description if the implicit query is in the prompt), and is there a training-time fix that doesn’t require a paired-preference corpus?

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-08

Section titled “2025-08”

2025-07

Section titled “2025-07”

2025-06

Section titled “2025-06”

2025-05

Section titled “2025-05”

2025-04

Section titled “2025-04”

2025-03

Section titled “2025-03”

2025-02

Section titled “2025-02”

2025-01

Section titled “2025-01”

2024-12

Section titled “2024-12”

2024-05

Section titled “2024-05”

Unknown date

Section titled “Unknown date”