Skip to content
Bud's Wiki

Unified Multimodal Models

TL;DR

Section titled “TL;DR”

Unified Multimodal Models (UMMs) put visual understanding and generation in one architecture, with the ambition that the two capabilities mutually reinforce. Filed papers cluster into three paradigms — AR (Show-o, Janus, Janus-Pro), AR+MAR (Harmon), and AR+Diffusion (BAGEL, OpenUni, BLIP-3-o, MetaQuery, DuoGen) — plus a from-scratch tri-modal masked-diffusion route (Apple’s tri-modal MDM) and an encoder-free Mixture-of-Transformer route (NEO-unify, which removes both the vision encoder and the VAE). Open questions on this page: does generation supervision actually transfer to understanding (and via what targets), what’s the cleanest way to bolt a generator onto a frozen MLLM without destroying it, what training objective and scaling regime the tri-modal/text-image-audio variants want, and whether removing pretrained-encoder priors entirely (NEO-unify) actually pays at scale.

Key claims

Section titled “Key claims”
  • The dominant UMM post-training direction has been understanding → generation (RecA uses understanding-encoder features for reconstruction; SRUM scores generated images with the model’s understanding head; UniCorn distills understanding into generation), and the reverse — generation supervision improving understanding — was unexplored until UniMRG (Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation §1, §2).
  • Auxiliary generation of intrinsic visual representations (depth from Depth Anything V2, segmentation from SAM, pixel reconstruction) on top of standard VQA supervision improves UMM understanding without architectural changes or inference overhead — gains on OpenUni-3.6B: VSR +7.2, MMVP +3.0, HallusionBench +3.7 (Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation Table 1, Table 2).
    • Conflict / scope: UniG2U-Bench evaluates the inference-time G2U variant (Generate-then-Answer: emit a free-form intermediate image, then answer) across >30 UMMs and finds unified models generally underperform their base VLMs, with GtA usually making things worse — gains appear only on spatial / illusion / multi-round subtasks (UniG2U-Bench: Do Unified Models Advance Multimodal Understanding? §1, Findings). The two results together suggest the G2U effect is real but is mediated by training-time intrinsic-representation supervision rather than test-time free-form image emission.
  • Off-the-shelf UMMs fail to emit plausible depth or segmentation maps when prompted, defaulting to RGB-like reconstructions — direct evidence that the standard joint-training objective does not induce intrinsic geometric/structural representations (Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation Fig. 1, Fig. 2).
  • Pixel reconstruction alone (the RecA setting) restores generation quality after understanding-only SFT but provides no understanding gains; depth and segmentation are separately doing work in the ablation (Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation Table 3).
  • Representational capacity ceilings can block UMM post-training: Show-o’s 4,096-token VQ codebook caps how many intrinsic representations can be jointly hosted, yielding only marginal UniMRG gains and the same RecA-era limit (Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation §4.4).
  • A pretrained video DiT can serve as the image-generation head of a unified text+image system at SOTA T2I quality (GenEval 0.88), by treating all prior conversation images as preceding “video frames” stacked along the temporal axis (DuoGen: Towards General Purpose Interleaved Multimodal Generation §4, §4.1).
  • Decoupled two-stage training — instruction-tune the MLLM alone on curated interleaved dialogues, then freeze the MLLM and fine-tune only the connector + DiT on context-alignment data — is the cleanest recipe for “add image generation to an MLLM without breaking the MLLM” (DuoGen: Towards General Purpose Interleaved Multimodal Generation §4.2).
  • For from-scratch tri-modal (text + image + audio) UMMs trained as masked diffusion, compute-optimal token-per-parameter is higher than AR-LLM Chinchilla: a 3B MDM wants ~250B tokens vs Chinchilla’s ~60B, fitted from 262 isoFLOP runs with D∗∝N0.476D^* \propto N^{0.476} (The Design Space of Tri-Modal Masked Diffusion Models §5.3, Fig. 3).
  • Modality-specific tokenizers + a single unified vocabulary + modality-specific boundary/mask/task tokens are sufficient to train a tri-modal MDM from scratch with no per-modality heads (The Design Space of Tri-Modal Masked Diffusion Models §3, §3.1).
  • Inference defaults (noise schedule, CFG, temperature) for a unified MDM are modality-dependent — there is no single “right” setting across text, image, and audio (The Design Space of Tri-Modal Masked Diffusion Models §1 contributions).
  • A useful taxonomy of UMM systems for evaluation purposes: End-to-End (parameter-coupled, jointly trained — Bagel, UniPic2-Metaquery-9B), Decoupled (module-separated, stitched at inference — Bagel-two-stage, OneCAT), and Agentic (tool-use unification — GPT-4o + image tool); only the first two support strict base-VLM pairing for isolating G2U gains (UniG2U-Bench: Do Unified Models Advance Multimodal Understanding? §3.2).
  • G2U effects cluster structurally: perception tasks correlate with perception, logic/reasoning with logic/reasoning; UMMs built on the same base model behave alike, while UMMs sharing only architecture do not — pretraining data dominates over architectural choices (UniG2U-Bench: Do Unified Models Advance Multimodal Understanding? §1).
  • An encoder-free UMM is feasible: NEO-unify removes both the vision encoder and the VAE, taking pixels and text as direct I/O through a Mixture-of-Transformer with separate understanding and generation pathways, trained with AR cross-entropy on text + pixel flow matching on vision. At 2B / 90K pretraining steps it reaches 31.56 PSNR / 0.85 SSIM on MS COCO 2017 vs Flux VAE’s 32.65 / 0.91 — within ~1 PSNR without a dedicated autoencoder (NEO-unify: Building Native Multimodal Unified Models End to End).
  • In an MoT encoder-free design, joint understanding + generation training shows “minimal intrinsic conflict”: even at low data ratios and loss weights, understanding stays stable and generation converges faster, and a frozen understanding branch is sufficient for the generation pathway to perform ImgEdit-style editing at 3.32 (NEO-unify: Building Native Multimodal Unified Models End to End).
  • Floor baseline (frozen-everything-but-connector): A 17M-param MLP adaptor that turns a frozen Gemini Embedding 2 API output into virtual tokens for a frozen Qwen3-4B reaches 97% CIFAR-10 / 89% closed-set sentence STT per-task, but 0% on open-vocabulary LibriSpeech across every chunking and token-count variant — evidence that the shared embedding space encodes semantic categories but not phonetic content, and that closed-set STT numbers from connector-only stacks are cluster recognition rather than transcription (Can Gemini Embeddings Be a Multimodal Encoder for LLMs? (Krafton AI experiment)).
  • Decoupled-then-E2E for video output: OmniWeaving extends the taxonomy by training an MLLM + MMDiT + VAE stack in three stages — Stages 1–2 freeze the MLLM and finetune only MMDiT (Decoupled), Stage 3 unfreezes the MLLM and co-trains it with the MMDiT under a next-token-prediction loss on reasoning traces alongside the diffusion loss (E2E). The MLLM is also elevated from a passive feature extractor to an active reasoner that emits an enhanced prompt under “thinking mode” before generation, with multi-layer hidden states DeepStacked into the first three MMDiT conditioning layers (OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning §4.1, §4.2). First filed unified-generation model where the output modality is video.
  • Audio-side hallucination is the dominant failure mode on the understanding side of audio-capable UMMs: under AHA’s strongest probe (audio-injection + implicit + adversarial), every evaluated unified audio-capable model — Qwen3-Omni 95.72%, Audio Flamingo 3 99.19%, GPT-4 Audio 95.93%, Gemini 3 Pro 79.65%, Gemini 3.1 Pro 50.8% — hallucinates a sound it cannot hear (Audio Hallucination Attacks: Probing the Reliability of Large Audio Language Models (AHA) Table 1). Mean-attention analysis on Qwen2.5-Omni shows that implicit queries cause the model to attend less to audio tokens than explicit ones, giving a mechanistic correlate for the language-prior failure (Audio Hallucination Attacks: Probing the Reliability of Large Audio Language Models (AHA) Investigating the cause of hallucinations). Inference-time Chain-of-Thought can increase implicit-attack ASR; only DPO post-alignment on the AHA-Guard 120K-pair dataset cuts ASR (up to 49 points on implicit-text attacks) — but the audio-injection + implicit-adversarial worst case only drops from 96.71% to 86.63% (Audio Hallucination Attacks: Probing the Reliability of Large Audio Language Models (AHA) Table 2).

Recent contributions

Section titled “Recent contributions”
  • [2026-07-20] Introducing Cosmos 3 Edge: Cosmos 3 Edge is a 4B unified backbone jointly handling vision+text understanding, video/audio generation, and native action tokens — the on-device variant of the flagship Cosmos 3 recipe, showing the unified-MM+action pattern remains structurally intact at edge scale.
  • [2026-07-19] RynnBrain 1.1 — Open Embodied Foundation Models (Alibaba DAMO Academy): RynnBrain 1.1 uses a single decoder-only Qwen3.5 / Qwen3-VL backbone across 2B / 9B / 122B-A10B scales that emits text, 2D boxes, pointing sequences, 3D bounding boxes (from RGB + intrinsics), and contact-point-plus-orientation tokens over a shared vocabulary — the output modalities are routed by instruction template rather than by separate task heads.
  • [2026-07-19] Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation: Self-SpectraReward generalizes the ABACUS (ABACUS: Adapting Unified Foundation Model for Bridging Image Count Understanding and Generation) cycle-consistent-GRPO recipe — understanding branch scoring its own generation branch — from narrow count-faithfulness to arbitrary T2I RL. The reward is a training-free teacher-forced prompt log-likelihood, so the closed loop needs no reward-model fine-tuning, no external annotators, and no preference labels — a UMM alone plus a diffusion policy is enough. Reports the negative-scaling result that same-model reward can match or beat much larger external reward MLLMs.
  • [2026-07-16] Kimi K3 — Open Frontier Intelligence (2.8T MoE with KDA + AttnRes): Kimi K3 ships as a natively multimodal (text + image + video) 2.8T MoE with 1M-token context, extending Moonshot’s Kimi K2.5 “visual agentic” line to a single unified frontier release.
  • [2026-07-16] RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imagination: RxBrain unifies language understanding/generation and vision understanding/generation in one 6.2B Mixture-of-Transformers via shared vision QKV projections but modality-routed FFN experts, with HY-ViT 2.0 as a single VAE-aligned vision encoder for both sides.
  • [2026-07-15] Introducing Inkling — Thinking Machines' open-weights 975B/41B MoE multimodal reasoning model: Inkling ships an encoder-free multimodal design at frontier scale: audio as discrete dMel spectrograms (Bai et al., 2024), images as 40×40 pixel patches through a 4-layer hMLP (Touvron et al., 2022), both projected via a lightweight embedding and processed jointly with text tokens — no frozen audio/vision encoder, no projector. Second frontier-scale datapoint for encoder-free after Gemma 4 12B’s 35M vision-embedding module, on a much larger backbone (975B total) with three input modalities.
  • [2026-07-14] Xiaomi-Robotics-U0: Unified Embodied Synthesis with World Foundation Model: Xiaomi-Robotics-U0 extends the UMM paradigm to embodied settings — a single 38B AR model over IBQ tokens jointly handling T2I, image editing, multi-view robot scene generation, structured embodied transfer, and embodied video rollout under one next-token loss, continually trained from EMU3.5.
  • [2026-07-13] InternVLA-A1.5: Unifying Understanding, Latent Foresight, and Action for Compositional Generalization: InternVLA-A1.5 (Shanghai AI Lab / InternRobotics) is a robotics-side datapoint for the “keep the VLM training live inside a unified model” position in the UniMRG debate: the native Qwen3.5-2B backbone continues to train on VQA and subtask prediction while a lightweight unified action expert attaches via shared full-attention layers, and the paper reports that this preservation of semantic priors is what delivers the strongest compositional generalization on held-out instruction bindings — where end-to-end unified designs that overwrite the VLM tend to erode compositional structure. Frames “understanding + generation + action” as one unified target rather than the standard “understanding + generation” pair, and adds foresight-token distillation from a frozen video-generation WFM as a UMM training signal alongside the depth/segmentation/pixel auxiliary generation of Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation.
  • [2026-07-13] Video Generation Models are General-Purpose Vision Learners: GenCeption is the video analog of Vision Banana: a pre-trained text-to-video diffusion backbone lightly instruction-tuned into a single feed-forward model that handles depth, surface normals, camera pose, expression-referring segmentation, and 3D keypoints — text-selected across tasks — with SOTA against specialists like DepthAnything3, SAM3, VGGT-Ω, and Sapiens. Extends the “generation is the unified output interface” position from 2D still-image tasks to inherently spatiotemporal ones (camera pose, 4D keypoints), and reports 7×–500× data efficiency vs. specialists on their home benchmarks.
  • [2026-07-10] Context Unrolling in Omni Models: Arxiv version of ByteDance Seed’s Omni — formalizes Context Unrolling as y = argmax p(y | Compose(x, φ_1(x), ...)) (Eq. 1) and provides the inline ablation tables (Table 2 T2I, Table 3 MMSI, Table 4 depth) that the project-page filing summarized. Confirms textual and visual contexts are non-redundant: even oracle Gemini-3-Pro text rewrites gain further from adding self-rollout visual tokens (52.20 → 57.21 on TIFAGM).
  • [2026-07-09] Introducing Muse Spark 1.1: Muse Spark 1.1 pitches native multimodal reasoning grounded in perception+action workflows — visual-to-code artifact generation, ultra-descriptive image/video captioning, and multimodal computer use — as core capabilities alongside coding and tool use.
  • [2026-07-08] Seedream 5.0 Pro — Multimodal Image Generation Model (ByteDance Seed): Seedream 5.0 Pro product page — ByteDance’s mid-2026 continuation of the Seedream cadence, pivoting the marketing axis from “unified gen+edit” toward information-dense multilingual generation with explicit layer separation as a first-class editing primitive. No technical disclosure; product-signal filing only.
  • [2026-07-08] Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models: First filed instance of a UMM (Bagel) used as a world model backbone for embodied VLA finetuning rather than for text-image understanding/generation. Actions are added as integer tokens over the existing vocabulary — no vocabulary or architectural change — and the UMM jointly emits pixel-level next-observations and a discrete reward token. Extends the “what UMMs are for” surface beyond the E2E / Decoupled / Agentic taxonomy on this page to a fourth use: pretrained-UMM-as-off-the-shelf-dynamics+reward-model for embodied RL. Also shows pretraining is load-bearing here — random-init UMM crashes reward F1 from 0.861 to 0.496.
  • [2026-07-07] Muse Video preview — MSL video generation model announcement (Alexandr Wang): Muse Image is an explicit counter-example to native-unified UMMs — a two-model pipeline where Muse Spark handles prompt reasoning / search / planning and Muse Image only generates. The architectural bet is the opposite of Emu3.5 / Ming-UniVision / Bagel and closer to a planner-generator split.
  • [2026-06-29] ABACUS: Adapting Unified Foundation Model for Bridging Image Count Understanding and Generation: ABACUS adapts a frozen 3B unified foundation model into a counting-specialist UMM that handles object/crowd/referring-expression counting and count-faithful generation; introduces cycle-consistent GRPO in which the understanding branch self-critiques generated outputs as the reward signal, closing the understanding-generation gap without external annotations — a UMM-internal alternative to the unified-reward-model pattern of Unified Reward Model for Multimodal Understanding and Generation.
  • [2026-06-23] Seed2.1: A Next-Generation Agent for Real-World Productivity: Seed2.1 closed-source frontier release with strong video numbers (VideoMME 89.2, TOMATO 79.5, Minerva 70.7, OVOBench 80.7, VideoSimpleQA 76.4) leading both Gemini 3.5 Flash and Gemini 3.1 Pro across streaming, motion/perception, reasoning, and knowledge axes; also MMLongBench-128K 78.3 vs Gemini 3.1 Pro 70.7 on multimodal long-context.
  • [2026-06-21] OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains: OmniVideo-100K is an instruction-tuning corpus targeted specifically at the audio-visual understanding leg of omni models (VITA-1.5, Qwen2.5-Omni-7B, Qwen3-Omni-30B), with a paired human-verified OmniVideo-Test benchmark; up to +20.59% on OmniVideo-Test and +12.64% on Daily-Omni / JointAVBench across all three architectures, suggesting AV-QA gains transfer cross-omni.
  • [2026-06-18] M*: A Modular, Extensible, Serving System for Multimodal Models: Stanford’s M* is the serving-side companion to the UMM taxonomy on this page. Where the page tracks UMM architectures (E2E / Decoupled / Agentic / encoder-free MoT), M* observes that all of them are composite — BAGEL has four heterogeneous components and per-request component subsets; Qwen3-Omni runs a Thinker → Talker → Code2Wav streaming pipeline; OmniWeaving stacks an MLLM, MMDiT, and VAE in three training stages — and so a serving framework that hard-codes the AR decode loop (vLLM, SGLang) or a flat stage DAG (vLLM-Omni, SGLang-Omni) loses 2.7–4× throughput vs a graph-of-components runtime with first-class Loop / Parallel / Stream primitives. The BAGEL example is particularly clean: image-generation requests never touch the ViT understanding path, and image-understanding requests never touch the diffusion loop or VAE decoder, because the Walk names exactly the components a request uses. Practical implication for the page: framework choice now shapes what UMM architectures are cheap to deploy — patterns that look expensive under a flat-DAG abstraction (per-Walk component subsets, three-way CFG fan-out, streaming-edge audio pipelines) are first-class under a Walk Graph.
  • [2026-05-26] Toward Native Multimodal Modeling: A Roadmap: NMM-Roadmap — curated dataset-side catalog of native multimodal modeling, decomposing the data question into Understand / Generate / Interact / Align axes with named corpora per modality (LAION-5B, MMC4, OBELICS, Open X-Embodiment, RLAIF-V, VBench, etc.). Complements the architectural taxonomy on this concept page by making the data-mixture question explicit.
  • [2026-05-24] Audio Hallucination Attacks: Probing the Reliability of Large Audio Language Models (AHA): AHA — first filed audio-side reliability probe for unified audio-capable models. 6.6K-QA evaluation suite (AHA-Eval) crosses three attack axes (query structure, language priors, audio injection) over AudioCaps and MusicCaps; documents 79–99% ASR on Qwen3-Omni, Audio Flamingo 3, AudioFlamingo-Next, MOSS-Audio-4B-Think, GPT-4 Audio, and Gemini 3 Pro under the strongest combination. Mitigation evidence: DPO on the paired 120K AHA-Guard corpus cuts Qwen2.5-Omni implicit-text ASR by ~30 points (68.74% → 39.01% on random, 79.19% → 40.24% on adversarial). Methodologically a benchmark + paired alignment dataset; sharper than the existing MMSU/MMAU numbers reported by Audio-Omni and Ming-flash-omni at separating models that are reliable from models that are confident.
  • [2026-05-24] OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning: Tencent Hunyuan’s OmniWeaving — MLLM + MMDiT + VAE built on HunyuanVideo-1.5, unifying T2V / I2V / V2V editing / key-frames-to-V / compositional multi-image-to-V / text-image-video-to-V in one architecture. Two architectural specifics: “thinking mode” — the MLLM autonomously emits an intermediate reasoning trace whose hidden states are forwarded alongside the original features; and DeepStacking — multi-layer MLLM hidden states added into the first three MMDiT conditioning layers. Three-stage training (frozen-MLLM alignment → frozen-MLLM multi-task → unfrozen-MLLM reasoning-augmented with NTP loss on reasoning traces). Adds a Decoupled-then-E2E hybrid to the UniG2U taxonomy and extends the page to video output.
  • [2026-05-23] Can Gemini Embeddings Be a Multimodal Encoder for LLMs? (Krafton AI experiment): Krafton AI / Kangwook Lee’s “Can Gemini Embeddings Be a Multimodal Encoder for LLMs?” — a 17M-param MLP adaptor over a frozen Gemini Embedding API feeding virtual tokens to a frozen Qwen3-4B. Strong on closed-set categorical tasks (CIFAR-10 97%, RAVDESS gender 99%) and shows zero-shot cross-modal transfer (image-trained CIFAR-10 adaptor → ESC-50 audio at 59.8%, 3.6× random); collapses to 0% on open-vocabulary LibriSpeech, exposing the closed-set “STT” success as cluster recognition over 169 known commands. A “frozen everything but the connector” data point outside the E2E / Decoupled / Agentic UMM taxonomy.
  • [2026-05-23] NEO-unify: Building Native Multimodal Unified Models End to End: NEO-unify (SenseTime × NTU) — an encoder-free UMM with no VE and no VAE, native pixel I/O, Mixture-of-Transformer backbone, AR cross-entropy on text + pixel flow matching on vision. 2B preview hits ~31.6 PSNR reconstruction (within 1 PSNR of Flux VAE) and 3.32 ImgEdit with frozen understanding branch. Reports better data-scaling efficiency than BAGEL. Adds an encoder-free MoT branch to the UMM taxonomy.
  • [2026-05-23] UniG2U-Bench: Do Unified Models Advance Multimodal Understanding?: 3000-sample / 30-subtask / 7-regime G2U benchmark that strictly pairs each of >30 UMMs against its base VLM under matched budgets; finds that unified training and Generate-then-Answer inference generally hurt understanding, with consistent gains only on spatial / illusion / multi-round subtasks. Introduces RA / AL metrics for intermediate-image quality and a three-class UMM taxonomy (E2E / Decoupled / Agentic).
  • [2026-05-23] Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation: introduces UniMRG, the first UMM post-training recipe that uses generation to improve understanding — auxiliary depth + segmentation + pixel generation gives +7.2 VSR / +3.7 Hallusion / +3.0 MMVP on OpenUni-3.6B without architectural changes.
  • [2026-05-23] The Design Space of Tri-Modal Masked Diffusion Models: Apple’s tri-modal (text + image + audio) masked-diffusion UMM trained from scratch; the paper’s main contribution is empirical — fitted multiplicative scaling laws (R2R^2=99.3%) showing tri-modal MDMs want more tokens per parameter than AR LLMs.
  • [2026-05-23] DuoGen: Towards General Purpose Interleaved Multimodal Generation: NVIDIA Cosmos Lab’s interleaved image-text UMM; bets that a video DiT is the right image-generation head and trains with a decoupled MLLM-first / DiT-second strategy that preserves the pretrained MLLM.

Open questions

Section titled “Open questions”
  • Does the “generation improves understanding” effect generalize beyond depth + segmentation? UniMRG explicitly leaves pose, sketches, normals for future work — and whether the effect scales to video UMMs is untested. OmniWeaving’s reasoning loss (NTP on reasoning traces alongside the diffusion loss in Stage 3) is the closest filed analogue in the video direction, but it’s an understanding-augments-generation recipe rather than generation-augments-understanding — the reverse arrow remains open for video UMMs.
  • What is the right capacity floor for a UMM to host multiple intrinsic-representation generation tasks? Show-o’s 4,096-token codebook fails; what is the codebook / hidden-width threshold above which the recipe pays?
  • Is the MLLM+DiT decoupled architecture (DuoGen, OpenUni, MetaQuery, BLIP-3-o) strictly better than fully joint AR (Janus-Pro, Show-o) for understanding metrics at matched scale, or is it confounded with backbone quality? OmniWeaving’s three-stage Decoupled→E2E recipe is a partial empirical answer in the video direction: full E2E is only used in Stage 3 for the reasoning-augmented loss, suggesting that as of mid-2026 even teams committed to E2E find Decoupled to be the cheaper recipe for the bulk of training compute.
  • For tri-modal/four-modal UMMs, does the masked-diffusion scaling law from The Design Space of Tri-Modal Masked Diffusion Models hold across architectures (MDM vs AR+Diffusion vs MoT), or is it specific to MDM?
  • Cross-modal scaling: does adding audio to a text-image UMM systematically harm or help text understanding? No filed paper directly tests this.
  • Why does inference-time GtA fail when training-time auxiliary generation succeeds? UniG2U attributes inference-time failure to inaccurate intermediate artifacts via the RA/AL metrics (UniG2U-Bench: Do Unified Models Advance Multimodal Understanding? §3.3), but the link to UniMRG’s training-time gains is not explicitly studied — is there a UMM that does well on both?
  • Does the encoder-free MoT approach (NEO-unify: Building Native Multimodal Unified Models End to End) scale beyond 2B preview, and does the data-scaling-efficiency advantage over BAGEL hold up under matched-compute eval suites (GenEval / MME / MMMU) — currently unpublished?
  • Does an encoder-free model evaluated under UniG2U-Bench show the same “unified hurts understanding” pattern, or does removing the pretrained-encoder prior change the conclusion?
  • What does the Gemini-Embedding floor baseline say about the embedding space? Can Gemini Embeddings Be a Multimodal Encoder for LLMs? (Krafton AI experiment) reports 0% LibriSpeech transcription through a frozen embedding API + frozen LLM, suggesting the embedding space exposes semantic categories but not phonetic content. Does the same hold for an open multimodal embedding stack like Qwen3-VL-Embedding, or is it a Gemini-Embedding-specific artifact of the retrieval-tuned training objective?
  • Should closed-set STT numbers in connector-only multimodal recipes be reported alongside open-vocabulary baselines? The 89% sentence STT → 0% LibriSpeech gap is a strong argument for yes, and would change how readers should interpret several recent “frozen LLM + projector” results.
  • Does OmniWeaving’s “thinking mode” reasoning trace actually pay off vs an enhanced-prompt-without-trace ablation? The architecture ties the reasoning trace into Stage 3 training jointly, so the trace’s contribution is not isolated from the unfreezing of the MLLM in the filed material.
  • Are audio-capable UMMs reliable on the understanding side, or do they hallucinate from language priors? AHA’s ~80–99% ASR on every evaluated model under the strongest probe says the standard MMSU/MMAU benchmarks dramatically over-report reliability. DPO on AHA-Guard halves implicit-attack ASR but doesn’t close it. Open: does the same vulnerability profile transfer to the generation side (e.g., does an audio-capable UMM emit audio content matching a hallucinated description if the implicit query is in the prompt), and is there a training-time fix that doesn’t require a paired-preference corpus?

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-08

Section titled “2025-08”

2025-07

Section titled “2025-07”

2025-06

Section titled “2025-06”

2025-05

Section titled “2025-05”

2025-04

Section titled “2025-04”

2025-03

Section titled “2025-03”

2025-02

Section titled “2025-02”

2025-01

Section titled “2025-01”

2024-12

Section titled “2024-12”

2024-05

Section titled “2024-05”

Unknown date

Section titled “Unknown date”