NEO-unify: Building Native Multimodal Unified Models End to End

TL;DR

NEO-unify (SenseTime × NTU) is an encoder-free unified multimodal model: no vision encoder, no VAE. It pairs a “near-lossless visual interface” (pixels and words go in directly) with a Mixture-of-Transformer (MoT) backbone that synergizes understanding and generation, trained jointly with autoregressive cross-entropy on text and pixel flow matching on vision. The preview release reports 31.56 PSNR / 0.85 SSIM on MS COCO 2017 (vs Flux VAE 32.65 / 0.91) after 90K pretraining steps, 3.32 on ImgEdit after 60K mixed steps with the understanding branch frozen, and substantially better data-scaling efficiency than BAGEL. It positions itself as a step toward UMMs whose representation space is shaped end-to-end by the model rather than inherited from pretrained VE/VAE priors.

Key claims

The system removes both the vision encoder and the VAE entirely, taking pixels and text directly as inputs and outputs through a “near-lossless visual interface” [§NEO-unify].
Backbone is a native Mixture-of-Transformer (MoT) with separate “understanding” and “generation” pathways that share the same backbone; training mixes AR cross-entropy for text with pixel flow matching for vision [§NEO-unify].
After ~90K pretraining steps the 2B model reaches 31.56 PSNR / 0.85 SSIM on MS COCO 2017, vs Flux VAE 32.65 / 0.91 — within ~1 PSNR of a dedicated VAE without using one [§Key Findings 1].
A separate generative pathway can recover fine-grained pixel detail even when the understanding pathway is frozen — same model, two roles, no shared encoder [§Key Findings 1].
With the understanding branch frozen, the model still does image editing: 3.32 on ImgEdit after 60K mixed steps using public T2I + editing datasets, while “largely improving token efficiency” [§Key Findings 1, Image Editing].
Joint training of understanding and generation in the MoT backbone shows minimal intrinsic conflict: even at low data ratios and loss weights, understanding remains stable and generation converges faster [§Key Findings 2].
NEO-unify shows higher data-scaling efficiency than BAGEL across web-scale pretraining → mid-training → SFT, reaching higher performance with fewer training tokens [§Key Findings 3].
Builds on the earlier NEO paper (Diao et al., ICLR 2026) which established that a native end-to-end model can learn rich semantic representations from pixels [§Key Findings 1].

Method

NEO-unify takes pixels and text as direct inputs through what the post calls a “near-lossless visual interface” — replacing both the vision encoder (typically CLIP / SigLIP for understanding) and the VAE (typically Flux-style for generation) with raw pixel I/O. The backbone is a Mixture-of-Transformer (MoT): two pathways — one for understanding, one for generation — share the same transformer infrastructure but route through separate parameters. Training mixes two objectives over the same network: autoregressive cross-entropy for text tokens, and pixel-space flow matching for vision targets.

In the image-editing experiments, the system routes all conditioning context (source image + edit instructions) through the understanding pathway, while the generation pathway directly produces the new image. The understanding branch is held frozen, so any editing capability comes from the generation pathway’s ability to read the frozen understanding-side representations. Pretraining proceeds web-scale, followed by mid-training (MT) and supervised fine-tuning (SFT) on diverse high-quality corpora.

Results

Reconstruction (2B, 90K pretraining steps, MS COCO 2017): 31.56 PSNR / 0.85 SSIM, vs Flux VAE at 32.65 / 0.91 — comparable pixel fidelity without a dedicated autoencoder [§Key Findings 1].
Editing (2B, 60K mixed-training steps, frozen understanding branch): 3.32 on ImgEdit using only public T2I + editing data [§Image Editing].
Joint training stability: Both branches co-evolve in the MoT backbone with “minimal conflict”; understanding remains stable and generation converges faster even at low data ratios / loss weights [§Key Findings 2].
Data-scaling efficiency: Reported “substantially better” than BAGEL across the pretraining → MT → SFT pipeline at matched token counts [§Key Findings 3]. The post does not publish the underlying scaling-curve numbers.

The post is a preview announcement; quantitative tables and per-task benchmark suites (GenEval, MME, MMMU, etc.) are not included in the released text. Model weights are promised “hopefully not too long.”

Why it’s interesting

NEO-unify is the third architectural family the wiki has now seen for unified multimodal models, alongside the AR / AR+MAR / AR+Diffusion taxonomy already catalogued under Unified Multimodal Models: a Mixture-of-Transformer with no tokenizer at either end. It directly challenges the design assumption behind DuoGen: Towards General Purpose Interleaved Multimodal Generation (which bets on a frozen MLLM + DiT decoupled stack) by going the opposite direction — one backbone, two pathways, no encoder. If the data-scaling claim against BAGEL holds up under independent evaluation, it would also be a counterpoint to The Design Space of Tri-Modal Masked Diffusion Models‘s position that tri-modal MDMs scale by increasing token-per-parameter — NEO-unify suggests the right move may be to remove the tokenizer-induced representation bottleneck entirely. Connection to Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation is also live: NEO-unify’s “frozen understanding + active generation” editing result is the inverse of UniMRG’s “generation helps understanding” recipe, and the MoT split makes the directionality cleanly testable.