Representation Forcing for Bottleneck-Free Unified Multimodal Models

TL;DR

Representation Forcing (RF) is a recipe for removing the frozen pretrained VAE from a unified multimodal model without paying the usual quality penalty. Instead of letting the decoder predict pixels directly from text tokens, RF forces it to autoregressively emit a sequence of visual-representation tokens first; those representation tokens then stay in the context and condition pixel diffusion inside the same backbone. The result is an end-to-end pixel-space UMM that matches state-of-the-art VAE-based unified models on image generation while outperforming a VAE-based variant on understanding benchmarks — empirical support for the thesis that the VAE bottleneck was holding both heads back.

Key claims

Existing unified multimodal models still depend on a separately pretrained, frozen VAE for image generation, which imposes a structural bottleneck on the unified architecture [§Abstract].
Naively removing the VAE creates a quality gap because the model must simultaneously learn high-level structure and low-level pixel detail from raw pixels [§Abstract].
Representation Forcing closes that gap by making representation prediction a native capability of the model: the decoder autoregressively predicts visual representation tokens as intermediate outputs before pixels, and those tokens stay in context as conditioning for the subsequent pixel-diffusion head — all within one backbone [§Abstract].
Predicting representations as a generation target (rather than only consuming them as a perception output) eliminates the need for any external generative latent space [§Abstract].
On image generation, RF in pixel space matches state-of-the-art VAE-based unified models [§Abstract].
On image understanding, the pixel-space RF model generally outperforms its VAE-based variant [§Abstract].

Method

RF keeps the architecture of a unified multimodal model essentially intact — one shared backbone, text tokens on input and output — but introduces a two-stage autoregressive-then-diffuse generation prefix on the visual side. When the model is asked to generate an image, it does not predict pixels directly from the text prompt. It first emits a sequence of intermediate “representation tokens” autoregressively. Those tokens are intended to be a learned analog of the kind of high-level features a SigLIP / DINOv2 encoder would extract from the target image — except produced from scratch by the same backbone, conditioned on the prompt. Once the representation prefix is in context, a pixel-diffusion head (sharing the backbone) generates the actual pixels conditioned on both the text and the predicted representations.

This means the representation tokens play two roles that the paper treats as the central design point. As generation targets they give the model an intermediate, lower-frequency objective that decouples “what is in the scene” from “what does it look like at the pixel level,” which the paper argues is the fix for the pixel-from-scratch quality gap. As conditioning they stay in the autoregressive context, so the pixel-diffusion step gets a structured, model-produced prior rather than just text and noise. Crucially, no external encoder/decoder is in the loop: the representation tokens are predicted by the same model that consumes them.

The recipe is thus the inverse of the standard UMM pipeline. Standard UMMs use a frozen vision encoder as a perception component (encode image → use features for understanding); RF reuses that same notion of “representation,” but turns it into a generation component (predict features → use them to drive pixel diffusion). The paper’s framing is that this turns representation prediction from a one-way perceptual operation into a native capability the model can both produce and consume — which is what makes the encoder removal viable.

Results

The paper’s headline results are stated comparatively rather than as raw scores in the abstract. Against VAE-based unified models, the pixel-space RF model matches state-of-the-art on image generation [§Abstract]. Against an internally-controlled VAE-based variant of itself, pixel-space RF generally outperforms on image understanding [§Abstract]. The framing is that both heads benefit: generation reaches parity with the VAE-based incumbents without using a VAE, and understanding gets actively better when the VAE is removed and replaced by the representation-forcing prefix.

The abstract does not commit to specific benchmark numbers, so the load-bearing empirical claim that can be repeated cleanly here is the direction: removing the VAE under the RF recipe is a Pareto improvement (better understanding, no worse generation) over keeping the VAE in this paper’s experimental setup. Quantitative comparisons against external SOTA UMMs and the controlled VAE-based variant are reported in the body.

Why it’s interesting

This is the third filed datapoint in a short run of “remove the VAE from the unified multimodal model” papers, and it isolates a different lever than its siblings. Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation removes both the VAE and the pretrained vision encoder, and uses a masking-based feature-learning scheme (random input-patch masking, predict clean pixels at masked + unmasked positions) to stabilize training. NEO-unify: Building Native Multimodal Unified Models End to End (NEO-unify) takes a Mixture-of-Transformer route with separate understanding and generation pathways, also encoder-and-VAE-free. RF keeps the same unified backbone but inserts an autoregressive representation prefix as a load-bearing intermediate target — a structurally different fix for the same “VAE is a bottleneck” thesis, and the closest filed analog to using RAE-style features but produced internally rather than consumed from a frozen external encoder.

The link to Pixel-space diffusion is direct: like PixelGen, CAFM, and Tuna-2, RF runs pixel-level generation under a JiT-style recipe, but with an autoregressive representation prefix as the new load-bearing trick (instead of perceptual losses or a learned adversarial criterion). The representation-tokens-as-target framing also rhymes with Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation‘s UniMRG (auxiliary depth/segmentation generation improves understanding) — both argue that having the UMM generate structured representations is what makes understanding improve. UniMRG keeps the VAE and predicts depth/segmentation tiles; RF removes the VAE and predicts learned representation tokens that double as the pixel-diffusion prior. Worth comparing whether the understanding-side gains in the two papers come from the same mechanism.