Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction

TL;DR

GAS is a training-only framework that uses visual generation as auxiliary supervision to improve the understanding capability of pretrained MLLMs, with zero inference overhead — the generation branch is discarded after training. Instead of using discrete visual tokens or diffusion targets (whose objectives diverge from the continuous embeddings MLLMs actually consume), GAS reinterprets generation as Next Embedding Prediction (NEP) in the continuous representation space, and hosts it inside a decoupled Mixture-of-Transformers (MoT) where a shared lower trunk absorbs generation gradients while the upper understanding layers are shielded from them. The most reliable gains show up on perception and spatial comprehension across model scales and training stages. Directly relevant to the ongoing “does generation supervision transfer to understanding” debate on the wiki, and complementary to UniMRG’s depth/segmentation-map recipe.

Key claims

Existing unified frameworks tie generation to discrete visual tokenization or diffusion targets, whose generative objectives differ from the continuous representations that visual understanding models consume, making direct transfer to enhance pretrained MLLMs non-trivial [Abstract].
GAS reinterprets visual generation as auxiliary supervision for representation learning rather than as a downstream output capability [Abstract, §1].
The generation paradigm is Next Embedding Prediction (NEP) — cross-modal generation in continuous embedding space — chosen precisely to match what MLLM understanding layers already consume [Abstract, §3].
Architecture is a decoupled Mixture-of-Transformers: a shared lower trunk receives both understanding and generation gradients, while parallel upper layers keep the understanding pathway shielded from direct generation gradients [Abstract, §3].
The design intent is to let generation losses enrich the shared visual pathway with finer spatial precision and stronger visual retention without polluting the upper understanding layers [Abstract].
The paper further constructs highly correlated generation tasks that require deep cognitive grounding rather than generic synthesis, arguing task choice controls whether generation helps understanding [Abstract, §3].
Across model scales and training stages, GAS improves aggregate multimodal understanding, with the most reliable gains on perception and spatial comprehension [Abstract, §4].
Auxiliary generation branch is discarded after training — inference cost is identical to the base MLLM [Abstract].
Controlled comparisons and representation-level analyses are presented as evidence for when and why generation-guided training benefits understanding [Abstract, §5].

Method

GAS starts from a pretrained MLLM and adds a decoupled Mixture-of-Transformers (MoT) generation branch. Structurally, the model shares a lower trunk between understanding and generation, then splits into parallel upper layers — one path handling standard visual understanding losses, the other handling Next Embedding Prediction (NEP). NEP replaces the usual discrete-token / diffusion generation objective with a cross-modal prediction over the continuous embedding space that MLLMs consume natively, avoiding the objective mismatch that has hampered prior generation-to-understanding transfer attempts. Gradient flow is asymmetric by construction: generation gradients propagate into the shared trunk (so the shared visual pathway becomes more spatially precise and visually faithful) but are blocked from the upper understanding layers, so the understanding path retains its pretraining-optimized representations. The final ingredient is task selection: rather than a generic “reconstruct the image” objective, GAS constructs correlated generation tasks that demand cognitive grounding — the paper argues that generic synthesis alone would provide the wrong training signal. After training, the generation branch and its upper layers are dropped; inference runs the MLLM as if GAS had never been applied.

Results

Reported gains are aggregate multimodal understanding improvements across model scales and training stages, with the most reliable improvements concentrated on perception and spatial comprehension benchmarks [Abstract]. Because the auxiliary generation branch is discarded after training, these gains come at zero inference-time cost [Abstract]. The paper includes controlled comparisons and representation-level analyses arguing that (i) the decoupled MoT topology is what makes the recipe safe (unblocked generation gradients would erode understanding), and (ii) the highly-correlated cognitive generation tasks are what makes the recipe useful (generic reconstruction would not) [§4, §5]. Specific benchmark numbers were not extractable from the abstract-level fetch — Method and Results claims here are structural rather than quantitative.

Why it’s interesting

GAS is the second filed paper (after Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation) that treats generation-as-auxiliary as a post-training recipe for improving MLLM understanding, and it lands on a substantially different architectural bet: continuous Next Embedding Prediction inside a decoupled MoT, versus UniMRG’s discrete depth/segmentation/pixel-map generation. Both papers converge on the same core claim — auxiliary generation helps understanding, particularly on spatial and perception axes, at no inference cost — but the mechanisms are almost orthogonal. UniMRG’s argument is about what to generate (intrinsic representations like depth and segmentation); GAS’s argument is about where the generation objective lives (in the same continuous embedding space understanding consumes, isolated architecturally in an MoT). This directly speaks to two open questions on Unified Multimodal Models: whether the “generation improves understanding” effect generalizes beyond depth+segmentation (GAS says yes, with NEP targets) and what the right capacity floor is for hosting auxiliary generation tasks (GAS’s answer is architectural — decoupled MoT — rather than about codebook size). Also worth contrasting with Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes, which found generation→understanding transfer is “neutral except as a fine-tuning prior on low-level concepts” — GAS’s zero-inference-cost gains on perception/spatial tasks look like exactly that fine-tuning-prior regime, done efficiently.