UnifiedVisual: A Framework for Constructing Unified Vision-Language Datasets

TL;DR

UnifiedVisual is a Fudan NLP framework (EMNLP 2025) for building training datasets that explicitly couple multimodal understanding and generation in a unified VLLM, and UnifiedVisual-240K is the 240K-sample dataset it produces — 120K understanding samples (sampled from LLaVA-CoT + CoT-Collection) plus 120K generation samples built via three pipelines: visual generation (image generation / editing / correction), multimodal reasoning (MM Reasoning O / MM / T), and curated multimodal internet data. The bet is that embedding reasoning rationales inside generation tasks, and embedding generated images inside reasoning rationales, makes the two capabilities mutually reinforcing rather than conflicting. Trained on Anole (and replicated on Emu3 and Liquid), the recipe lifts average understanding 33.4 → 46.3 over the base and beats a matched Laion-aesthetics generation-data baseline on both understanding (MME 1371 vs 953) and GenEval generation.

Key claims

Existing UMM training datasets treat understanding and generation in isolation; the standard recipe of adding multimodal generation data to understanding data actually hurts understanding performance because the two objectives conflict — Anole-NormalData (Laion-aesthetics generation + UnifiedVisual-T understanding) scores 39.9 average vs Anole-UnifiedVisualT’s 43.7 (understanding-only on the same understanding samples) [Table 1, §5.2.1].
UnifiedVisual-240K reverses this conflict: trained on the combined understanding + generation data the dataset constructs, Anole-UnifiedVisual reaches 46.3 average — beating both its understanding-only (43.7) and generation-only (40.9) variants, evidence the two halves mutually enhance rather than interfere [Table 1, §5.2.1].
Construction principle for image generation: do not use direct caption→image pairs. Two enhanced routes — Topic-and-Scene-Based (implicit caption → GPT-4 rationale → DALL-E-3 image) and Category-and-Image-Based (real image + category → implicit instruction → GPT-4 rationale → aligned outputs) — both embed a reasoning rationale between prompt and image [§3.1.1].
Image correction is introduced as a new task paradigm: Stable Diffusion generates an intentionally inconsistent image from a perturbed caption, GPT-4o then analyzes the inconsistency and provides a modification rationale, and the final sample is (original caption, inconsistent image, analysis rationale, original image) — training the model to detect and explain image-text mismatches [§3.1.1].
Three multimodal-reasoning data flavors interleave generated images with text rationales: MM Reasoning (O) demands rationales that snapshot details of the input image; MM Reasoning (MM) demands joint image+text reasoning, with rationale images either generated or Bing-retrieved and CLIP-filtered for style consistency; MM Reasoning (T) takes text-only inputs but emits DALL-E-3-generated rationale images mid-trace [§3.1.2].
Trained on UnifiedVisual-240K, generation quality also rises: Anole-UnifiedVisual beats Anole-NormalData on GenEval overall (Fig. 3) despite Laion-aesthetics having higher per-image visual quality, with the largest GenEval sub-metric gains in single/double-object generation, color, and quantity — attributable to the understanding data refining the model’s grasp of object attributes (Fig. 6, §5.2.2).
Mixing UnifiedVisual generation data with NormalData (Laion) generation data, while keeping understanding data fixed, beats either alone — image quality (Laion) and reasoning-rich generation (UnifiedVisual) compose [§5.2.2].
Text reasoning improves too: AlpacaEval win-rates against Anole-NormalData (Fig. 7) put Anole-UnifiedVisual highest, indicating the generation half of the dataset is also boosting text-only reasoning [§5.2.3].
Controlled scaling: with generation data fixed at 120K and understanding data varied 0K → 120K, GenEval overall improves monotonically; symmetrically, with understanding fixed at 120K and generation varied 0K → 120K, POPE F1 improves monotonically. Neither curve has converged at 120K, suggesting both halves still have headroom [Fig. 5a, Fig. 5b, §6.2].
Generalizes across UMM architectures: the same UnifiedVisual-vs-NormalData win on both understanding and generation reproduces on Emu3 and Liquid (Fig. 4), so the effect is not Anole-specific [§6.1].
Training scale: 64 H100 80G, batch size 512, max-seq 4096, AdamW with cosine + 5% warmup, max LR 2e-5, 2 epochs [§4.3].

Method

The contribution is a data-construction pipeline, not a new architecture. Three pipelines populate the 120K generation half of UnifiedVisual-240K:

Visual Generation. Image Generation uses two routes that never expose a direct caption→image mapping: implicit topic/scene captions filtered for diversity by embedding deduplication, with GPT-4-written rationales that explain what should be in the image before DALL-E-3 draws it (Topic-and-Scene); and real images + category labels reverse-engineered into implicit instructions via GPT-4, with a rationale explaining the logic of the desired output (Category-and-Image). Image Editing rewrites simple edits into nuanced multi-step prompts, with GPT-4o rationales that outline objective and intended outcome. Image Correction generates intentionally inconsistent images via Stable Diffusion from perturbed captions, then uses GPT-4o to diagnose and correct — yielding 4-tuples (original caption, generated image, analysis rationale, original image).

Multimodal Reasoning. Three sub-flavors. MM Reasoning (O) trains the model to attend to details of the input image — rationale “snapshots” critical regions, following the inline-grounding style of Shao et al. (2024a). MM Reasoning (MM) builds joint image+text reasoning by: (1) deduplicating images via CLIP, (2) generating reasoning questions with GPT-4o, (3) producing rationales with GPT-4o that use textual descriptions in place of images when needed, (4) rewriting those descriptions into Bing-search keywords for style-consistent retrieval, (5) CLIP-similarity-filtering retrieved images. MM Reasoning (T) starts text-only — GPT-4 generates questions that require image-aided rationales, dedups them via embeddings, GPT-4 writes rationales with image descriptions inline, DALL-E-3 renders the images.

Multimodal Internet. Interleaved text-image crawl data filtered with pretrained VLLMs for coherence between sentences and adjacent images; questions then generated so each sample becomes an answerable QA over the interleaved content (29,399 samples).

Final sub-counts of the 120K generation half [§3.2, Tab. 3]: MM Internet 29,399; Image Editing 9,024; Image Generation 22,755; Image Correction 20,000; MM Reasoning (O) 21,000; MM Reasoning (T) 7,276; MM Reasoning (MM) 17,761. The 120K understanding half is sampled 60K + 60K from LLaVA-CoT and CoT-Collection.

The base model is Anole — a fully-autoregressive UMM that represents images as discrete tokens, with image generation via an attached vision decoder.

Results

Anole-UnifiedVisual vs the three baselines on 6 understanding benchmarks (RWQA / MMVP / SQA / VStar / MME / POPE → average) [Table 1, §5.2.1]:

Anole base: 33.4 average
Anole-NormalData (Laion-gen + UnifiedVisual-und): 39.9
Anole-UnifiedVisualT (understanding-only): 43.7
Anole-UnifiedVisualMM (generation-only): 40.9
Anole-UnifiedVisual (combined): 46.3

Adding Laion generation data to understanding data drops understanding 43.7 → 39.9 — a clean demonstration of the standard objective conflict. Replacing the generation half with UnifiedVisual’s generation data reverses the loss and adds 6.4 points on top.

Generation (MS-COCO CLIP score; GenEval overall) [§5.2.2, Fig. 3, Fig. 6]: Anole-UnifiedVisual beats Anole-NormalData on GenEval overall despite Laion’s higher per-image aesthetic quality; the per-metric breakdown locates the gain in single/double-object, color, and quantity sub-metrics. A 50/50 mix of UnifiedVisual-generation + Laion-generation (understanding held fixed) further improves over UnifiedVisual alone, isolating image-quality as an orthogonal axis that should be combined with reasoning-rich generation supervision.

Text reasoning (AlpacaEval win-rate vs Anole-NormalData) [§5.2.3, Fig. 7]: Anole-UnifiedVisual highest among the four; Anole-NormalData lowest. Generation data improves text-only reasoning when it contains rationales.

Cross-architecture replication on Emu3 and Liquid [§6.1, Fig. 4]: same UnifiedVisual-vs-NormalData win pattern on both understanding and generation. The result is not Anole-specific.

Scaling ablations [§6.2, Fig. 5]: monotone improvement in GenEval as understanding samples grow 0K → 120K (gen fixed); monotone improvement in POPE F1 as generation samples grow 0K → 120K (und fixed). Neither curve has converged.

Why it’s interesting

The standard recipe of mixing understanding + generation data has a known failure mode that Unified Multimodal Models has been documenting via multiple angles: Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation (UniMRG) gets generation→understanding lift via auxiliary depth + segmentation targets, while UniG2U-Bench: Do Unified Models Advance Multimodal Understanding? (UniG2U-Bench) shows that inference-time Generate-then-Answer usually hurts. UnifiedVisual sits earlier in the stack — same conflict, attacked at the training-data level rather than at the architecture or inference protocol level: the synergy comes from embedding rationales into generation samples and embedding generated images into reasoning samples, so the two losses see related rather than orthogonal supervision. This complements DuoGen: Towards General Purpose Interleaved Multimodal Generation‘s architectural decoupled-MLLM-then-DiT recipe by suggesting a parallel data-side recipe that might compose with it. Methodologically the heterogeneous-generator stack (GPT-4 / GPT-4o / DALL-E-3 / Stable Diffusion / CLIP / Bing retrieval) and the implicit-prompt + rationale design place it squarely in the lineage Synthetic Training Data tracks — closest in spirit to Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning‘s hindsight curation (regenerate until the trace is coherent), though UnifiedVisual lacks an explicit verification gate and instead relies on diversity filtering + the implicit-instruction design to keep the data from collapsing into shortcut-mappable pairs. And the MM Reasoning (MM) / (T) flavors — where the rationale itself interleaves generated images — is a precursor of the substrates Thinking with Modalities now catalogs at inference time (DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models, Introducing Agentic Vision in Gemini 3 Flash), but applied at training time so the behavior is trained in rather than prompted out.