Factuality Matters: When Image Generation and Editing Meet Structured Visuals

TL;DR

StructVisuals is a full-stack attack on the failure mode that modern text-to-image and image-edit models cannot reliably produce or modify structured visuals — charts, diagrams, math figures, tables, puzzles — because these demand composition planning, accurate text rendering, and factual fidelity rather than aesthetic plausibility. The authors release (1) a 1.3M-pair dataset rendered from executable Python/LaTeX programs and annotated with GPT-5 chain-of-thought trajectories, (2) a unified model that bolts Qwen-VL onto FLUX.1 Kontext via a lightweight MLP connector trained in a three-stage curriculum with an inference-time external VLM reasoner, and (3) StructBench, a 1,714-item benchmark using a multi-round VLM Q&A protocol (StructScore) iteratively refined to >95% reliability on ground-truth images. Their model leads StructEditBench at 55.98% Overall Accuracy, beating Seedream 4.0 (52.85), GPT-Image (52.20), and Nano Banana (51.57).

Key claims

Even leading closed-source generators fail at structured visuals: on StructEditBench, Nano Banana / GPT-Image / Seedream 4.0 score 51.57 / 52.20 / 52.85% accuracy averaged across Math, Chart, Graph, Puzzle, Science, Table — the proposed model reaches 55.98% and is the only system above 60% on Graph, Math, and Science [Leaderboard tables].
The 1.3M training corpus is built by extracting ~2M executable drawing programs (Python, LaTeX), rendering source images, using GPT-5 to (a) extract salient visual features, (b) jointly produce aligned image-editing and code-editing instructions, (c) apply the code edit to synthesize a target program+image, then post-processing to drop invalid, low-difference, and low-information samples [§“Structured Image Dataset”].
Each sample carries a source image, target image, dense caption, image-editing instruction, and a three-step reasoning trajectory — the reasoning trace is data, not just a downstream training trick [§“Structured Image Dataset”].
The model architecture keeps FLUX.1 Kontext’s MMDiT backbone and VAE-encodes both input and target images into joint-attention tokens, but discards CLIP and adds a learned MLP connector that pipes Qwen-VL multimodal features into the backbone for high-level semantic grounding [§“Model Training”].
Training is a three-stage curriculum: (1) Unified Alignment freezes the backbone and trains only the connector on simple data with T5 suppressed to prevent shortcutting; (2) Hybrid Visual Learning jointly fine-tunes on structured + general datasets with a mask-based loss that downweights backgrounds and unchanged regions; (3) Thinking Enhancement injects CoT supervision and enables an inference-time external Qwen-VL reasoner that analyzes the input and guides the generator [§“Model Training”].
StructScore is a multi-turn VLM Q&A evaluation: GPT-5 decomposes a detailed image description into atomic Q&A pairs covering fine-grained attributes and relations, the judge VLM answers each open-endedly, and answers are matched against concise ground truths; iterative refinement raised reliability on ground-truth images from ~80% to >95% [§“Benchmark Construction”].
The benchmark spans 1,714 items across Math, Graph, Chart, Puzzle, Science, Table with 32,031 / 37,941 Q&A pairs for editing and generation respectively, and 15 models are evaluated [§“Benchmark Construction”, Leaderboard tables].
On StructEditBench Charts only, the model is #1 on Category (50.81), Color (64.10), and Auxiliary (66.34) sub-skills, and #1 overall (50.58 Accuracy) — closed frontier models top out in the mid-40s [StructEditBench charts-only table].
On StructT2IBench (generation-from-text only, no source image), the top three are still closed frontier models (GPT-Image 49.58, Nano Banana 48.45, Seedream 4.0 47.52) — the proposed model is the strongest open system at 28.80 overall but the closed-vs-open gap is larger here than on editing [StructT2IBench table].
Inference-time reasoning yields consistent gains across diverse architectures — the external reasoner is a generic add-on, not specific to the proposed backbone [§Abstract].

Method

The pipeline is split into three subsystems that line up with the three releases:

Dataset construction. Start from ~2M executable drawing programs (Python with matplotlib/seaborn/networkx etc., LaTeX for math). Render each into a source image. Use GPT-5 to extract salient visual features and emit aligned image-edit instructions and code-edit instructions for the same change. Re-render the edited code to get the target image. Post-process: discard invalid renders, near-identical source/target pairs (low-difference), and low-information samples. Result is 1.3M source/target pairs each with dense captions, edit instructions, and a three-step reasoning trace.
Model. Base is FLUX.1 Kontext — a unified DiT for generation+editing where text comes from T5 (CLIP is discarded), and source and target images are VAE-encoded into the same joint-attention token stream. The new component is a lightweight MLP connector that takes Qwen-VL multimodal features and projects them into the backbone’s conditioning space. Three-stage training: (i) freeze backbone, train connector only on simple alignment data with T5 suppressed; (ii) joint fine-tune backbone+connector on a structured+general data mix with a mask-based loss (downweight backgrounds and unchanged pixels); (iii) inject GPT-5 / Qwen-VL CoT supervision and enable an external Qwen-VL reasoner at inference that analyzes the input and writes guidance for the generator.
Evaluation. StructBench is filtered from the rendered corpus by clustering + stratified sampling + GPT-5/human review. For each item, GPT-5 produces a detailed description that is decomposed into atomic Q&A pairs. At score time a judge VLM answers each Q open-ended; answers are matched against concise ground truths. Iterative refinement (loop: identify Q&A pairs where judge disagrees with GT on the GT image; rewrite them) raised reliability from ~80% to >95% on ground-truth images.

Results

StructEditBench (Overall, Accuracy% / PSNR): Ours 55.98 / 24.01 > Seedream 4.0 52.85 / 24.45 > GPT-Image 52.20 / 16.64 > Nano Banana 51.57 / 21.09. The model is +3.13 absolute over the strongest closed system, and far ahead of open baselines (FLUX.1 Kontext 37.56, Qwen-Edit 38.12, Step1X-Edit 34.11).
StructEditBench by domain: Ours leads on Math (54.74 vs 51.49 GPT-Image), Chart (50.58 vs 46.83 Seedream 4.0), Graph (60.18 vs 52.97 Nano Banana), Science (75.05 vs 69.22 Seedream 4.0); second-best on Puzzle (73.00 vs 76.03 GPT-Image) and Table (77.08 vs 88.19 Seedream 4.0).
StructEditBench Charts edit-type breakdown: Ours leads Category (50.81), Color (64.10), Auxiliary (66.34), and Add&Del (38.10). On Num it’s third (33.45 vs Nano Banana 35.64 / Seedream 4.0 36.00). PSNR ordering does not track accuracy ordering — visually faithful is not the same as factually correct.
StructT2IBench (Overall, Accuracy%): GPT-Image 49.58 > Nano Banana 48.45 > Seedream 4.0 47.52 > Qwen-Image 41.03 > Ours 28.80 > FLUX.1 Kontext 20.36. From-scratch generation of structured visuals is dominated by closed frontier models; the open ecosystem (including this work) trails by ~20 points.
StructScore reliability on ground-truth images improved from ~80% (single-pass) to >95% after iterative Q&A refinement [§“Benchmark Construction”].

Why it’s interesting

The cleanest filed example of “code-rendered programs → image pairs” as a scalable supervision signal for structured visual generation. This pattern complements Synthetic Training Data‘s existing entries: where Video-Thinker uses heterogeneous LLM/VLM stacks to label real videos, and SpaceTimePilot uses Blender to synthesize a control axis that doesn’t exist in nature, StructVisuals uses executable programs as a deterministic source of paired (source, target) images with both image-space and code-space edit instructions — a third synthesis recipe driven by the fact that the underlying domain (charts, diagrams, math) is natively compositional and programmatic.
Strongest filed instance of the VLM-as-evaluator pattern for editing benchmarks. StructScore extends the IGenBench-style atomic-Q&A judging protocol (IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation) to image editing (1,714 items × 32k+37k Q&A) with iterative reliability refinement quantified end-to-end (80% → 95%). It cleanly sits in VLM-as-Evaluator‘s benchmark-time shape but adds the iterative-rubric-refinement loop as a methodological contribution. Open question this surfaces: how much of the 80→95% gain is judge-model-specific vs prompt-rubric-specific?
Decouples the “thinking” from the generator. The Stage-3 inference-time external Qwen-VL reasoner is closest to substrate (c) on Thinking with Modalities (external-tool reasoner appending guidance to the generator) rather than the trained-in tagged-CoT of Video-Thinker or the diffusion-native CoT of DiffThinker. The paper’s claim that “inference-time reasoning yields consistent gains across diverse architectures” would, if reproduced, generalize the substrate to any backbone — a stronger generality claim than DiffThinker’s (which depends on a specific diffusion fine-tune).
Open-vs-closed gap is bigger for generation than editing. Ours leads StructEditBench but is far behind on StructT2IBench. This is consistent with the broader Unified Multimodal Models pattern where text-to-image quality scales with closed-frontier compute budget, but instruction-following editing is a place where careful curriculum + targeted data closes the gap.