WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation

TL;DR

WEAVE is a paired dataset + benchmark for multi-turn, in-context image comprehension and editing — the gap that all existing single-turn editing datasets (AnyEdit, MagicBrush, GPT-Image-Edit-1.5M, Echo-4o) leave open. WEAVE-100k is a 100K-sample interleaved corpus (~370K turns, ~500K images) generated by four task pipelines (multi-image fusion, remove-then-back, derivative imagination, sequential procedures) explicitly designed to require visual memory of prior turns. WEAVEBench is a 100-task human-annotated benchmark across science/creation/logic/game with a hybrid VLM-judge that scores against both a reference image and the original-image+instruction pair, evaluating four metrics (Key-Point Correctness, Visual Consistency, Image Quality, Accuracy). Fine-tuning Bagel on WEAVE-100k gives +5.4 MMMU, +4.8 GEditBench overall, +21% RISE spatial reasoning, +42.5% on WEAVEBench, with reported emergent visual-memory behavior.

Key claims

Existing editing datasets are fundamentally single-turn — even MagicBrush, which is nominally multi-turn, treats each instruction as independent rather than reasoning over prior images [§2, Table 1].
Four data-generation pipelines together produce interleaved samples that require visual memory: (i) multi-image fusion (reference prior edits when generating), (ii) remove-then-back (delete an object, then restore it from memory), (iii) derivative imagination and comparison (imagine alternatives before fusing), (iv) sequential procedures (narrative/structured edit chains) [§3.1].
WEAVE-100k contains 100K chats, ~370K dialogue turns, ~500K images, with an average of 3.8 turns per chat and the majority of instances containing more than five images [§3.3, Table — Statistics].
WEAVEBench is 100 tasks across science / creation / logic / game, annotated by humans with graduate-level STEM degrees, using web-sourced and synthetically-generated source images from Seedream 4.0, Nano Banana, and SeedEdit 3.0 [§3.1, §3.3].
The evaluation uses a hybrid VLM-judge — keypoint-based scoring with two evaluator instantiations, one comparing to the reference image and one comparing to the original-image+instruction pair — across four metrics: Key Point Correctness (KP), Visual Consistency (VC), Image Quality (IQ), and Accuracy (Acc, comprehension only) [§3.2].
On WEAVEBench, the best edit model (FLUX.1 Kontext, 12B) and best UMM (Bagel-Zebra, 14B) score 0.689 and 0.449 respectively; proprietary Seedream 4.0 / Nano Banana lead at 0.765-0.767 — best open systems trail by ~30%, and all models exhibit creative > scientific/logical domain bias [§4.1, Table 2].
In-context content has divergent effects across model families: increasing context degrades open-source editors (Qwen-Image-Edit loses several points moving from no-context to full-context), while proprietary Nano Banana improves — interpreted as a localization-accuracy failure of single-turn-trained models confronted with long histories [§4.1].
Sequential image input dominates concatenated input on UMMs — concatenation drops Bagel by ~36% (paper reports >36% delta on Bagel), arguing that UMMs are the right substrate for editing because traditional editors can’t even ingest multiple historical images natively [§4.1, Fig. 7c].
Fine-tuning Bagel on WEAVE-100k yields +5.4 MMMU (55.3 → 60.7), +4.8 overall on GEditBench (6.52 → 6.83), +13.4 on material-change and +15.6 on style-change subtasks of GEditBench, and 2-4× absolute gains on RISE Temporal/Causal/Spatial/Logical [§4.2, Tables 3-4].
On WEAVEBench itself, the fine-tuned Bagel goes from 0.449 → 0.640 — a 42.5% relative improvement, with the largest gains on the hardest “science” domain (+34.6%) [§4.2, Table 2].
VLM-as-judge reliability — Pearson correlation between GPT-4.1 scores and three human experts exceeds 0.8 across three target models (Nano-banana, Qwen-Image-Edit, SeeDream); Claude Opus 4.1 shows strong cross-VLM consistency with GPT-4.1 [§4.4].

Method

WEAVE-100k is built by a four-pathway pipeline that produces interleaved (text, image)* sequences where later turns reference earlier images. The pipelines are: (1) multi-image fusion, which generates a new image by combining a prior edited result with new instruction content, building cross-turn visual reference; (2) remove-then-back, which deletes an object in turn t and asks the model to restore it in turn t+k, testing whether the model retained the removed element; (3) derivative imagination and comparison, which asks the model to imagine an alternative target before fusing it with prior context; (4) sequential procedures, which chain edits along a narrative or structured operation sequence (e.g. step-by-step refinement of a comic frame). Each pipeline is followed by multi-stage filtering and refinement to keep accuracy/quality up.

WEAVEBench is human-curated. Annotators with graduate-level STEM degrees write 100 tasks spanning multi-turn editing under visual-memory load and tasks that require external world knowledge (cultural contexts, physical phenomena, chemical processes). Source images mix web content with synthetic outputs from Seedream 4.0, Nano Banana, and SeedEdit 3.0. Each task ships with a text prompt, one or more initial images, ground-truth examples, and the explicit key information the correct output must satisfy.

Evaluation uses VLM-as-judge with two instantiations per item: one comparing the model output to a held-out reference image, one comparing it to the (original-image, editing-instruction) pair. Four metrics:

Key Point Correctness (KP) — does the edit satisfy the required key points?
Visual Consistency (VC) — are non-target regions unchanged, and are edited objects identity-preserving?
Image Quality (IQ) — overall fidelity / artifact load.
Accuracy (Acc) — for comprehension turns only, did the reasoning produce the right answer?

The benchmark also varies image placement (yes-first: images at their first text mention; yes-front: all images consolidated at the start) and in-context regime (no in-context / partial in-context / complete in-context) — the yes-front × complete regime is the headline number.

To validate the data, the authors fine-tune Bagel-14B on WEAVE-100k and re-evaluate against MMMU, GEditBench, RISEBench, and WEAVEBench itself.

Results

WEAVEBench (yes-front, complete in-context, averaged over Science / Creation / Logic / Game) — Edit models top out at FLUX.1 Kontext 0.689 and Qwen-Image-Edit 0.665; UMMs top out at Bagel-Zebra 0.449; proprietary Seedream 4.0 and Nano Banana score 0.765 / 0.767. Bagel + WEAVE-100k reaches 0.640 — bridging most of the Bagel→Edit-model gap [§4.1, Table 2].
In-context regime effects — proprietary models gain from more context (Nano Banana increases monotonically), open-source editors lose (Qwen-Image-Edit drops moving from partial to complete context) [§4.1].
Domain bias — every model scores higher on creative tasks than on scientific or logical tasks (e.g. Bagel-Zebra: Creation 0.456 vs Logic 0.393 vs Science 0.399). Fine-tuning narrows but does not close this gap [Table 2].
GEditBench (image editing, 11 task categories) — Bagel 6.52 → +WEAVE-100k 6.83 (+4.8% overall). Largest deltas on material change (+13.4) and style change (+15.6) [Table 4].
RISEBench (comprehension-generation collaboration) — Bagel Temporal 2.4 → 4.7, Causal 5.6 → 6.7, Spatial 14.0 → 21.0 (+50%), Logical 1.2 → 2.4 (+100%) [Table — Understanding + RISEBench, last row].
Understanding benchmarks — MMB 85.0 → 85.2, MMMU 55.3 → 60.7 (+9.8%), MMVet 67.2 → 67.4; GenEval Overall 0.82 → 0.84. Fine-tuning on a generation-heavy interleaved corpus measurably improves understanding metrics on a UMM [§4.2].
Judge reliability — Pearson r > 0.8 between GPT-4.1 keypoint scores and three independent human raters across three target models; Claude Opus 4.1 evaluations track GPT-4.1 closely — VLM-judge choice has minimal impact [§4.4].

Why it’s interesting

WEAVE is the multi-turn pendant to EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning — both papers argue that in-context image creation is the natural endpoint for unified multimodal models, but they attack the gap from opposite directions: EditVerse builds a single-transformer architecture and a 232K video-editing pipeline aimed at cross-modal transfer (image-edit → video-edit), while WEAVE builds a 100K interleaved multi-turn corpus aimed at cross-turn transfer (single-turn-edit → memory-conditioned-edit). The WEAVE pipeline (remove-then-back, derivative imagination, sequential procedures) is the most explicit attempt yet to manufacture visual-memory supervision; the +50% spatial-reasoning lift on RISEBench is a strong external validation that the recipe transfers to ordinary single-image comprehension tasks.

The fine-tuning result on Bagel — generation-heavy interleaved SFT measurably improves understanding (MMMU +5.4) — is a clean second datapoint for the thesis Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation makes with UniMRG: generation supervision can enhance understanding in a UMM. UniMRG’s recipe used auxiliary intrinsic-representation generation (depth, segmentation, pixel reconstruction); WEAVE’s recipe uses task-realistic interleaved editing. The two are complementary evidence that the open-source UMM stack is leaving understanding gains on the table by training only on single-turn pairs.

The hybrid VLM-judge framework adds to the growing dispute on this wiki about what evaluators should look like. Compared to Unified Personalized Reward Model for Vision Generation (UnifiedReward-Flex: per-prompt dynamic rubric) and the static-rubric ERNIE-Image-Aes: Robust Image Aesthetics Scoring with Balanced Category Generalization (ERNIE-Image-Aes), WEAVEBench sits closer to the benchmark-time keypoint-driven shape that IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation uses for infographics — but it doubles the judge on two reference frames (held-out reference image + original-image-with-instruction), which is a recipe none of the existing pages document. The Pearson-0.8 cross-judge consistency claim is also at odds with IGenBench’s finding that Gemini-2.5-Pro is the only MLLM clearing 0.8 against humans — WEAVEBench reports the same threshold with GPT-4.1 and Claude Opus 4.1, suggesting the judge-stability differs by task type (editing keypoints vs infographic atomic Q/A).