Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning

TL;DR

Zebra-CoT is a 182,384-sample dataset of logically coherent interleaved text-image reasoning traces, built to train models to natively perform Visual Chain-of-Thought (sketching while reasoning). Traces span 4 categories — scientific questions (geometry, physics, algorithms), 2D visual reasoning (visual search, jigsaw), 3D reasoning (multi-hop, embodied/robot planning), and visual logic/strategic games (e.g. chess) — across 18 domains and 50+ tasks. The motivation is that off-the-shelf visual CoT is weak (which hampers RL bootstrap) and large interleaved reasoning corpora don’t exist; Zebra-CoT addresses the second gap so the first can be attacked. Fine-tuning Anole-7B on Zebra-CoT yields +12% in-distribution accuracy and up to +13% on standard VLM benchmarks; fine-tuning Bagel-7B yields a model that generates the interleaved reasoning chains natively.

Key claims

Off-the-shelf frontier VLMs perform poorly at visual chain-of-thought in zero-shot settings, which is the cause of the data scarcity loop: weak base performance hinders RL bootstrap, and the absence of large interleaved reasoning corpora prevents SFT from breaking the loop [§1].
The dataset contains 182,384 interleaved text-image reasoning traces across 4 categories, 18 subdomains, and >50 task types — claimed to be the first dataset covering this breadth of multimodal reasoning [§1, Fig. 2, Table 3].
Data is constructed by two routes: (a) sourcing and cleaning raw traces from real-world domains (e.g. existing visual problem corpora), and (b) generating synthetic examples from templated reasoning skeletons filled in by VLMs [Fig. 1].
Fine-tuning Anole-7B on Zebra-CoT improves in-distribution accuracy from 4.2% to 16.9% (~+12 points) and yields up to +13% on out-of-distribution standard VLM benchmark evaluations [§abstract; corroborated by Liner review].
Fine-tuning Bagel-7B (a stronger interleaved-generation base) produces a model that natively emits high-quality interleaved visual reasoning chains, demonstrating the dataset is sufficient signal to learn the trace format end-to-end, not just answer accuracy [§abstract].
The four task categories were chosen because sketching or visual reasoning is the natural human strategy — geometry diagrams, 2D search/puzzles, 3D multi-hop and embodied/robot planning, strategic-game visualizations [§1, §abstract].
Released open-source: dataset under multimodal-reasoning-lab/Zebra-CoT and fine-tuned model multimodal-reasoning-lab/Anole-Zebra-CoT on HuggingFace [§release].

Method

The pipeline (Fig. 1) has two parallel branches feeding a common cleaning/validation stage:

Real-trace sourcing: existing problem corpora (geometry, physics, algorithms, puzzles, chess) are scraped or licensed; reasoning steps are extracted and aligned with their accompanying figures or sketch states.
Synthetic generation: templated reasoning skeletons (e.g. “to solve this geometry problem, first identify the relevant figure, then construct auxiliary line X, then…”) are filled in by VLMs that generate both the natural-language reasoning text and the intermediate image (sketch / annotation / partial diagram). The resulting traces are then logical-coherence-checked.

The final corpus is interleaved: each sample is a sequence of (text segment, image segment, text segment, image segment, …) terminating in a final answer. Evaluation is performed by fine-tuning two open interleaved-generation bases (Anole-7B, Bagel-7B) on the training split and measuring (a) Zebra-CoT test-set accuracy, (b) transfer to standard VLM benchmarks, and (c) qualitative interleaved-trace quality for the Bagel variant.

Results

In-distribution: Anole-7B accuracy rises from 4.2% (zero-shot) to 16.9% (fine-tuned on Zebra-CoT) — a +12.7 absolute-point gain on the held-out test split [§4 via Liner review].
Out-of-distribution transfer: up to +13% on standard VLM benchmark evaluations after fine-tuning, indicating the visual-CoT training signal generalizes beyond the in-distribution task distribution [§abstract].
Generation quality: Bagel-7B post-fine-tune produces interleaved text-image reasoning chains directly — i.e. the model learns the production side of visual CoT, not only the evaluation side [§abstract].
Frontier zero-shot baselines (unspecified frontier models in §4) are reported as scoring poorly in zero-shot but improving significantly when multimodal-CoT prompting is invoked [Liner review §RQ1].

Why it’s interesting

This is the obvious missing dataset for the Thinking with Modalities line of work — the concept page currently splits four substrates (tagged-latent-skill, diffusion-native, external-tool, input-scaffolding) but every filed instance is bottlenecked on either training data scarcity (Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning hand-curated 10K samples) or training-free workarounds (Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs horizontal-line overlays). Zebra-CoT is the first attempt at a broad-coverage interleaved-reasoning corpus aimed at native visual CoT training, sitting alongside VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (video CoT, single domain) and the synthetic-data verification-gate lineage on Synthetic Training Data. The Bagel-7B result also lands a marker: with the right base model, the interleaved trace format is learnable from this corpus alone, suggesting the data — not the architecture — is the bottleneck for substrate-(a)-style native visual CoT.