Thinking with Visual Primitives

TL;DR

DeepSeek interleaves spatial markers — <|box|> followed by a [ [x1,y1,x2,y2] ] coordinate list closed by <|/box|>, and <|point|> followed by a [ [x,y] ] coordinate list closed by <|/point|> — directly into the autoregressive chain of thought of a 284B/13B-active MoE MLLM built on DeepSeek-V4-Flash, so the model literally “points while it reasons” at the pixel coordinates it’s referring to. The framing is the Reference Gap: natural language is an ambiguous pointer into a continuous image, and prior “thinking with images” work (high-res cropping) only addresses the orthogonal Perception Gap. With a Compressed Sparse Attention path that turns a 756×756 image into ~81 visual KV entries (~7,056× pixel→KV compression), the model matches or beats GPT-5.4, Claude-Sonnet-4.6, and Gemini-3-Flash on a focused suite of spatial-reasoning / counting / VQA tasks at a fraction of the visual token budget. The repo was published by the authors then made private; the team-of-record is DeepSeek-AI + Peking + Tsinghua.

Key claims

The Reference Gap — the ambiguity of natural language as a pointer to continuous visual space — is a distinct failure mode from the Perception Gap that high-res cropping / dynamic patching addresses, and is what limits MLLMs on dense counting and multi-step spatial deduction even when “seeing” is perfect [§1].
Bounding boxes and points are elevated to “minimal units of thought” by interleaving them into the CoT trace itself (not used post-hoc as verification), and the response format makes them first-class vocabulary tokens: <|ref|>TARGET<|/ref|><|box|>[[x1,y1,x2,y2],…]<|/box|> and <|point|>[[x,y],…]<|/point|>, with coordinates normalized to integers 0–999 [§2.2, §2.3.4].
The 284B/13B-active MoE backbone is DeepSeek-V4-Flash with a from-scratch DeepSeek-ViT vision encoder; a 14×14 patch + 3×3 spatial-token compression + 4× CSA KV compression yields ~7,056× pixel→KV compression (756×756 image → 2,916 patches → 324 ViT tokens → 81 KV entries) [§2.2].
An 800×800 input is reported at ~361 tokens-before-compression / ~90 KV entries — roughly 1.8× fewer than Gemma-4-31B (~660), ~2× fewer than GPT-5.4 (~740), ~2.4× fewer than Claude-Sonnet-4.6 (~870), and ~3× fewer than Gemini-3-Flash (~1100) [Fig. 1a].
Average score across 7 selected counting + spatial-reasoning benchmarks: 77.2% for the proposed model vs Gemma-4-31B 69.7%, Qwen3-VL-235B-A22B 68.1%, GPT-5.4 71.1%, Claude-Sonnet-4.6 65.3%, Gemini-3-Flash 76.5% [Fig. 1b].
Bounding-box data is prioritized for large-scale pretraining (over points) for three reasons: deterministic annotation (a point inside an object is ambiguous; a tight box is not), task generalizability (a box is two points and so subsumes the point format), and information richness (boxes carry geometry, points do not) [§2.3.2].
Web-scale grounding data is curated through a two-step filter pipeline applied to ~98k raw HuggingFace object-detection / grounding datasets: a semantic-review pass (drops gibberish labels, private-entity labels, ambiguous “OK/NG” labels) keeps 43,141; a geometric-quality pass (drops low-recall datasets, severe truncation, “mega box” classification-as-detection) keeps 31,701; per-category sampling (N=1,000) yields ~40M high-quality boxes [§2.3.3].
Coarse- vs fine-grained counting are split as distinct cold-start tasks: coarse uses batch grounding (locate all candidates in one box list, then sum) because the model’s localization is strong and batch-mode avoids repetitive enumeration; fine-grained uses a sequential scan over GQA scene-graph metadata so each candidate is verified against attribute / spatial constraints before being counted [§2.4.1].
Training pipeline is Pretraining → Specialized SFT → Specialized RL → Unified RFT → On-Policy Distillation; spatial reasoning is unified with general VQA on the hypothesis that the visual-primitive skill learned on spatial tasks transfers to broader VQA [§2.1 Fig. 2, §2.4.2].

Method

The architecture is a standard LLaVA-shape: image → DeepSeek-ViT (14×14 patches, supports arbitrary resolution, ViT-output 3×3 spatial token compression) → visual-token sequence concatenated with the language prompt → DeepSeek-V4-Flash MoE LLM (284B total, 13B active). The key efficiency lever is the Compressed Sparse Attention (CSA) mechanism inherited from V4-Flash, which compresses every m visual tokens into one KV entry — so the ViT output (already 9× smaller than raw patches) is compressed another 4× in the KV cache. The pipeline outputs response strings that mix natural language with the <|ref|>…<|/ref|><|box|>…<|/box|> and <|point|>…<|/point|> primitive tokens.

Pretraining objective is unified next-token prediction over multimodal data, with the visual-primitive output format treated as part of the vocabulary — no separate detection head. The trillions of multimodal tokens combine general web-crawled (un-rewritten by LLMs) data with the curated ~40M box-grounding corpus and public datasets (COCO, Pixmo-Points, etc.). Post-training is a five-stage cold-start: (1) Specialized SFT and (2) Specialized RL bootstrap instruction-following under the primitive output interface on four target tasks — counting, spatial reasoning / general VQA, maze navigation, and path tracing — using rule-based verifiers wherever possible to limit label noise; (3) Unified RFT consolidates the specialists; (4) On-Policy Distillation propagates the consolidated behavior back into the base.

Cold-start data construction has a consistent shape: take a source corpus with ground-truth boxes (or scene-graph metadata), prompt an MLLM to synthesize a CoT trace that interleaves visual primitives with the reasoning steps (e.g. counting: intent analysis → batch grounding → statistical summation), and apply rule-based syntactic / coordinate-alignment / answer-consistency checks before retaining the sample. For fine-grained counting and spatial reasoning, the MLLM is also given the ground-truth IDs and the rationale for each QA pair, so the synthesized trace performs the right sequential verification.

Results

77.2% average across 7 counting + spatial-reasoning + visual-QA benchmarks (in-house benchmarks excluded) vs Gemini-3-Flash 76.5%, GPT-5.4 71.1%, Gemma-4-31B 69.7%, Qwen3-VL-235B-A22B 68.1%, Claude-Sonnet-4.6 65.3% [Fig. 1b].
800×800 image budget: ~361 visual tokens before compression → ~90 KV-cache entries, vs ~1100 for Gemini-3-Flash, ~870 for Claude-Sonnet-4.6, ~740 for GPT-5.4 (which makes this the cheapest model in the comparison by KV-cache footprint while still topping the benchmark average) [Fig. 1a].
For a 756×756 image: 571,536 pixels → 2,916 ViT patch tokens → 324 LLM-input tokens → 81 KV entries; overall 7,056× compression from pixels to KV [§2.2].
Data curation funnel: 97,984 raw grounding datasets → 43,141 after semantic review → 31,701 after geometric review → ~40M high-quality samples after per-category capped sampling at N=1,000 [§2.3.3].
The paper notes the benchmark suite covers “only a subset of evaluation dimensions that are directly relevant to the research focus” and is “not indicative of the models’ overall capabilities” — a useful caveat given the headline comparison numbers [Fig. 1 caption].

Why it’s interesting

Adds a fifth, mechanically distinct substrate to Thinking with Modalities: tagged spatial primitives — discrete coordinate tokens (boxes, points) — interleaved into an autoregressive CoT. This sits between Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning‘s tagged latent-skill substrate (which calls perception ops over time spans) and Introducing Agentic Vision in Gemini 3 Flash‘s external-tool substrate (which executes Python crops): the primitives here are first-class output tokens with deterministic ground truth, so the model can be supervised end-to-end with rule-based verifiers and trained at pretrain scale rather than only at the post-train cold-start phase.

The Reference-Gap vs Perception-Gap framing is also a sharper articulation of why “Thinking with Images” (crop-and-zoom) plateaus on dense spatial tasks — and the empirical answer (more pixels are not the bottleneck; better pointers are) is the strongest single argument on the wiki for spending pretrain compute on grounding format rather than higher-resolution towers. The CSA-driven ~7,056× pixel→KV compression complements DeepSeek-OCR 2: Visual Causal Flow‘s “visual causal flow” thesis from the same group — both papers say the efficient frontier for MLLMs is fewer, denser visual tokens with a primitive-rich output vocabulary, not more visual tokens with text-only output. Built on the DeepSeek-V4 collection release (Flash + Pro, up to 1.6T) V4-Flash backbone, this is the first paper on the wiki showing what V4-Flash’s CSA actually buys when the workload is multimodal rather than text-only.