DeepSeek-OCR 2: Visual Causal Flow

TL;DR

DeepSeek-OCR 2 replaces the CLIP-ViT inside the DeepEncoder of DeepSeek-OCR with an LLM-style mini-encoder (Qwen2-0.5B) plus a new “causal flow query” mechanism: a set of learnable query tokens, equal in cardinality to the visual tokens, is appended after the visual prefix and attends causally over both the visual tokens and previous queries. Only these reordered query outputs are fed to the 3B-MoE/A500M DeepSeek decoder. The cascade — encoder doing causal reordering, decoder doing autoregressive reading — is pitched as decomposing 2D visual reasoning into two stacked 1D causal reasoners. On OmniDocBench v1.5 the model reaches 91.09% overall with a 1120 max-visual-token budget, +3.73 over DeepSeek-OCR at parity-ish tokens, beating Gemini-3 Pro (0.115 ED) at the same 1120-token budget (0.100 ED). Code + weights are open; the work is positioned both as a research vehicle for LLM-style visual encoders and as a production OCR for generating LLM pretraining data.

Key claims

Rigid raster-scan ordering of visual tokens with positional encoding is an unwarranted inductive bias for documents whose layout (formulas, tables, columns) carries causal reading logic; replacing it with learned, semantics-driven reordering is the central design lever [§1, §3.2].
DeepEncoder V2 keeps the 80M SAM-base + conv 16× compressor of DeepEncoder V1 but swaps the CLIP-ViT for a Qwen2-0.5B-style decoder backbone, in which visual tokens use bidirectional attention as a prefix and learnable “causal flow queries” attend causally over them and over earlier queries [§3.2.2, Fig. 5].
Equal cardinality between causal queries and visual tokens (only the latter half of encoder outputs is forwarded to the LLM) is presented as critical “capacity for re-fixation” — and a cross-attention mBART-style alternative is reported to fail to converge, attributed to insufficient visual-token interaction when isolated in a separate encoder [§3.2.2].
Multi-crop visual-token budget is 256–1120 (one 1024² global view → 256 queries, plus 0–6 local 768² crops → 144 each), explicitly matching Gemini-3 Pro’s 1120-token max while staying under DeepSeek-OCR’s 1156 (Gundam mode) [§3.2.3].
On OmniDocBench v1.5 the model reaches 91.09 Overall at V-token_max = 1120, beating DeepSeek-OCR (87.36 at 1156), Qwen3-VL-235B (89.15 at >6000), Gemini 2.5 Pro (88.03), and dots.ocr (88.41), with reading-order ED dropping from 0.085 → 0.057 [Table 1].
At a comparable 1120 visual-token budget, DeepSeek-OCR 2 (0.100 overall ED) beats Gemini-3 Pro (0.115 ED) and Seed-1.8 (0.106 ED at 5120 tokens) on the same edit-distance metric [Table 2].
Production-side repetition rate drops from 6.25% → 4.17% on online-user-log image inputs and from 3.69% → 2.88% on PDF pretrain-data extraction, with repetition rate used as a proxy quality metric since ground truth is unavailable in production [Table 4, §5.3].
The decoder is unchanged from DeepSeek-OCR — a 3B DeepSeek-MoE with ~500M active parameters — so the entire performance gain is attributed to the encoder swap [§3.3, §4.2].
The paper frames DeepEncoder V2 as a candidate omni-modal encoder: a single LLM-style backbone with modality-specific learnable query embeddings could in principle compress text, audio, and images in a shared parameter space [§6.2].
Training is 3-stage on 160× A100: (1) encoder pretraining with a lightweight decoder at 768²/1024² for 40k iters / 100M pairs, (2) query enhancement jointly training encoder + DeepSeek-3B-A500M for 15k iters, (3) freeze encoder and continue-train only the LLM for 20k iters to >2× throughput [§4.2.1–4.2.3].

Method

DeepEncoder V2 keeps DeepSeek-OCR’s two-stage encoder topology — an 80M SAM-base + 2 conv layers as a 16× vision tokenizer, followed by a knowledge-compression backbone — but replaces that backbone with a Qwen2-0.5B-initialized decoder-only transformer. The trick is in the attention mask: the input sequence is [visual_tokens (m); causal_flow_queries (n)] with n = m. Within the visual prefix, attention is fully bidirectional (ViT-like); within the query suffix, attention is causally triangular; queries can attend to all visual tokens and to all earlier queries, but visual tokens cannot attend forward into queries. Only the n causal-query outputs are passed to the language decoder. The result is a cascade of two 1D causal reasoners — encoder reorders visual content semantically, decoder reads it left-to-right — which the authors argue is closer to human eye-movement causality on structured documents than fixed raster scans.

The visual-token count is set by a multi-crop scheme with a fixed pair of query banks: 256 “global” queries for one 1024² view and 144 “local” queries shared across k ∈ {0..6} crops at 768², for a total of k·144 + 256 ∈ [256, 1120] tokens fed to the LLM decoder. The downstream language model is unchanged from DeepSeek-OCR — DeepSeek-3B-A500M MoE. Training proceeds in three stages: encoder-only LM pretraining at fixed resolutions, joint encoder+decoder query enhancement under 4-stage pipeline parallelism, and a final encoder-frozen LLM continue-training stage that more than doubles data throughput per FLOP. Data mixes follow DeepSeek-OCR (80% OCR 1.0/2.0 + 20% general vision) with two tweaks: a 3:1:1 text/formula/table balance on OCR-1.0, and label refinement merging semantically similar layout-detection categories.

Results

On OmniDocBench v1.5 (1355 pages, 9 doc categories, EN+ZH) DeepSeek-OCR 2 reaches 91.09% Overall at V-token_max = 1120, vs DeepSeek-OCR 87.36 (1156 tokens), Qwen3-VL-235B 89.15 (>6000), OCRVerse 88.56 (>6000), dots.ocr 88.41 (>6000), and Gemini 2.5 Pro 88.03 (token budget unspecified) [Table 1]. Edit-distance metrics: Text 0.073→0.048, Formula CDM 84.14→90.31, Table TEDS 85.25→87.75, Reading-Order ED 0.085→0.057. At parity with Gemini-3 Pro on 1120 tokens, the overall ED is 0.100 vs 0.115 [Table 2]. Per-document-type Reading-Order ED improves across all 9 categories vs DeepSeek-OCR [Table 3]. The hardest residual category is newspapers (Text ED 0.139), which the authors attribute to (a) the 1120-token cap throttling text-dense pages and (b) only 250k newspaper samples in training. In production, repetition rate falls from 6.25%→4.17% (online image OCR) and 3.69%→2.88% (PDF pretrain-data) [Table 4].

Why it’s interesting

For a Luma audience this is most relevant as a visual encoder architecture paper, not strictly an OCR paper. The proposal — replace ViT-style fixed-order encoding with an LLM-style mini-encoder whose causal queries semantically reorder visual tokens — is a different answer to the same question token-reduction methods like Token Merging and RLT (Don't Look Twice: Faster Video Transformers with Run-Length Tokenization) ask: how should visual tokens be processed when their natural structure isn’t a 1D sequence? Where RLT drops or merges redundant tokens, DeepEncoder V2 keeps the count fixed (n queries = n visual tokens) but re-permutes them semantically. The “cascade of two 1D causal reasoners” framing is also worth tracking against generative-side work: it’s the same shape-of-problem (decompose 2D into stacked 1D autoregression) that recurs in autoregressive image/video models. The open release (HF weights + GitHub code + paper) and the explicit second-use-case of “high-quality training data for LLM pretraining” make this an unusually production-ready research artifact.