DeepSeek-OCR: Contexts Optical Compression

TL;DR

DeepSeek-OCR is pitched as an OCR model but the actual thesis is optical context compression: render long text as an image, encode it into a small bag of vision tokens, and recover the text autoregressively. The model pairs a custom 380M DeepEncoder (SAM-base window-attention front end + 16× conv compressor + CLIP-large global attention) with a DeepSeek-3B-MoE/A570M decoder. On the Fox benchmark, decoding precision stays at ~97% while text-to-vision-token compression is below 10×, holds ~60% even at 20× compression, and on OmniDocBench it matches MinerU 2.0 quality at <800 vision tokens versus ~7,000. Framed by the authors (and by Luma’s poster) as evidence that “a picture is worth a thousand tokens” is literally measurable, and that visual modality may be a viable substrate for LLM long-context memory and forgetting mechanisms.

Key claims

Visual modality can act as a high-ratio compression channel for text: at ≤10× text-to-vision-token ratio the model decodes at ~97% precision; at ~20× compression precision still reaches ~60% [Abstract, §4.1, Table 2].
DeepEncoder is co-designed around four constraints — high input resolution, low activation memory, few output vision tokens, and multi-resolution support — none of which existing open VLM encoders (NaViT, tile-based InternVL, dual-tower Vary) satisfy simultaneously [§2.1, §3.2].
The DeepEncoder architecture is two serial components — a window-attention SAM-base front end (80M) and a global-attention CLIP-large back end (300M) — separated by a 2-layer convolutional 16× token compressor; e.g. a 1024² input produces 4096 patch tokens that are compressed to 256 before global attention runs [§3.2.1].
The DeepSeek-3B-MoE decoder activates 570M parameters at inference (6 of 64 routed experts + 2 shared), so the total end-to-end inference cost of the OCR model is closer to a 500M model than a 3B one [§3.3].
Five resolution modes (Tiny 64 / Small 100 / Base 256 / Large 400 vision tokens, plus a Gundam dynamic mode mixing n×640² tiles with a 1024² global view) are trained jointly via positional-encoding interpolation [§3.2.2, Table on modes].
On OmniDocBench, Small (100 tokens) beats GOT-OCR2.0 at 256 tokens; Gundam (795 tokens) beats MinerU2.0 at ~6790 tokens (overall ED 0.127 vs 0.133) [§4.2, Table 3].
Per-document-type analysis: slides need only 64 tokens for satisfactory parsing; books and reports work at 100; newspapers require Gundam or Gundam-master modes because raw text count exceeds the ~10× compression boundary [§4.2, Table 4].
Production throughput: 200k+ pages per day per A100-40G; with a 20-node cluster (160 A100s) the authors report 33M pages/day, framed as an LLM/VLM pretraining-data flywheel [Abstract, §1].
Training is two-stage with pipeline parallelism: (a) DeepEncoder alone trained via next-token prediction with a compact LM on all OCR 1.0/2.0 + 100M LAION samples, then (b) full DeepSeek-OCR trained on a 70% OCR / 20% general-vision / 10% text-only mix using PP=4 (SAM+compressor frozen in PP0, CLIP unfrozen in PP1, decoder split across PP2/PP3) [§3.5.1, §3.5.2].
The Fox-benchmark compression curve is positioned as a quantitative starting point for studying forgetting mechanisms in LLMs — historical context tiers could in principle be re-rendered to images at increasingly low resolution as memories age [§1, §4.1].

Method

DeepEncoder is the central artifact. The forward pass on a 1024² image: 64×64=4096 patch tokens after SAM-base patch embedding → window-attention through 80M SAM-base → 2-layer conv compressor with kernel 3, stride 2, padding 1, channels 256→1024 → 16× token reduction to 256 tokens → CLIP-large global attention with patch-embedding layer removed (its input is no longer image pixels). The first half operates on many tokens but with cheap window attention; the second half operates with expensive dense attention but on only 256 tokens, keeping peak activation bounded. Resolution-mode switching is implemented by dynamic positional-encoding interpolation rather than separate encoder weights, with five modes co-trained so a single checkpoint handles all of them. The Gundam dynamic mode tiles ultra-large inputs (e.g. newspapers) into n×640² local views plus one 1024² global view, with n∈[2,9].

The decoder (DeepSeek-3B-MoE) is a standard MoE language model with 12 layers, 64 routed experts, 6 active, 2 shared, ~570M activated parameters; the only OCR-specific adaptation is feeding the compressed vision tokens as a prefix and conditioning on a small set of prompts (e.g. <image>\nFree OCR., <image>\n<|grounding|>Convert the document to markdown., <image>\nLocate <|ref|>xxxx<|/ref|>). Training data covers ~30M PDF pages across ~100 languages with both coarse (fitz-extracted) and fine (PP-DocLayout + MinerU + GOT-OCR2.0-labeled) annotations, plus 20M scene-OCR images, OCR 2.0 synthetic data (10M charts via pyecharts/matplotlib, 5M chemical formulas via RDKit on PubChem SMILES, 1M plane-geometry images), 20% general-vision data, and 10% in-house text-only pretrain at 8192 sequence length [§3.4, §3.5].

Results

Fox compression study (English, 600–1300-token documents, 100 pages): at Vision Tokens = 100, precision is 98.5%/97.3%/96.8%/96.8%/91.5%/89.8%/87.1% across text-token buckets 600–1300; compression ratios go from 6.7× to 12.6×. At Vision Tokens = 64, precision degrades from 96.5% (10.5×) down to 59.1% at 19.7× [§4.1, Table 2].
OmniDocBench overall edit distance: Tiny=0.386 (EN) / 0.361 (ZH); Small=0.221 / 0.284; Base=0.137 / 0.240; Large=0.138 / 0.208; Gundam=0.127 / 0.181; Gundam-master=0.123 / 0.157 [§4.2, Table 3]. Compared baselines: GOT-OCR2.0 (256 tokens) 0.287 / 0.411; MinerU 2.0 (6790 tokens) 0.133 / 0.238; Qwen2.5-VL-72B (3949 tokens) 0.214 / 0.261; Gemini 2.5 Pro 0.148 / 0.212.
Document-type breakdown at the four resolution modes [§4.2, Table 4]: slides are best-served by Tiny (0.116) and barely improve with more tokens; newspapers require Gundam (0.122) or Gundam-master (0.099); academic papers are tractable at Base (0.052).
Throughput: 90B text tokens/day or 70B multimodal tokens/day during training on the 20-node A100-40G HAI-LLM setup; 200k+ pages/day inference per A100-40G in production [§3.5.2, Abstract].

Why it’s interesting

The paper’s framing makes it most relevant as a sibling thread in Long context as weights: where TTT-E2E, Doc-to-LoRA, and SEAL push long-context memory into network weights, DeepSeek-OCR pushes it into a small bag of vision tokens that the LLM ingests through its existing multimodal interface. The forgetting-mechanism framing (re-rendering aged context at progressively lower image resolution) is a different point in the design space than KV-cache eviction or hypernetwork-emitted LoRAs, and it’s the first filed datapoint that proposes optical compression as the substrate. It also pairs cleanly with the already-filed DeepSeek-OCR 2: Visual Causal Flow (DeepSeek-OCR 2), which keeps the SAM+conv front end of this paper unchanged and swaps only the CLIP back end for an LLM-style causal-query mini-encoder — so the two papers together isolate the “where does the compression actually come from” question to the global-attention back end. On the encoder-architecture axis, this work contrasts with Don't Look Twice: Faster Video Transformers with Run-Length Tokenization (RLT) which drops redundant tokens pre-network rather than compressing them via a learned bottleneck; the comparison is fair because both target the “video/document tokens are not naturally 1D” problem from opposite directions.