Karpathy: pixels may be better LLM inputs than text tokens (DeepSeek-OCR springboard)

TL;DR

Andrej Karpathy uses the DeepSeek-OCR release as a springboard for a broader thesis: pixels may be a strictly better input substrate to LLMs than text tokens. The argument is that rendering even “pure text” to images before feeding it to a model (a) compresses context dramatically, (b) widens the input information stream (layout, bold/color, diagrams, arbitrary embedded images come for free), (c) admits bidirectional attention at the input stage by default, and (d) deletes the tokenizer along with its Unicode baggage, security pitfalls, and visual-blindness. The OCR ranking (“maybe a bit worse than dots”) is not the point — the point is that the choice of input modality has first-order systems implications.

Key claims

DeepSeek-OCR is a “good” OCR model — possibly behind dots.ocr on raw accuracy — but the OCR ranking is incidental; the interesting content is the modality-choice argument it surfaces [tweet body].
Pixel inputs compress information more densely than text tokens, so the same context fits in shorter sequences — the main lever DeepSeek-OCR demonstrates quantitatively [tweet body].
Pixel inputs are a more general information stream: text formatting (bold, color), arbitrary embedded images, and document layout flow through the same channel that carries the text itself — no separate modality handling [tweet body].
Pixel inputs naturally admit bidirectional attention at the input stage; text-token inputs are autoregressive by default, which Karpathy characterizes as the weaker option for processing (as opposed to generation) [tweet body].
Deleting the tokenizer removes a known source of pathologies: Unicode-related security issues, identical-looking-but-differently-tokenized strings, and the fact that current tokenizers see emoji as opaque symbols rather than pixel structure [tweet body].
The asymmetry persists across the I/O boundary: pixels are the right input modality, but text remains the natural output modality — a generation pipeline still produces text tokens, only the comprehension side switches [tweet body].

Method

A short opinion thread, not an experiment. Karpathy’s structure: (1) note that DeepSeek-OCR is a competent OCR model whose more interesting contribution is the optical-context-compression framing, (2) generalize from that framing to a four-bullet argument for pixels-as-LLM-input, (3) flag that the implication is asymmetric — text is fine for outputs, the case is specifically for inputs.

Results

The tweet is an argument, not a measurement. The empirical anchor it leans on is DeepSeek-OCR’s own Fox-benchmark compression curve (~97% decoding precision at ≤10× text-to-vision-token compression, ~60% at 20×), already filed at DeepSeek-OCR: Contexts Optical Compression [§4.1, Table 2 of that paper]. No new numbers in the tweet itself.

Why it’s interesting

The framing matters for Long context as weights because it elevates the “vision tokens as long-context substrate” footnote already in the page (added when DeepSeek-OCR was filed) into a first-class hypothesis: rather than relocating long context into weights (SGD over context as in End-to-End Test-Time Training for Long Context, hypernetwork-LoRA as in Doc-to-LoRA: Learning to Instantly Internalize Contexts, or RL-distilled-LoRA as in Self-Adapting Language Models), relocate it into a visual representation that the existing multimodal stack already ingests. Karpathy’s argument is the strongest non-paper articulation on the wiki that the optical substrate may be load-bearing, not a curiosity. It also intersects Unified Multimodal Models from the input side — if pixels are the universal input interface, the current text-encoder + image-encoder split in most VLMs is the wrong factoring, and the obvious next move is a single pixel-only encoder that treats rendered text and natural images uniformly (something Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation also argues from the empirical direction).