microgpt: GPT training and inference in ~200 lines of dependency-free Python

TL;DR

Karpathy distills GPT training and inference down to ~200 lines of dependency-free Python, framed explicitly as an art project: the goal is to expose the entire algorithmic core of an LLM (no NumPy, no PyTorch, no SciPy — just + * ** log exp) sitting on top of a scalar-valued autograd engine in the spirit of micrograd. After the first release at 243 lines, a follow-up simplification — returning only local gradients per op and letting backward() do the chaining with the global gradient — drops it to ~200 lines. Everything in a “real” LLM stack beyond this is, by Karpathy’s claim, in service of efficiency rather than algorithmic content.

Key claims

The full algorithmic content of training and running a GPT can be expressed in ~200 lines of pure Python with no external dependencies; everything else in a production stack is efficiency, not algorithm [tweet, follow-up post].
The forward/backward of every primitive op can be expressed by returning only its local gradients, with backward() performing the chain-rule multiply against the global gradient from the loss — this is what enables the 243 → ~200 line collapse [tweet, simplification follow-up].
The atomic mathematical primitives needed are exactly + * ** log exp; Adam is used as the optimizer [tweet].
The simplified code lays out cleanly in 3 columns at ~200 lines (presentation choice, but the author flags it as evidence the structure has reached a stable minimum) [tweet, follow-up].

Method

A scalar-valued autograd engine in the style of micrograd underlies the whole thing — every value is wrapped in a node that records its op and parents. The GPT architecture (token + positional embeddings, multi-head self-attention, MLP blocks, layernorm, output head) and its training loss are then expressed entirely in terms of those atomic ops, so gradients flow through the scalar graph rather than through tensor kernels. Adam is the optimizer. The follow-up refactor changes the autograd contract so that each op only declares its local derivatives (e.g. for c = a * b, the local grads w.r.t. a and b are b and a respectively); the chain-rule multiplication with the upstream global gradient is centralized inside backward() instead of being repeated in every op.

The artifact is hosted as a single HTML page at karpathy.ai/microgpt.html, mirroring the original gist for single-page readability.

Results

This is not a benchmark paper — there are no FID/PPL numbers reported. The headline number is line count: 243 → ~200 lines (~18% reduction) after the local-gradient refactor, for a code base that contains the complete forward + backward + optimizer + training loop of a GPT.

Why it’s interesting

A useful pedagogical and reference artifact: a single screen of code that bounds, from below, how much algorithmic surface area an LLM actually has. For Luma researchers it sets a clear delineation between “what the network is” and “what production code adds” (kernels, fused attention, mixed precision, sharding, dataloaders, checkpointing) — the latter is the entire stack but apparently none of the algorithm. The simplification trick — push chain-rule multiplication out of per-op backward into the autograd core, so ops only declare local gradients — is a generic autograd-design lesson worth filing for anyone touching custom op libraries.