Post-training 101: A hitchhiker's guide into LLM post-training

TL;DR

A tutorial-format introduction to LLM post-training by Han Fang and Karthik Abinav Sankararaman, framed as EP1 of a “Tokens for Thoughts” series. The post walks from pre-training (next-token prediction on Internet-scale corpora) to the post-training stack that turns a base model into an instruction-following assistant: supervised fine-tuning (SFT), RLHF, and — post DeepSeek R1 — reinforcement learning with verifiable rewards (RLVR). The framing piece is a conceptual side-by-side: prompted with “What is the capital city of U.S” (no question mark), a base model continues with the missing punctuation before answering, while an instruction-tuned model directly answers “Washington, D.C.” This is filed as a reference-style overview, not a research result.

Key claims

Pre-training encodes world knowledge into the base model via next-token prediction on large-scale text and image data; the same objective that makes the model knowledgeable also makes it less directly useful for most applications [§Pre-training vs. Instruction-tuning].
A base model prompted with an unterminated question (“What is the capital city of U.S”) will predict the missing question mark before continuing, while an instruction-tuned model answers directly — illustrated as the canonical behavioral gap that post-training closes [Fig. 1].
Post-training (a.k.a. fine-tuning) uses much smaller, far more curated data than pre-training and targets alignment (helpful / honest / harmless) plus capability improvement on top of pre-trained knowledge [§Post-training].
The current named techniques in the post-training stack are SFT, RLHF, and — post DeepSeek R1 — RLVR, with RLVR specifically called out as effective for reasoning and coding capabilities [§Post-training, citing DeepSeek R1].

Method

The post is an expository walk-through rather than an experimental paper. It introduces a vocabulary (base model, post-training, SFT, RLHF, RLVR) and uses a single worked example (the capital-city prompt with no question mark) to motivate why next-token-prediction models need an alignment step before they are usable as assistants. The piece positions itself as the first in a series — the authors note follow-ups will cover post-training “intermediate topics” and explicitly solicit reader requests.

Results

No quantitative results — this is a primer. The substantive content is the framing of the post-training stack as SFT + RLHF + (post-R1) RLVR and the side-by-side base-vs-instruct behavior on an unterminated prompt.

Why it’s interesting

Useful as a single-link onboarding pointer for the post-training vocabulary that the wiki’s reasoning-RL cluster takes for granted. It connects directly to Reasoning RL, which catalogs the GRPO/RLVR recipes that this post points at as the post-DeepSeek-R1 frontier, and to Training-Free Group Relative Policy Optimization and MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling — both of which assume the SFT → RL pipeline that this post introduces. It’s also a sibling-in-genre to The Big LLM Architecture Comparison and How To Scale in the “long-form educational ML reference” slot.