OpenAI Model Craft: Parameter Golf

TL;DR

OpenAI’s Parameter Golf is an eight-week (March 18 – April 30, 2026) open challenge to train the best language model that fits in a 16 MB artifact (weights + training code combined) and trains in under 10 minutes on 8×H100 SXM GPUs, scored by tokenizer-agnostic bits-per-byte on a fixed FineWeb validation set. OpenAI framed it as an L(N) scaling-law exercise — minimize validation loss at fixed parameter budget — explicitly modeled on the NanoGPT Speedrun, with the README naming depth recurrence, aggressive parameter tying, low-rank training, quantization-aware training, novel tokenizers, and test-time compute as expected design directions. Runpod sponsored $1M in compute credits; submissions went through PRs to the openai/parameter-golf GitHub repo. OpenAI also positioned the challenge as a talent-discovery surface and a way to study how coding agents change the dynamics of open ML competitions.

Key claims

The challenge is an L(N) optimization problem: minimize FineWeb held-out loss at fixed parameter budget, with the artifact (weights + training code) capped at 16 MB and training capped at 10 minutes on 8×H100 SXM [challenge page; GitHub README].
Evaluation is tokenizer-agnostic via bits-per-byte on a fixed FineWeb validation set — chosen explicitly so submitters cannot game the metric by manipulating vocabulary [challenge page; baseline ships with a SentencePiece BPE tokenizer at vocab 1024].
The README enumerates the expected design directions for staying competitive under the constraint: depth recurrence, aggressive parameter tying, low-rank training, quantization-aware training, novel tokenizers, test-time training, test-time compute, long context, and megakernels [GitHub README].
The challenge is “heavily inspired by” the NanoGPT Speedrun, but constrains parameter count instead of wall-clock time — making it the parameter-constrained cousin of the speedrun (time-constrained) and the slowrun (data-constrained) variants [GitHub README].
Non-record submissions and an “Unlimited Compute Track” are explicitly welcomed for ideas that don’t beat SOTA but demonstrate novel approaches — the contest is also a research-signal collection mechanism, not just a leaderboard [GitHub README; challenge page].
The challenge doubles as an OpenAI talent pipeline: standout participants may be invited to interview, and OpenAI’s framing references competitive math/programming olympiads as the design template [challenge page].

Method

This is a contest spec, not a paper. The substantive design choices are:

Artifact cap: combined model weights + training code bytes must fit in 16 MB (after standard compression of the model bytes). Forces submitters to write training code that is itself terse — text de-duplication / minification matters.
Compute cap: final leaderboard runs are bounded at 10 wall-clock minutes on 8×H100 SXM, costed at roughly $20/hour, making a single record attempt sub-$ 5 of GPU time. Iteration is meant to happen on cheaper SKUs (1×H100 or A100).
Scoring: bits-per-byte on FineWeb validation, computed in a tokenizer-agnostic way (the contest scores compression of the byte stream, so changing vocabulary doesn’t shift the metric). This is what makes “use a tiny novel tokenizer with high semantic density per token” a legitimate strategy rather than a metric exploit.
Submission protocol: PRs to openai/parameter-golf, with a new folder under records/track_10min_16mb/ containing the code, logs, and a write-up.
Infrastructure: pre-built Runpod template with dependencies installed, $1M in compute credits for participants, baseline reference implementation (an ~80-shard FineWeb-10B subset, ~8B tokens, SentencePiece-1024 BPE tokenizer).

Results

This page is the announcement, not the retrospective. Quantitative outcomes were reported separately in OpenAI’s May 12 follow-up Parameter Golf retrospective:

Eight-week run drew >2,000 submissions from >1,000 participants [May 12 post].
Winning approaches clustered into: careful optimizer / training tuning of existing components; aggressive compression and export (low precision, QAT, bitnet-style); test-time training; and some submissions that pushed evaluation-rule boundaries hard enough to need organizer review [May 12 post].
The “vast majority” of submitters mentioned using coding agents (Claude / Codex / equivalents) as part of their workflow — and submissions clustered around top scorers via agent-mediated diffusion of ideas, both helping and creating scoring noise [May 12 post].
No single magic trick dominated; the leaderboard converged via many small, disciplined trade-offs across architecture, compression, evaluation, and engineering [May 12 post].

The May 12 post is not in this Slack thread but is the natural follow-up artifact; it may be filed separately when posted to #research-external.

Why it’s interesting

This is the LLM analog of the speedrun lineage that has been driving recent diffusion-training-efficiency work — NanoGPT Speedrun → Parameter Golf for language, and the same template inspiring PRX Part 3 — Training a Text-to-Image Model in 24h (24h $1500 T2I speedrun on H200s), Bhanded’s “Speedrunning ImageNet Diffusion” (arXiv:2512.12386), and the Modal “Run FLUX.1-dev three times faster” blog (Run FLUX.1-dev three times faster). Constraint-driven open contests appear to be the medium where ideas like aggressive parameter tying, low-rank training, and test-time compute get pressure-tested at a meaningful scale before they show up in production recipes. The 1024-vocab baseline is also a useful provocation for the STEM: Scaling Transformers with Embedding Modules / sparse-ffn-via-embedding-tables thread: when every token has to carry more semantic weight, the trade-off between learned-routing MoE and static embedding-table sparsity is unusually sharp. Finally, the agent-mediated submission dynamics noted in the retrospective are an early empirical reading on what open ML competitions look like once participants routinely delegate code-writing to coding agents — relevant for any team designing in-house leaderboards.