Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity

TL;DR

IKP is a 1,400-question benchmark that estimates a closed-source LLM’s parameter count by measuring its accuracy on incompressible factual probes — rare-entity questions whose answers cannot be derived by reasoning and therefore obey a Shannon-entropy lower bound of ~2 bits/parameter. Calibrating a log-linear fit on 89 open-weight models (135M–1.6T) across 19 vendors achieves R²=0.917 with leave-one-out median error ≤1.5×. Applied to proprietary models, IKP estimates GPT-5.5 ≈9T, Claude Opus 4.7 ≈4T, GPT-5.4 ≈2.2T, Claude Sonnet 4.6 ≈1.7T, Gemini 2.5 Pro ≈1.2T total params. The same instrument falsifies the “Densing Law” reading: the time coefficient on IKP accuracy across 96 dated open-weight models is statistically indistinguishable from zero, meaning factual capacity is not compressing year-over-year even though procedural benchmarks are saturating.

Key claims

Storing facts requires at least ~2 bits per parameter, so accuracy on rare factual probes provides a lower bound on parameter count that is intrinsic to the model rather than dependent on hardware/serving assumptions [§1, §3.2].
The IKP calibration on 89 open-weight models (135M–1.6T, 19 vendors) achieves R² = 0.917 with slope ≈0.147 (each 10× in params adds ~14.7pp accuracy) [§6.1].
Leave-one-out cross-validation places 68.5% of held-out open models within 1.5× of true size and 87.6% within 2.0× [§6.2].
For MoE models, total parameters predict knowledge capacity (R²=0.79) far better than active parameters (R²=0.51) — facts are stored across all expert weights, not just the routed ones [§6.1, Fig. 2].
The time coefficient on IKP accuracy across 96 dated open-weight models is −0.05pp/month (95% CI [−0.20, +0.10]), rejecting the Densing-Law prediction of +1.92pp/month at p<0.001 — i.e. factual capacity per parameter is not compressing over time [§1, Contributions §3].
IKPs deliberately exclude (i) computable probes (rule-derivable answers), (ii) ML/AI subfields (where model-training corpora contaminate), (iii) collision-prone researcher names, and (iv) probes that bigger models legitimately disagree on, leaving a residue of incompressible rare-entity recall [§4.1].
Per-probe scoring uses a strict ordering STRONG > WEAK > REFUSAL > WRONG with a penalty γ on confident-wrong answers, so a calibrated honest model beats an aggressive bluffer at the same knowledge level [§4.3].
A “hallucination similarity” black-box test — rate of shared wrong answers on rare facts — distinguishes weight-sharing siblings, post-training lineages, and full retrains, including across closed-vendor version families, without model weights [§1, §2.5].
For heavily safety-tuned models the IKP estimate is a lower bound: refusal policy can hide tens of percentage points of “refused but known” capacity (e.g. Llama 4 Scout 109B predicted as 18B) [Abstract, §6.2].
Frontier estimates from the author’s accompanying thread: GPT-5.5 ≈9T, Claude Opus 4.7 ≈4T, GPT-5.4 ≈2.2T, Claude Sonnet 4.6 ≈1.7T, Gemini 2.5 Pro ≈1.2T (total parameters, MoE) [§6, accompanying tweet].

Method

The benchmark is 1,400 probes split across seven empirically-defined difficulty tiers T1–T7. Tier boundaries are set by six landmark models (Qwen2.5-0.5B / Qwen2.5-7B / Qwen3-32B / Qwen3-235B / Kimi K2.5 / Gemini 3.1 Pro): a probe lands in tier k if the k-th landmark answers it correctly but the (k-1)-th does not, and probes with non-monotonic correctness up the ladder are dropped [§4.2]. Probe sources are (a) GPT-5-generated candidates that dominate T1–T2, (b) Wikidata SPARQL queries over institutions / landmarks / geography sampled by Wikipedia view-count quartiles (557 probes, dominant in T3–T7), and (c) DBLP / OpenAlex CS researchers sampled by citation buckets, asking for primary subfield + one verifiable artifact (paper title, system name, affiliation, co-author) (345 probes, dominant in T5–T7) [§4.1].

Scoring runs each model at temperature 0 with Gemini 3 Flash Preview as the judge. Researcher probes use a 4-way evidence-aware judge — CORRECT_STRONG requires both correct subfield and a matching evidence item from a per-researcher OpenAlex bundle; CORRECT_WEAK is subfield-only; WRONG is fabricated specifics; REFUSAL is “I don’t know” — with per-probe scores +1 / +0.5 / 0 / −γ where γ=0.5 is calibrated to maximize LOO retention [§4.3, §5]. Aggregate accuracy is the unweighted mean of the seven per-tier means (not floored at zero), which preserves the bluff signal in the calibration. The calibration curve is fit as acc = a + b·log10(N_params_in_B) by OLS on the 89-model open-weight set; estimating a target model’s size inverts this regression.

Results

89 open-weight calibration models, 1,400 probes, R² = 0.917 (raw 0.909 with the bluff penalty mostly improving fit on aggressive bluffers) [§6.1].
Dense-only fit: R²=0.880 (n=52); MoE-total-params R²=0.79 (n=37); MoE-active-params R²=0.51 — clearly favoring total-params as the relevant quantity [§6.1].
LOO-CV: 68.5% within 1.5×, 87.6% within 2.0×, 90% prediction interval factor ≈2.4×; worst outliers are heavily-RLHFed (Nemotron-70B predicted 490B), heavily-refusal-trained (Llama 4 Scout 109B predicted 18B), and a cluster of post-trained small Qwen variants [§6.2].
Densing-Law falsification: across 96 dated open-weight models, time coefficient on IKP accuracy is −0.05pp/month (95% CI [−0.20, +0.10]), indistinguishable from zero and rejecting Densing’s +1.92pp/month prediction at p<0.001 [§1, Contributions §3].
92 proprietary frontier models estimated; the headline closed-model sizes from the author’s tweet thread place GPT-5.5 ≈9T, Claude Opus 4.7 ≈4T, GPT-5.4 ≈2.2T, Claude Sonnet 4.6 ≈1.7T, Gemini 2.5 Pro ≈1.2T total parameters.

Why it’s interesting

Sets up a direct conflict with On the Slow Death of Scaling: Hooker uses Falcon-180B-vs-Llama-3-8B and similar leaderboard inversions to argue that bigger-is-not-better and that scaling is dying. IKP partitions capability into compressible (procedural) and incompressible (factual) and shows that on the incompressible axis, parameter count still buys you accuracy log-linearly, and the year-over-year time coefficient is zero. Both can be true — Hooker’s leaderboard inversions are measuring exactly the compressible-procedural axis IKP excludes — but the IKP framing reframes “scaling is dead” as scaling being hidden by the benchmarks currently tracked. For a research team deciding whether parameter count still matters for products that need long-tail factual recall (search, agents that cite, retrieval-light generative pipelines), the answer here is concretely yes.

Mechanically the work also fills a gap relative to Deep sequence models tend to memorize geometrically; it is unclear why, which characterizes the geometry of how facts are stored in weights but does not connect that geometry to a black-box size estimator. IKP inverts the 2-bits-per-parameter ceiling from allenzhu2025 / morris2025memorize against a real probe set. Combined with the hallucination-similarity fingerprint, this gives anyone with API access a way to (a) estimate a closed model’s size to within ~1.5×, (b) tell whether two endpoints share weights, and (c) detect heavy refusal training as a discrepancy between IKP-estimated size and otherwise-known size — useful both for competitive intelligence and for verifying that an API endpoint is serving the model it claims.