MemPalace launch tweet (@bensig) and the Community Note debunking the 100% benchmark claim

TL;DR

On April 7, 2026, Ben Sigman (@bensig, CEO of Bitcoin Libre) launched “MemPalace” — an open-source AI memory system co-credited to actress Milla Jovovich — claiming a 100% perfect score on the LongMemEval benchmark, 92.9% on ConvoMem, and 100% on LoCoMo, framed as the first product to ever hit a perfect score on the standard memory benchmark. The tweet went viral (23k+ GitHub stars in two days, ~41k within a week). Within hours, X readers added a Community Note pointing out that the 100% LongMemEval used targeted fixes for the 3 failing questions plus LLM reranking (held-out score: 98.4%), and the 100% LoCoMo used top-k=50 against a candidate pool that maxes out at 32 sessions (honest top-10 no rerank: 88.9%). Subsequent code analysis showed the “palace” architecture (wings/halls/rooms) was uninvolved in the raw score, which essentially measured ChromaDB’s default embedding model. The “Lu” referenced in repo internals — initially suspected to be a ghost-developer — turned out to be a persistent Claude Code instance.

Key claims

Original tweet headline: “perfect score on the standard benchmark — beating every product in the space, free or paid”; lists 100% LongMemEval (500/500), 92.9% ConvoMem (2× Mem0), 100% LoCoMo including temporal-inference [tweet body].
Community Note attached to the same tweet: 100% LongMemEval used targeted fixes for the 3 failing questions and LLM reranking; held-out score is 98.4% [tweet Community Note].
Community Note: 100% LoCoMo used top-k=50 exceeding the 32-session candidate pool with reranking; honest top-10 no-rerank score is 88.9% [tweet Community Note].
After the controversy, the repo README was revised to lead with 96.6% R@5 on LongMemEval in raw mode (no LLM, no API), and the team explicitly stopped headlining the 100% figure [repo README].
Independent reproduction on M2 Ultra confirmed raw 96.6% but found the palace structure (wings/halls/rooms) regresses retrieval vs. flat ChromaDB — i.e., the headline architecture is anti-functional on the benchmark it was sold on [external analysis cited in repo HISTORY].
“Lu” is not a ghost developer hired to write the code; it is a persistent Claude Code instance that Jovovich works with daily — clarified by Sigman after the GitHub-history sleuthing [follow-up X posts].

Method

The artifact is a launch tweet plus its attached X Community Note. The thread is structurally interesting because the debunk lives on the tweet itself (Community Notes), not in a reply: anyone seeing the original viral claim now sees the corrected numbers in the same render. The underlying technical claim is straightforward: MemPalace stores conversation chunks verbatim in ChromaDB and retrieves them with semantic search, organized into a hierarchy of “wings” (top-level domains), “rooms” (topic categories within a wing detected by a 60-keyword scorer), “halls” (memory types), and “drawers” (raw chunks). The benchmark gaming used (a) hand-coded fixes for the specific failing questions on LongMemEval and (b) a top-k that exceeded the candidate-pool size on LoCoMo, effectively turning retrieval into “give the LLM everything and let it read.”

Results

LongMemEval R@5: claimed 100% → corrected to 96.6% raw / 98.4% held-out / 100%-with-targeted-fixes.
LoCoMo: claimed 100% → corrected to 88.9% honest top-10 no-rerank.
GitHub virality: 23k stars in 48h, ~41k in a week — a strong signal of how much “celebrity + perfect benchmark” beats “actually good system” in audience capture, independent of the benchmark validity.
Code analysis (independent): the palace structure reduces retrieval quality vs. flat ChromaDB; the 96.6% number is essentially ChromaDB’s default embedding model on this benchmark.

Why it’s interesting

This is the cleanest live example the wiki has of benchmark gaming via top-k inflation and per-question patching, caught and corrected on the same surface as the original claim via Community Notes — a useful counter-data-point against trusting headline numbers from viral launches. It pairs with TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate in an unexpected way: SDxCentral cites both MemPalace and TurboQuant in the same breath as “architectural workarounds for the AI memory crisis,” even though TurboQuant is a legitimate KV-compression paper and MemPalace is a ChromaDB wrapper with a metaphor on top — a reminder that the press conflates infrastructure-level memory (KV cache) with application-level memory (conversation history). It also touches the same evaluation-trust problem flagged in A Very Big Video Reasoning Suite (VBVR) and Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoning — namely, that retrieval/eval suites with small candidate pools or fixed seeds are trivially gameable, and the right defense is held-out splits and honest top-k disclosure.