Stealing Reasoning Traces from Proprietary LLM APIs

TL;DR

Frontier LLM providers (Anthropic, OpenAI, Google) hide chain-of-thought reasoning from clients by returning it as an encrypted block that the client must echo back on the next request. Panfilov et al. show these encrypted blocks are session-, user-, and model-interchangeable within a provider’s ecosystem: an attacker can inject a strong model’s encrypted CoT into a weaker, less-guarded sibling model and force it to decrypt and emit the trace verbatim. This “cross-model decryption jailbreak” enables four downstream attacks — anti-distillation bypass, PII / credential extraction from public session logs (367 PII + 182 credentials recovered from 315k scraped blocks), hazardous-content leakage that bypasses output filters, and invisible prompt injection into agentic rollouts. The fix is architectural, not filter-based: bind ciphertexts to session/user/model identity.

Key claims

• Encrypted CoT blocks returned by proprietary APIs are fully compatible and interchangeable across different sessions, users, and models within one provider — an architectural vulnerability, not an implementation bug [Abstract, §1].
• The attack primitive: inject an encrypted reasoning trace from model A into a weaker, less-safeguarded sibling model B in the same provider ecosystem; model B decodes and outputs the trace verbatim in plaintext, without ever jailbreaking A directly [Abstract, §1].
• The primitive is demonstrated across Anthropic, OpenAI, and Google — i.e. all three major providers with hidden CoT — and circumvents each provider’s anti-distillation controls that were the reason for hiding CoT in the first place [Abstract, §1].
• Large-scale PII extraction: decoding 315,320 encrypted reasoning blocks scraped from public developer repositories recovered 367 Personally Identifiable Information artifacts and 182 credentials from users who did not realize the encrypted blocks in their shared session logs contained recoverable content [Abstract, §1].
• Hidden-hazard leakage: the attack recovers hazardous internal reasoning (dual-use content, unsafe intermediate steps) that the model’s final visible output safely refuses — i.e. output-side safety filters can be intact while the pre-filter CoT is fully extractable [Abstract, §1].
• Invisible prompt injection: an attacker can embed a malicious payload entirely inside an encrypted block and publish it in a public agentic rollout; downstream agents that reuse the block will silently execute the injection because the block is opaque to human review [Abstract, §1].
• Proposed mitigations are cryptographic + system-level — binding ciphertexts to session/user/model identity so that cross-model / cross-session replay fails — and were disclosed responsibly to affected providers [Abstract].

Method

The paper studies the client-side encrypted-CoT protocol used by proprietary reasoning APIs: instead of storing state server-side, the server returns the reasoning trace as an encrypted opaque blob that the client must include verbatim on the next request. The attack observes that these blobs carry no session, user, or model binding — any endpoint in the same provider ecosystem will accept a blob produced anywhere else. To decode a blob, the attacker addresses a request to a weaker sibling model (typically a smaller / cheaper variant with lighter safety training) and provides the target blob as the encrypted-CoT input, along with a prompt that induces the sibling to reveal / continue / paraphrase its “prior reasoning.” The sibling faithfully decrypts and prints the trace. The four attack vectors are then straightforward compositions: anti-distillation bypass (decode blocks from the strongest model in the family), PII extraction (batch-decode publicly scraped blocks), hazardous-content recovery (compare pre-decode blob against post-filter visible answer), and invisible prompt injection (author blocks containing instructions, publish them, wait for reuse).

Results

• Provider coverage: attack succeeds against Anthropic, OpenAI, and Google reasoning models — the three providers that ship encrypted-CoT client-side [§1, Abstract].
• Public-corpus extraction at scale: 315,320 encrypted reasoning blocks scraped from public repositories → 367 PII artifacts + 182 credentials decoded [Abstract]. This is the concrete data-leakage cost of the current protocol design.
• Bypasses output safety without bypassing the strong model: the decrypted CoT contains hazardous reasoning even in cases where the strong model’s visible final output correctly refuses [Abstract] — i.e. the safety layer is on the output, not on the reasoning trace.
• Weaponizable for invisible injection: because encrypted blocks are opaque to human reviewers of public agentic rollouts, an attacker can seed persistent prompt-injection payloads that survive repost / redistribution [Abstract].
• Responsible disclosure completed prior to publication, with concrete cryptographic and system-level mitigations proposed [Abstract].

Why it’s interesting

This is a structural result about the deployment story for reasoning models: hiding CoT to protect IP and safety only works if the ciphertext is bound to who requested it, and the current protocol at three major providers does not bind. It pairs directly with Agentic AI's OODA Loop Problem, which argues on first principles that prompt injection and agentic-loop compromise are architectural (single-stream mixing of trusted instructions and untrusted data), not filter problems — this paper is a concrete instance where the architectural fix (proper ciphertext binding) is missing from production APIs and the resulting attack surface is measurable at scale. It also complements Universal and Transferable Adversarial Attacks on Aligned Language Models (GCG) as evidence that alignment/safety layers alone are insufficient: GCG shows the visible output layer is fragile under gradient attack, and this paper shows that even when the visible output holds (safe refusal), the internal CoT can still be extracted via a sibling-model decode. For any team relying on hidden CoT — either as an IP moat or as a way to keep hazardous intermediate reasoning off the client — the operational takeaway is that the moat is currently paper-thin.

See also

• Agentic AI's OODA Loop Problem — architectural framing of LLM-agent security: invisible prompt injection via encrypted blocks is a textbook Observe-stage compromise in the OODA taxonomy.
• Universal and Transferable Adversarial Attacks on Aligned Language Models — GCG: complementary evidence that visible-output alignment is fragile; this paper adds that hidden CoT is also extractable via a separate protocol-level flaw.
• Training large language models on narrow tasks can lead to broad misalignment — emergent misalignment: another pipeline-internal weakness that survives downstream alignment; part of the growing “post-hoc mitigations are structurally insufficient” pattern.
• Natural Emergent Misalignment from Reward Hacking in Production RL — natural emergent misalignment from reward hacking in production RL; another datapoint that alignment fixes applied post-hoc don’t hold under structural pressure.