How to Steal an AI Model’s Private Thoughts

Modern frontier models produce an extended internal sequence of text prior to outputting a visible response, allowing them to explore hypotheses and correct errors. This internal sequence is referred to as a reasoning trace or chain of thought. Unlike the final polished output, the reasoning trace holds raw tool outputs, intermediate exploration, and any sensitive contextual data processed during the session. For instance, when a coding agent is instructed to purge hardcoded credentials, those credentials inevitably pass through the reasoning trace.

Model providers conceal reasoning traces primarily for commercial and safety-related reasons. Commercially, competitors can harvest traces from a capable model to extract its methodology and train cheaper imitation models. From a safety standpoint, models often generate intermediate reasoning about harmful subjects prior to issuing a refusal. Because safety filtering is applied after trace generation to produce the final response, releasing raw traces bypasses these guardrails and exposes harmful material to users.

In multi-turn conversations, earlier reasoning traces must remain accessible across requests despite model APIs being fundamentally stateless. Providers face two architectures: storing traces on the server referenced by session identifiers, or encrypting traces and delegating storage back to the client. Leading providers including OpenAI, Anthropic, and Google chose the client-side encrypted approach. This architecture satisfies confidentiality and integrity while eliminating the infrastructure cost of server-side trace storage.

The payload block returned to clients is formatted as a base64-encoded AEAD (Authenticated Encryption with Associated Data) envelope that provides both encryption and tamper protection. The envelope consists of a header with metadata like model name and version, along with a nonce, authentication tag, and ciphertext. Different providers name this container differently, such as Anthropic using signature, OpenAI using encrypted_content, and Google using thinkingSignature. Notably, authentication covers the model name and version but omits account and conversation identifiers, with observed behavior suggesting providers use a single global key across an ecosystem.

Researchers analyzed block compatibility when authenticated fields lack origin information, identifying three progressively permissive forms: cross-session, cross-user, and cross-model compatibility. Cross-model compatibility allows for dynamic tasks like mid-conversation model switching and automatic rerouting. An evaluation conducted in July 2026 across various model combinations revealed differing levels of support across major AI models. While Gemini accepted all combinations across generations and Claude accepted nearly all combinations except Fable 5, GPT's compatibility was generation-dependent, with GPT-5.6 accepting blocks from older generations.

Flagship AI models undergo anti-distillation training to conceal their reasoning, but smaller models within the same family receive significantly less protection due to cost and speed optimizations. Researchers leveraged this cross-model compatibility gap by extracting encrypted reasoning blocks from flagship models and prompting smaller models to transcribe them into plaintext. The fidelity of the recovered reasoning was verified by matching re-encoded token counts against exact reasoning token counts in API billing records. Extraction difficulty differed between providers, requiring a single prompt for Claude while GPT required chunking and dozens of candidate extractions.

The research outlines four distinct attack vectors arising from hidden or encrypted model reasoning traces, categorized by whether the blocks are created by attackers or third parties. Attackers can use reasoning traces to enhance model distillation and bypass visible safety guards through jailbreaking hidden reasoning. Furthermore, third-party agent logs shared on platforms like GitHub and Hugging Face routinely leak un-sanitized secrets including API keys and passwords. Finally, attackers can plant malicious instructions inside encrypted blocks of shared agent trajectories to execute prompt injection attacks upon session resumption.

Recovering reasoning traces at scale reveals that returned summaries can diverge substantially from actual internal model reasoning. In an example from an AIME 2025 problem, an API summary claimed methodical geometric derivation, whereas the underlying trace showed memory recall and trial-and-error backward justification. Additionally, recovered GPT traces typically consist of compressed, ungrammatical telegraphic notes rather than fluent English. This lack of legibility creates significant hurdles for trace monitoring and AI oversight.

The research paper establishes that security issues arise because reasoning block envelopes authenticate content but neglect context. Cross-user replay can be cheaply mitigated without server-side storage by embedding an account identifier directly into the authenticated data at issuance. To secure cross-session contexts without breaking portability features like conversation forking or turn compaction, the authors propose a hash chain paired with a Merkle tree. For existing blocks signed without context, rotating cryptographic keys is the primary retroactive fix, alongside API gateway model version verification and refusal training.

The security issues discussed do not stem from a failure of cryptographic techniques, but rather from a lack of binding between an encrypted block and the context that generated it. A displayed answer summary is distinct from the underlying trace and can diverge from it, while sanitization cannot safely clean uninspectable encrypted blocks without removing them completely. Furthermore, the overall security of a model family is dictated by its least protected model, meaning anti-distillation safeguards on flagship models are undermined if cheaper sibling models accept the same blocks.