Do LLMs Have the Memory of a Goldfish?

The term 'memory' in the context of large language models refers to multiple distinct concepts rather than a single unified mechanism. It is critical to differentiate between these meanings to avoid confusion. This section serves as an introduction to examining and clarifying the various types of memory associated with LLMs.

During training, large language models encode patterns from vast amounts of data into billions of parameters or weights, enabling general knowledge recall without explicit prompt context. However, standard API interactions do not update these weights, meaning models lack persistent personal memory of individual conversations. Instead, models rely on a temporary context window as working memory to process instructions, conversation history, and retrieved context. Once this context window is cleared, the model loses access to that information unless it is reintroduced by the application.

Persistent memory operates external to the model, typically residing in systems such as databases, files, vector stores, or profile services. When required, the application retrieves this stored information and injects it directly into the active context. Consequently, models do not inherently possess persistent memory on their own. Instead, they depend on external auxiliary systems to supply persistent information during execution.

AI model API calls are inherently stateless, meaning the model does not store an internal record or memory of prior interactions once a request finishes. If an application requires context from earlier turns, it must re-send the preceding conversation history within the current request payload. As a result, most providers implement a stateless Messages API that depends on the client supplying the full dialogue history for multi-turn conversations.

A new model API call does not start completely from zero because it retains its trained weights, general language capabilities, platform safety instructions, and provided context. However, models inherently lack an automatically updated personal memory of specific users or past conversations. Outputs are strictly generated from the model's static weights combined with whatever context is supplied for that specific invocation. Even when APIs offer server-managed conversation state via session identifiers, the server is simply reconstructing the context for the model rather than imparting personal memory to it.

AI chatbots create the appearance of conversational recall by appending the full dialogue history, including system instructions and past user-assistant turns, into each subsequent prompt. To the underlying model, this interaction is simply an evaluation of a single large input rather than true temporal retention. Because application-level continuity exists, this technique is more appropriately classified as reconstructed or context-based memory rather than fake memory.

The context window of a large language model acts as a finite working-memory budget measured in tokens. Beyond the visible conversation, this budget must accommodate system instructions, tool definitions, retrieved documents, and the output response. When capacity is reached, applications must remove, compress, or replace content to stay within limits. Crucially, larger context windows do not ensure perfect recall, as models face a degradation phenomenon known as context rot when processing excessive information.

LLM APIs charge for processed tokens, causing the cumulative token cost of long conversations to grow rapidly as earlier history is reprocessed with every new turn. In addition to escalating financial costs, longer contexts increase generation latency. Prompt caching can mitigate latency and repetitive processing expenses by reusing prefixes such as system prompts and history. However, because cached tokens still consume context window space, prompt caching serves as an efficiency optimization rather than a true memory architecture.

Large language models and their hosting applications have no single universal behavior when reaching context window limits. Instead, systems use various strategies such as request rejection, message truncation, rolling windows, and compaction. Compaction preserves the core state of a conversation in a reduced representation to enable long-running sessions. However, this summarization technique is inherently lossy, meaning seemingly unimportant details might be prematurely discarded.

Real-world applications typically require combining multiple approaches to extend memory rather than relying on an isolated solution. Integrating different techniques allows systems to handle memory constraints more effectively in practice. This section introduces an exploration into several of these specific memory-extension methods.

Sliding-window memory retains only the most recent portion of a conversation by discarding the oldest messages when new ones arrive. A typical setup preserves system instructions, a fixed number of recent turns, and the current user input. While simple, fast, and predictable, this technique is best suited for dialogues where recent context matters far more than historical context. Its main disadvantage is permanent information loss once earlier requirements or facts exit the active context window.

Conversation summarization periodically compresses older conversation turns into concise descriptions to maintain the context efficiently. A standard context structure consists of system instructions, the conversation summary, recent unsummarized messages, and the current user query. However, summarization acts as an interpretation that strips nuance, simplifies uncertainty, and risks converting assumptions into false facts. To avoid progressive information distortion from repeated summarization, essential facts should be stored separately rather than entrusted to a narrative summary alone.

Structured entity extraction records specific conversational facts into predefined fields rather than storing interactions as raw prose. This approach provides higher reliability when an application needs to access exact project state and confirmed parameters. While effective for discrete items like user preferences, technical decisions, and workflow statuses, it struggles to capture nuanced narrative details that resist structured schemas.

Vector-store-backed memory enables conversational systems to perform semantic retrieval rather than appending entire conversation histories into context. The system segments past dialogue into chunks, transforms them into embeddings, and compares them against incoming queries to identify relevant passages. Only the highest-ranking passages are inserted into the prompt, effectively managing the model's context window. However, semantic search can retrieve irrelevant, outdated, or poorly matched memories, making metadata filtering crucial for accuracy.

A long-term user profile stores durable preferences and facts that remain useful across multiple conversations. Systems must distinguish between stable, repeatedly confirmed preferences and transient session-level details. Profiles require ongoing updates because user preferences evolve over time. Effective implementations enrich memory entries with metadata such as timestamps, sources, and confidence scores.

Cross-session memory enables information to persist across separate chat sessions and influence subsequent interactions. A background process identifies durable details from a conversation and saves them into a user-, project-, or organization-level store. When a new session begins, relevant memories are retrieved and explicitly inserted into the model's active context window. This context insertion step is critical, as the model requires the remembered information to be present in its prompt to inform responses.

Large language models do not intrinsically carry personal experiences or episodic memory from one interaction to the next. Instead, apparent LLM memory is produced by an architectural system combining model weights, context windows, conversation storage, long-term memory, and a memory manager. This setup relies on processes including context reconstruction, summarization, retrieval, and persistent storage. Ultimately, the quality and effectiveness of an LLM application's memory depend heavily on its surrounding software architecture rather than just the underlying model.