Why An LLM’s Memory Gets Expensive and How to Fix It

To generate each token, a model executes an attention step comparing the newest token against earlier tokens using key and value vectors. Recomputing these vectors for every preceding token at each step is computationally wasteful because their values remain static. The KV cache resolves this inefficiency by storing computed key and value vectors so that only the new token requires computation. However, reading this cache on every step introduces a new cost bottleneck and can cause unexpected out-of-memory errors because the cache retains high-dimensional vectors rather than text.

Token generation operates in two distinct phases with different hardware bottlenecks: prefill and decoding. During prefill, all input tokens are processed in parallel to construct the key-value cache, making this phase compute-bound and limited by arithmetic throughput. Decoding generates tokens autoregressively one by one, requiring a full read of the key-value cache from GPU memory for every step, which makes it memory-bound. As a result, the primary bottleneck in long-context generation stems from repeatedly streaming the entire cache across the memory bus rather than simply storing it.

The key-value cache size of a large language model is determined by multiplying the number of layers, key-value heads, head dimension, bytes per number, context length, batch size, and a factor of two. Cache requirements scale linearly with both token count and batch size, making long contexts and concurrent users resource-intensive. For instance, a single 128,000-token request on Llama 3 70B demands approximately 40 gigabytes of memory, consuming half of an 80-gigabyte GPU card. Optimization techniques aim to mitigate memory demands by reducing individual factors in this multiplication.

Attention architecture modifications can substantially reduce the key-value cache memory footprint per token, though these designs must be committed to during training. Grouped-query attention saves memory by having multiple query heads share key-value heads, offering a stable middle ground compared to aggressive multi-query attention. Multi-head latent attention further compresses cache demands by projecting key and value representations into a smaller latent space before caching, as demonstrated by DeepSeek-V3. While these architectural techniques reduce memory traffic at large scale, they require trade-offs in serving overhead, quality, or implementation complexity.

Quantization reduces the memory footprint of the key-value cache by converting numbers from standard 16-bit precision down to 8-bit or 4-bit representations. This technique requires no model retraining, cutting cache size in half at 8 bits and in half again at 4 bits. While 8-bit quantization typically produces negligible accuracy drops under one percent, 4-bit storage shows measurable degradation on complex tasks like multi-needle retrieval. Although specialized algorithms outperform plain rounding by preserving precision for key values, simple rounding still captures the majority of compression gains.

Eviction reduces token counts in language models by discarding entries deemed unlikely to be needed. A typical implementation maintains both the most recent context window and a few opening tokens, which function as attention anchors to stabilize model generation. However, eviction faces a structural flaw because the model cannot predict whether a discarded token will be essential for subsequent queries. As a result, aggressive eviction techniques often fail on retrieval tasks where key information is buried mid-sequence.

Serving systems historically suffered high memory fragmentation by reserving contiguous blocks sized for worst-case output lengths. Paged attention addresses this by splitting the cache into small fixed-size pages allocated on demand, reducing fragmentation from 60-80% to under 4% and boosting throughput by two to three times. Furthermore, storing cache in shareable blocks enables prefix caching, allowing requests with identical prefixes to point to the same physical memory. Providers such as OpenAI and Anthropic report 50 to 90 percent cost and latency reductions on cache hits, though cross-user cache sharing introduces timing side-channel security risks.

Different memory-saving techniques for model caches offer varied tradeoffs between memory reduction, implementation complexity, and output quality. Methods like grouped-query attention, paged attention, and prefix caching provide memory benefits with virtually no quality loss. In contrast, techniques such as low-bit quantization, latent attention, and eviction introduce notable quality risks or require significant engineering effort. Ultimately, these optimizations become vital in long-context and high-concurrency settings where cache overhead dominates, requiring strategies to be matched to specific workload patterns.

The cost of long-context inference is fundamentally determined by the size of the attention cache. Because decoding reads the entire cache for every generated token, the cache creates both storage and memory-bandwidth bottlenecks. Shrinking the cache improves generation speed, and optimization techniques target different aspects of cache size and management. Approaches include reducing per-token cost, quantizing values, evicting tokens, and improving memory layout and sharing.