KV Cache
The KV cache stores previously computed key-value pairs so the model doesn't recompute them — trading GPU memory for generation speed.
Two Phases of Generation
The KV cache is built during prefill and consumed during decode
Prefill
The model processes the entire input prompt at once. All key-value pairs are computed and stored in the KV cache.
Decode
The model generates one token at a time, reading cached keys/values and appending new ones. Each step depends on the previous.
What the KV Cache Stores
Each token gets a Key and Value vector stored for every layer
| Token | Key (K) — Layer 0 | Value (V) — Layer 0 | Key (K) — Layer 1 | Value (V) — Layer 1 |
|---|---|---|---|---|
| "The" | K₀,₀ | V₀,₀ | K₀,₁ | V₀,₁ |
| " cat" | K₁,₀ | V₁,₀ | K₁,₁ | V₁,₁ |
| " sat" | K₂,₀ | V₂,₀ | K₂,₁ | V₂,₁ |
| " on" | K₃,₀ | V₃,₀ | K₃,₁ | V₃,₁ |
For a 7B model with 32 layers: each token stores 32 × 2 key-value pairs
KV Cache Memory Formula
KV Cache Growth with Sequence Length
7B model, 32 layers, hidden_size=4096, FP16
KV Cache Optimizations
Techniques to reduce memory usage and improve throughput
FlashAttention
Reduces memory access by computing attention in tiles, keeping KV cache in fast on-chip memory instead of global VRAM.
2× FasterPagedAttention (vLLM)
Manages KV cache like virtual memory — non-contiguous blocks eliminate fragmentation, enabling 2-4× higher batch sizes.
2-4× BatchKV Cache Quantization
Store KV cache in INT8 or INT4 instead of FP16. Reduces cache memory by 2-4× with minimal quality impact.
2-4× SmallerSliding Window
Only keep the most recent K/V pairs, discarding older ones. Enables long context without linear memory growth.
O(1) Memory