A cache for Attention key-value data from already generated tokens.
KV cache is like a pizza shop leaving your order slip on the counter. When you add garlic bread, they do not ask for your whole life story again.
It speeds up long text generation. You meet it in chatbots and local LLMs.
Attention
KV cache stores past keys and values for Attention, so they are not recomputed.
Inference
KV cache speeds up Inference during token-by-token generation.
VRAM
KV cache uses VRAM, and long context can use a lot.
Continuous batching
Online inference often uses KV cache with Continuous batching.