PagedAttention

Fact

An attention method that manages the KV cache in small memory pages.

In Plain Words

PagedAttention is like a parking lot with neat spaces. Each car takes only its spots, so no giant truck steals a whole row.

LLM servers use it when many chats run at once. It saves VRAM, so more chats move through faster.

Related Concepts

• Key-Value Cache
• Video RAM
• Continuous batching
• LLM inference engine

KV cache
PagedAttention splits the KV cache into small pages and gives them out as needed.

VRAM
It cuts VRAM waste, so one card can handle more requests.

Continuous batching
It works with Continuous batching to reduce waiting during busy traffic.

Inference engine
An Inference engine uses it to improve LLM serving throughput.