Self-Attention

Fact

A way for each item in a row to pay attention to the others.

In Plain Words

Self-attention is a sentence full of nosy neighbors. Each word peeks at every other word and remembers the important ones.

This helps AI catch context between words. Transformers use it to understand sentences.

Related Concepts

• Attention Mechanism
• Transformer architecture
• Key-Value Cache
• Flash Attention

Attention
Self-attention is a type of Attention for linking one row to itself.

Transformer
Transformers use self-attention as the main tool for context.

KV cache
KV cache saves self-attention keys and values during generation.

Flash Attention
Flash Attention speeds up self-attention without changing its answer.