Mixing attention types per head: full attention for precise-recall heads, cheap linear attention for the rest.
Specialized attention hybridization is like staffing a workshop. Fine parts go to the master. Bulk parts go to the fast line.
Each attention head gets the job it is best at. The model stays sharp and runs cheaper on long text.
Attention
It mixes full and linear attention inside one layer, head by head.
Self-Attention
Self-Attention heads in one layer do different jobs, so it matches tasks to strengths.
Transformer
It makes Transformers sharp and cheap on long text.
Flash Attention
Flash Attention computes attention faster; this design computes less of it.