A way to split text into smaller word pieces called tokens.
BPE is like making words with fridge magnets. Common chunks stay stuck, and weird new words still get built.
It keeps the AI’s word list small. Many LLMs use it before reading your prompt.
Token
BPE decides which tokens a piece of text becomes.
Embedding
BPE splits text first, then each piece can get an embedding.
LLM
Many LLMs use BPE as their basic text splitter.