A tool that cuts text into tokens an AI model can read.
A tokenizer is like a lunch lady with a pizza wheel. It cuts a giant text pizza into bites the AI can chew.
You meet it when prompts and long docs get counted. Its cuts affect how much fits, and how much you pay.
Token
A Token is the basic piece a tokenizer cuts out for the model to read and write.
BPE
BPE is a common way a tokenizer cuts words into smaller pieces.
LLM
An LLM needs the tokenizer to encode text before it can read it.
Context-window
Tokenizing changes the token count, so it changes how much text fits.