A model that predicts the next token from the text before it.
It is like phone autocomplete. Type “Dear Principal,” and it probably will not suggest “pizza dragon.”
It builds text one token at a time. You meet it in chatbots. You meet it in writing tools. You meet it in code autocomplete.
Token
It predicts Tokens in order to build the full text.
LM
LM often teaches it to continue with the next token.
Transformer
Modern autoregressive models often use a Transformer.
MLM
It writes forward, while MLM fills in blanks.