A task where a model learns language patterns by predicting the next piece of text.
It is like a never-ending fill-in-the-blank quiz. The AI sees some words. Then it guesses the next one.
This builds the base skill for writing text. You see it in pretraining, chat, and Q&A.
Pretraining
Language modeling is one of the main learning tasks in pretraining.
Token
Language modeling usually predicts the next token from the text before it.
LLM
An LLM’s writing, Q&A, and chat skills are built on language modeling.
Transformer
Transformer is the most common setup for modern language modeling.