Cross-Entropy Loss

Fact

A loss function that scores how far predicted probabilities are from the truth.

In Plain Words

Cross-entropy is a picky quiz teacher. Pick pizza for lunch, but mumble it, and you still lose points.

It scores how far the model’s probabilities are from the right answer. You meet it in classifiers and language models.

Related Concepts

• Softmax normalization function
• Gradient Descent
• Backpropagation
• Language Modeling

Softmax
Cross-Entropy often follows Softmax and scores its probabilities.

Gradient Descent
Cross-Entropy gives Gradient Descent a target to improve.

Backpropagation
Cross-Entropy creates an error signal, and Backprop sends it back.

LM
LMs often use Cross-Entropy to learn the next word.