The theory of measuring, coding, and sending information.
Info Theory is like packing a lunchbox for a bumpy bus ride. You want it snug, safe, and still edible at noon.
It tells us how to shrink messages and send them without mangling them. In AI, it shapes entropy and training loss.
Cross-Entropy Loss
Cross-Entropy Loss brings entropy from Information Theory into model training.
KL Divergence
KL Divergence is a classic Information Theory tool for comparing distributions.
Token
Information Theory can measure how much information one Token carries.
SLT
Information Theory covers information, and SLT covers learning and generalization.