Information Bottleneck

Fact

A rule for keeping task clues while squeezing data into a smaller form.

In Plain Words

Information Bottleneck is packing for a weekend with one tiny backpack. Socks stay. Your rock collection stays home.

You meet it in representation learning and compression. It keeps task clues and drops noise.

Related Concepts

• Information Theory
• Representation Learning
• Dimensionality Reduction
• Kullback-Leibler Divergence

Information Theory
Information Bottleneck uses information amount to choose what to keep and what to drop.

Representation Learning
It guides models to learn smaller representations with more task clues.

Dim. Reduction
Both compress data, but Information Bottleneck cares more about task relevance.

KL Divergence
KL often measures the information gap before and after compression.