Large networks may hide small sparse subnetworks able to train well alone.
A huge AI is like a giant roll of lottery tickets. One ticket inside was born a winner. Tear away the rest, and it still wins.
The idea says a big network can hide a small trainable network. It helps explain pruning, compression, and training research.
Network Pruning
Network Pruning is a common way to find the winning subnetwork.
Overparameterization
Overparameterization gives the big network room to hide a strong small subnetwork.
Weight Initialization
Lottery Ticket Hypothesis says the starting weights can decide if training works.
Model Compression
It inspires Model Compression by keeping a small sparse subnetwork.