AdaGrad — Adaptive Gradient Algorithm

Fact

An optimizer sets each parameter's learning rate from past gradients.

In Plain Words

AdaGrad is like a gym coach with a tiny clipboard. Busy muscles get lighter weights. Lazy muscles get the spotlight.

It helps with rare clues, like rare words. But its steps keep shrinking. Learning can crawl near the end.

Related Concepts

• Gradient Descent
• Stochastic Gradient Descent
• Root Mean Square Propagation
• Adaptive Moment Estimation optimizer

Gradient Descent
AdaGrad follows the same gradient direction, but each parameter uses its own step size.

SGD
SGD often uses one shared step size, but AdaGrad changes it per parameter.

RMSProp
RMSProp uses a moving average to slow AdaGrad's shrinking step sizes.

Adam
Adam keeps AdaGrad's adaptive step sizes and adds momentum estimates.