An optimizer trick that uses past gradients to speed up model updates.
Momentum is like pushing a grocery cart. Once it rolls straight, tiny bumps in the cereal aisle do not boss it around.
In neural network training, it helps updates move faster and wobble less. You often see it with SGD, and inside Adam.
SGD
Momentum often upgrades SGD, so it updates faster and more smoothly.
Gradient Descent
Momentum improves Gradient Descent by reducing zigzag movement.
Adam
Adam builds on momentum and also adjusts step sizes.