Gradient Descent

Fact

A method for updating parameters toward lower loss.

In Plain Words

Gradient descent is like finding the lowest spot on a mini-golf green. You tap the ball downhill, not into next Tuesday.

It trains models by lowering error. Big steps can overshoot. Tiny steps crawl.

Related Concepts

• Backpropagation
• Stochastic Gradient Descent
• Adaptive Moment Estimation optimizer
• Parameter

Backpropagation
Backprop calculates the gradient, and gradient descent updates the parameters with it.

SGD
SGD is a random mini-batch version of gradient descent, and it is very common.

Adam
Adam improves gradient descent by changing the step size and update style.

Parameter
Gradient descent moves parameters in a better direction.