The process of nudging model settings toward better results during training.
Optimization is like a fussy hotel shower knob. You keep tiny-turning it until the water stops attacking you.
In model training, it keeps nudging the model’s settings. Each nudge aims for better answers.
Gradient Descent
Gradient Descent is the classic way to do optimization.
SGD
SGD updates settings with small batches of data.
Adam
Adam adjusts step size, so optimization often moves faster.
Backpropagation
Backprop gives the direction for each optimization step.