An optimizer that updates model parameters using small batches of data.
SGD is cooking soup for a school cafeteria. You taste one spoon, tweak the salt, and hope lunch survives.
It trains models by updating parameters with small data batches. It is a basic tool for deep learning, pretraining, and fine-tuning.
Parameter
SGD keeps adjusting parameters so the model gets closer to the target.
Pretraining
Pretraining uses SGD-like methods to keep updating model weights.
Adam
Adam grew from SGD and learned to adjust step sizes.
Deep Learning
Deep Learning needs optimizers like SGD to make models learn.