A method for searching training settings to make a model work better.
HPO is like making pancakes on a fussy stove. One heat notch makes fluffy pancakes, or a rubber frisbee.
It tries many training settings and keeps the best mix. You see it in training and fine-tuning. It costs time and compute.
Parameter
HPO tunes training settings, not weights the model learns.
Fine-tuning
Fine-tuning often uses HPO to find better training settings.
SGD
HPO often searches SGD settings like learning rate and momentum.
Optimization
HPO is an optimization problem about hyperparameter mixes.