Gradient Clipping

Fact

A technique that caps huge gradients so training updates do not fly out of control.

In Plain Words

Gradient clipping is a speed limiter on a go-kart. The kid stomps the gas, but the snacks survive.

You meet it in RNN and LLM training. It tames exploding gradients, so training steps stay steady.

Related Concepts

• Backpropagation
• Gradient Descent
• Stochastic Gradient Descent
• Long Short-Term Memory

Backpropagation
Backpropagation finds the gradients, and clipping blocks the huge ones.

Gradient Descent
Gradient Descent updates parameters, and clipping stops steps from getting too big.

SGD
SGD can make noisy updates, and clipping helps keep training under control.

LSTM
LSTM models often use clipping to reduce exploding gradients.