Gradient clipping

Gradient clipping caps the size of the gradient before each weight update. If the gradient's norm is above a threshold, commonly 1.0, it is scaled down, so one bad batch cannot throw the weights off.

How it works

After the backward pass, the trainer measures the norm of the whole gradient: the length of all the gradients taken as one vector. If the norm is above the threshold c, every gradient is multiplied by c / norm. The direction stays and only the length shrinks. Below the threshold nothing changes.

Why it matters

Now and then a batch produces a very large gradient, from an unusual example or a long sequence. Without a cap, that one step can undo many good ones, and the loss spikes or turns to NaN (not a number).

How to set it

1.0 is the usual value, and the default of max_grad_norm in Hugging Face trainers. Log the gradient norm before changing it.

Clipping is a safety net for rare spikes. Spikes that keep coming point to a cause worth fixing: a learning rate that is too high, too short a warmup, or bad rows in the data.

In Halo

Halo configs are Hugging Face TrainingArguments fields plus Halo's own, so the key is the same. The supervised fine-tuning example sets max_grad_norm: 1.0. Loss, learning rate and gradient norm are logged at every logging_steps. See Monitoring.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL