# Gradient clipping

Gradient clipping caps the size of the gradient before each weight update. If the gradient's norm is above a threshold, commonly 1.0, it is scaled down, so one bad batch cannot throw the weights off.

## How it works

After the backward pass, the trainer measures the norm of the whole gradient: the length of all the gradients taken as one vector. If the norm is above the threshold `c`, every gradient is multiplied by `c / norm`. The direction stays and only the length shrinks. Below the threshold nothing changes.

## Why it matters

Now and then a batch produces a very large gradient, from an unusual example or a long sequence. Without a cap, that one step can undo many good ones, and the loss spikes or turns to NaN (not a number).

## How to set it

`1.0` is the usual value, and the default of `max_grad_norm` in Hugging Face trainers. Log the gradient norm before changing it.

Clipping is a safety net for rare spikes. Spikes that keep coming point to a cause worth fixing: a [learning rate](/glossary/learning-rate) that is too high, too short a [warmup](/glossary/learning-rate-warmup), or bad rows in the data.

## In Halo

Halo configs are Hugging Face `TrainingArguments` fields plus Halo's own, so the key is the same. The supervised fine-tuning example sets `max_grad_norm: 1.0`. Loss, learning rate and gradient norm are logged at every `logging_steps`. See [Monitoring](/halo/docs/monitoring#what-gets-logged).

