Gradient accumulation adds up the gradients of several small batches before making one weight update. It gives the effect of a larger batch in the memory of a small one.
How it works
Normally every batch ends with a weight update. With accumulation set to N:
- Run a forward and a backward pass on a small batch. Keep the gradients.
- Do it
Ntimes, adding each new gradient to the sum. - Update the weights once with the average, then clear the sum.
The update is the one a batch N times larger would give. Peak memory stays that of one small batch.
When to use it
Use it when the batch size you want does not fit in memory. That is the usual case in fine-tuning, where a few long sequences fill a GPU. It also lets you keep the same effective batch on fewer GPUs.
Two limits:
- It is not faster. The same examples go through the model.
- It does not help a loss that compares examples within a batch. Contrastive embedding training takes its negatives from the small batch, and accumulation adds none.
A step in logs and schedules is one update, not one pass. See epochs and steps.
In Halo
YAML
per_device_train_batch_size: 2
gradient_accumulation_steps: 8The effective batch is per_device_train_batch_size × gradient_accumulation_steps × data_parallel_size. The performance guide says to fill the global batch with gradient_accumulation_steps, not with more parallelism.
