# Gradient accumulation

Gradient accumulation adds up the gradients of several small batches before making one weight update. It gives the effect of a larger batch in the memory of a small one.

## How it works

Normally every batch ends with a weight update. With accumulation set to `N`:

1. Run a forward and a backward pass on a small batch. Keep the gradients.
2. Do it `N` times, adding each new gradient to the sum.
3. Update the weights once with the average, then clear the sum.

The update is the one a batch `N` times larger would give. Peak memory stays that of one small batch.

## When to use it

Use it when the [batch size](/glossary/batch-size) you want does not fit in memory. That is the usual case in fine-tuning, where a few long sequences fill a GPU. It also lets you keep the same effective batch on fewer GPUs.

Two limits:

- It is not faster. The same examples go through the model.
- It does not help a loss that compares examples within a batch. Contrastive [embedding training](/glossary/embedding-fine-tuning) takes its negatives from the small batch, and accumulation adds none.

A step in logs and schedules is one update, not one pass. See [epochs and steps](/glossary/epochs-and-steps).

## In Halo

```yaml
per_device_train_batch_size: 2
gradient_accumulation_steps: 8
```

The effective batch is `per_device_train_batch_size × gradient_accumulation_steps × data_parallel_size`. The [performance guide](/halo/docs/performance#levers-that-help) says to fill the global batch with `gradient_accumulation_steps`, not with more parallelism.

