# Batch size

Batch size is the number of training examples a model processes before each weight update. The effective batch size is the per-GPU batch times the number of GPUs times the gradient accumulation steps.

## How it works

A training step runs a batch through the model, averages the loss over it and updates the weights once. A larger batch gives a steadier gradient. It also needs more memory, because every example's activations are held at the same time.

Three numbers multiply into the effective batch, the number of examples behind one update:

- **Per-device batch.** Examples on one GPU in one pass. Memory limits this one.
- **GPUs.** Under [data parallelism](/glossary/data-parallelism) each GPU takes its own slice of the batch.
- **Accumulation steps.** [Gradient accumulation](/glossary/gradient-accumulation) adds up several passes before one update.

A per-device batch of 2 on 8 GPUs with 8 accumulation steps is an effective batch of 128.

## How to set it

Let memory decide the per-device batch, then reach the effective batch you want with accumulation. A larger effective batch usually goes with a higher [learning rate](/glossary/learning-rate).

## In Halo

The effective batch is `per_device_train_batch_size × gradient_accumulation_steps × data_parallel_size`, and full fine-tunes land at 64 to 128.

For speed, count tokens per GPU per step: `M = per_device_train_batch_size × max_length` should be at least about 8k on a B300 [mixture-of-experts](/glossary/mixture-of-experts) run. Below that the GPUs are waiting, not computing. Of the levers in the [performance guide](/halo/docs/performance#levers-that-help), a bigger `M` has the single largest effect.

