Batch size

Batch size is the number of training examples a model processes before each weight update. The effective batch size is the per-GPU batch times the number of GPUs times the gradient accumulation steps.

How it works

A training step runs a batch through the model, averages the loss over it and updates the weights once. A larger batch gives a steadier gradient. It also needs more memory, because every example's activations are held at the same time.

Three numbers multiply into the effective batch, the number of examples behind one update:

  • Per-device batch. Examples on one GPU in one pass. Memory limits this one.
  • GPUs. Under data parallelism each GPU takes its own slice of the batch.
  • Accumulation steps. Gradient accumulation adds up several passes before one update.

A per-device batch of 2 on 8 GPUs with 8 accumulation steps is an effective batch of 128.

How to set it

Let memory decide the per-device batch, then reach the effective batch you want with accumulation. A larger effective batch usually goes with a higher learning rate.

In Halo

The effective batch is per_device_train_batch_size × gradient_accumulation_steps × data_parallel_size, and full fine-tunes land at 64 to 128.

For speed, count tokens per GPU per step: M = per_device_train_batch_size × max_length should be at least about 8k on a B300 mixture-of-experts run. Below that the GPUs are waiting, not computing. Of the levers in the performance guide, a bigger M has the single largest effect.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL