How to fix CUDA out of memory

To fix CUDA out of memory in training, cut what each GPU must hold at once. Shorten sequences, lower the batch size, turn on gradient checkpointing, train a LoRA adapter, or split the model across more GPUs.

What fills the memory

A training step holds four things on the GPU: the weights, their gradients, the optimizer state, and the activations kept for the backward pass. Activations grow with batch size × sequence length. Gradients and optimizer state grow with the number of trained parameters.

Steps

Try them in this order, cheapest first.

  1. Shorten sequences. Lower the maximum length, or drop the few longest rows. Activation memory falls with it.
  2. Lower the batch size. Raise gradient accumulation by the same factor to keep the effective batch.
  3. Turn on gradient checkpointing. It recomputes activations in the backward pass instead of storing them, so steps get slower.
  4. Train an adapter. LoRA (low-rank adaptation) drops the gradients and optimizer state of the frozen weights. QLoRA also loads the base model in 4-bit.
  5. Shard the model. Split the weights or the sequence across more GPUs. See distributed training.

With Halo

Halo's troubleshooting page says activations dominate and gives its own order:

  1. gradient_checkpointing: true (20 to 30% slower).
  2. Lower per_device_train_batch_size or max_length.
  3. Shard: tensor parallelism for dense models, expert or expert-tensor parallelism for mixture-of-experts models, or LoRA or QLoRA.

Fill the global batch with gradient_accumulation_steps. In supervised fine-tuning a row longer than max_length is dropped, not truncated, so check how many rows a lower limit removes. Offline GRPO has one more switch for long completions: use_chunked_grpo_logprobs: true. See Troubleshooting.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL