To fix CUDA out of memory in training, cut what each GPU must hold at once. Shorten sequences, lower the batch size, turn on gradient checkpointing, train a LoRA adapter, or split the model across more GPUs.
What fills the memory
A training step holds four things on the GPU: the weights, their gradients, the optimizer state, and the activations kept for the backward pass. Activations grow with batch size × sequence length. Gradients and optimizer state grow with the number of trained parameters.
Steps
Try them in this order, cheapest first.
- Shorten sequences. Lower the maximum length, or drop the few longest rows. Activation memory falls with it.
- Lower the batch size. Raise gradient accumulation by the same factor to keep the effective batch.
- Turn on gradient checkpointing. It recomputes activations in the backward pass instead of storing them, so steps get slower.
- Train an adapter. LoRA (low-rank adaptation) drops the gradients and optimizer state of the frozen weights. QLoRA also loads the base model in 4-bit.
- Shard the model. Split the weights or the sequence across more GPUs. See distributed training.
With Halo
Halo's troubleshooting page says activations dominate and gives its own order:
gradient_checkpointing: true(20 to 30% slower).- Lower
per_device_train_batch_sizeormax_length. - Shard: tensor parallelism for dense models, expert or expert-tensor parallelism for mixture-of-experts models, or LoRA or QLoRA.
Fill the global batch with gradient_accumulation_steps. In supervised fine-tuning a row longer than max_length is dropped, not truncated, so check how many rows a lower limit removes. Offline GRPO has one more switch for long completions: use_chunked_grpo_logprobs: true. See Troubleshooting.
