Fine-tuning an LLM takes one GPU for LoRA or QLoRA on a small model, and from one to many for full fine-tuning as models grow. The count depends on model size, method, sequence length and how fast you need the run.
What decides it
- Model size. Weights, gradients and optimizer state all grow with the parameter count.
- Method. LoRA keeps gradients and optimizer state only for a small adapter. QLoRA also loads the base in 4-bit. Full fine-tuning pays for every weight.
- Sequence length and batch size. Activation memory grows with both.
When one GPU is not enough
Shard the model across GPUs: expert parallelism for mixture-of-experts (MoE) models, tensor parallelism for dense ones. Sharding buys capacity and costs tokens per second, so take the least that fits. If a run almost fits, try the fixes for CUDA out of memory first.
In Halo
Hardware for recipes in the quickstart:
The performance page adds that a 119B MoE trains on four GPUs at ep2 + etp2. -n sets the GPU count, as in -n 8. Without it a run uses one GPU.
