How many GPUs do you need to fine-tune an LLM?

Fine-tuning an LLM takes one GPU for LoRA or QLoRA on a small model, and from one to many for full fine-tuning as models grow. The count depends on model size, method, sequence length and how fast you need the run.

What decides it

  • Model size. Weights, gradients and optimizer state all grow with the parameter count.
  • Method. LoRA keeps gradients and optimizer state only for a small adapter. QLoRA also loads the base in 4-bit. Full fine-tuning pays for every weight.
  • Sequence length and batch size. Activation memory grows with both.

When one GPU is not enough

Shard the model across GPUs: expert parallelism for mixture-of-experts (MoE) models, tensor parallelism for dense ones. Sharding buys capacity and costs tokens per second, so take the least that fits. If a run almost fits, try the fixes for CUDA out of memory first.

In Halo

Hardware for recipes in the quickstart:

RecipeHardware
Qwen3 4B, LoRA1 GPU
Qwen3 4B, QLoRA1 GPU (7.9 GiB peak at batch 1 and 4k tokens, fits 24 GB)
Qwen3 4B, full fine-tuning1 to 8 GPUs
GPT-OSS 20B, expert parallel2 nodes of 8 GPUs, as the recipe is written

The performance page adds that a 119B MoE trains on four GPUs at ep2 + etp2. -n sets the GPU count, as in -n 8. Without it a run uses one GPU.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL