# How many GPUs do you need to fine-tune an LLM?

Fine-tuning an LLM takes one GPU for LoRA or QLoRA on a small model, and from one to many for full fine-tuning as models grow. The count depends on model size, method, sequence length and how fast you need the run.

## What decides it

- **Model size.** Weights, gradients and optimizer state all grow with the parameter count.
- **Method.** [LoRA](/glossary/lora) keeps gradients and optimizer state only for a small adapter. [QLoRA](/glossary/qlora) also loads the base in 4-bit. Full fine-tuning pays for every weight.
- **Sequence length and batch size.** Activation memory grows with both.

## When one GPU is not enough

Shard the model across GPUs: [expert parallelism](/glossary/expert-parallelism) for mixture-of-experts (MoE) models, [tensor parallelism](/glossary/tensor-parallelism) for dense ones. Sharding buys capacity and costs tokens per second, so take the least that fits. If a run almost fits, try the fixes for [CUDA out of memory](/glossary/cuda-out-of-memory) first.

## In Halo

Hardware for recipes in the [quickstart](/halo/docs/quickstart#1-pick-a-recipe):

| Recipe | Hardware |
| --- | --- |
| Qwen3 4B, LoRA | 1 GPU |
| Qwen3 4B, QLoRA | 1 GPU (7.9 GiB peak at batch 1 and 4k tokens, fits 24 GB) |
| Qwen3 4B, full fine-tuning | 1 to 8 GPUs |
| GPT-OSS 20B, expert parallel | 2 nodes of 8 GPUs, as the recipe is written |

The [performance](/halo/docs/performance) page adds that a 119B MoE trains on four GPUs at `ep2 + etp2`. `-n` sets the GPU count, as in `-n 8`. Without it a run uses one GPU.

