Full fine-tuning updates every weight of a pretrained model on your data. It has the most capacity to change a model's behavior and the highest memory cost, and it saves a complete copy of the model.
What it costs
Training a weight takes more memory than loading it. For every trained parameter the GPU holds:
- the weight itself;
- its gradient;
- optimizer state, which for Adam-style optimizers is two more values.
That is about four values per parameter, before the activations, which grow with batch size and sequence length. A model that fits one GPU for inference can need several to train in full.
When it beats LoRA
LoRA (low-rank adaptation) trains a small adapter and skips the gradients and optimizer state for the frozen weights. Full fine-tuning is worth its cost when:
- the change is large, such as a new language or a new domain;
- you have enough data that a low-rank update becomes the limit.
It also moves the model further from where it started, so catastrophic forgetting is a bigger risk. Use a lower learning rate than for LoRA. LoRA vs full fine-tuning sets the two side by side.
In Halo
Full fine-tuning is a run without use_peft: true. The Qwen3 4B recipe runs on 1 to 8 GPUs:
Bash
halo launch sft examples/sft/qwen3/qwen3-4b-ultrachat.yaml -n 8With -n 8, FSDP2 (fully sharded data parallel) shards the weights across the GPUs. The quickstart lists the recipes.
