LoRA trains a small adapter beside a frozen model, and full fine-tuning updates every weight. LoRA needs far less memory and storage. Full fine-tuning has more capacity for large changes in behavior.
Side by side
Which to pick
Start with LoRA (low-rank adaptation). It fits on less hardware, and on a narrow task it often matches full fine-tuning. It also changes the base less, which limits catastrophic forgetting.
Move to full fine-tuning when:
- quality stalls and raising the rank no longer helps;
- the model has to learn a new language or domain, not a new habit;
- you have the data and the GPUs for it.
Serving is the same either way, because an adapter can be merged into the base.
In Halo
Both are the same script. LoRA is a config switch:
YAML
use_peft: true
lora_r: 64
lora_alpha: 128
learning_rate: 1.0e-04The full fine-tuning example for the same 4B model uses learning_rate: 2.0e-05. LoRA is rejected under tensor parallelism. The supervised fine-tuning guide covers both.
