LoRA vs full fine-tuning

LoRA trains a small adapter beside a frozen model, and full fine-tuning updates every weight. LoRA needs far less memory and storage. Full fine-tuning has more capacity for large changes in behavior.

Side by side

LoRAFull fine-tuning
What is trainedlow-rank matrices beside frozen weights, a few percent of the parameters or lessevery weight
Training memoryno gradients or optimizer state for the frozen weightsgradients and optimizer state for every weight
Outputan adapter of tens or hundreds of megabytesa full copy of the model
Learning ratehigher, 5 to 10 times the full fine-tuning ratelower
Wins whenmemory is tight, data is small, or one base serves many tasksthe change is large and data is plentiful

Which to pick

Start with LoRA (low-rank adaptation). It fits on less hardware, and on a narrow task it often matches full fine-tuning. It also changes the base less, which limits catastrophic forgetting.

Move to full fine-tuning when:

  • quality stalls and raising the rank no longer helps;
  • the model has to learn a new language or domain, not a new habit;
  • you have the data and the GPUs for it.

Serving is the same either way, because an adapter can be merged into the base.

In Halo

Both are the same script. LoRA is a config switch:

YAML

use_peft: true
lora_r: 64
lora_alpha: 128
learning_rate: 1.0e-04

The full fine-tuning example for the same 4B model uses learning_rate: 2.0e-05. LoRA is rejected under tensor parallelism. The supervised fine-tuning guide covers both.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL