LoRA (low-rank adaptation) fine-tunes a model by training small low-rank matrices added to its frozen weights. It trains a few percent of the parameters or less and saves a small adapter file.
How it works
A weight matrix W stays frozen. LoRA adds two small matrices, A and B, and trains only those. The layer computes W + (alpha / r) · B · A, where r is the rank: the inner size of the pair.
With r = 64 on a 4,096-wide layer, the pair has about 3% of the layer's parameters. Across a whole model the trained share is usually a few percent or less.
Why use it
- Memory. There are no gradients or optimizer state for the frozen weights, so a model that does not fit for full fine-tuning often fits for LoRA.
- Storage. The result is an adapter of tens or hundreds of megabytes, not a copy of the model.
- Swapping. One base model can serve many adapters.
The cost is capacity. For a large change in behavior, or a new language, full fine-tuning usually wins. See LoRA vs full fine-tuning.
In Halo
LoRA is a config switch on a training run, not a separate script:
YAML
use_peft: true
lora_r: 64
lora_alpha: 128
lora_target_modules: [q_proj, k_proj, v_proj, o_proj]
learning_rate: 1.0e-04LoRA wants 5 to 10 times the learning rate of full fine-tuning. On mixture-of-experts models, name the expert projections in lora_target_modules: all-linear misses the fused expert tensors. The SFT guide has the details.
