What is LoRA?

LoRA (low-rank adaptation) fine-tunes a model by training small low-rank matrices added to its frozen weights. It trains a few percent of the parameters or less and saves a small adapter file.

How it works

A weight matrix W stays frozen. LoRA adds two small matrices, A and B, and trains only those. The layer computes W + (alpha / r) · B · A, where r is the rank: the inner size of the pair.

With r = 64 on a 4,096-wide layer, the pair has about 3% of the layer's parameters. Across a whole model the trained share is usually a few percent or less.

Why use it

  • Memory. There are no gradients or optimizer state for the frozen weights, so a model that does not fit for full fine-tuning often fits for LoRA.
  • Storage. The result is an adapter of tens or hundreds of megabytes, not a copy of the model.
  • Swapping. One base model can serve many adapters.

The cost is capacity. For a large change in behavior, or a new language, full fine-tuning usually wins. See LoRA vs full fine-tuning.

In Halo

LoRA is a config switch on a training run, not a separate script:

YAML

use_peft: true
lora_r: 64
lora_alpha: 128
lora_target_modules: [q_proj, k_proj, v_proj, o_proj]
learning_rate: 1.0e-04

LoRA wants 5 to 10 times the learning rate of full fine-tuning. On mixture-of-experts models, name the expert projections in lora_target_modules: all-linear misses the fused expert tensors. The SFT guide has the details.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL