# What is LoRA?

LoRA (low-rank adaptation) fine-tunes a model by training small low-rank matrices added to its frozen weights. It trains a few percent of the parameters or less and saves a small adapter file.

## How it works

A weight matrix `W` stays frozen. LoRA adds two small matrices, `A` and `B`, and trains only those. The layer computes `W + (alpha / r) · B · A`, where `r` is the rank: the inner size of the pair.

With `r = 64` on a 4,096-wide layer, the pair has about 3% of the layer's parameters. Across a whole model the trained share is usually a few percent or less.

## Why use it

- **Memory.** There are no gradients or optimizer state for the frozen weights, so a model that does not fit for full fine-tuning often fits for LoRA.
- **Storage.** The result is an adapter of tens or hundreds of megabytes, not a copy of the model.
- **Swapping.** One base model can serve many adapters.

The cost is capacity. For a large change in behavior, or a new language, full fine-tuning usually wins. See [LoRA vs full fine-tuning](/glossary/lora-vs-full-fine-tuning).

## In Halo

LoRA is a config switch on a training run, not a separate script:

```yaml
use_peft: true
lora_r: 64
lora_alpha: 128
lora_target_modules: [q_proj, k_proj, v_proj, o_proj]
learning_rate: 1.0e-04
```

LoRA wants 5 to 10 times the learning rate of full fine-tuning. On mixture-of-experts models, name the expert projections in `lora_target_modules`: `all-linear` misses the fused expert tensors. The [SFT guide](/halo/docs/training-methods/sft#lora-and-qlora) has the details.

