LoRA rank and alpha

LoRA rank and alpha are the two numbers that size a LoRA adapter. Rank (r) sets how many parameters the adapter trains. Alpha scales its output, and the update is multiplied by alpha divided by r.

What they do

LoRA adds two small matrices beside a frozen weight W: A with r rows and B with r columns. The layer computes W + (alpha / r) · B · A.

  • Rank (r) is the adapter's capacity. The parameter count grows in step with it: twice the rank is twice the adapter.
  • Alpha scales the update. What counts is the ratio alpha / r. A larger ratio makes the update weigh more, much like a higher learning rate for the adapter.

How to pick

Typical ranks run from 8 to 64, with alpha equal to the rank or twice it.

  1. Start at rank 16 and alpha 32.
  2. If the model does not learn the task, raise the rank.
  3. If it overfits, lower the rank or train for fewer steps.

Keep alpha / r fixed when you change the rank. Otherwise capacity and step size change together. On a narrow task a small rank often matches a large one.

In Halo

The Qwen3 4B LoRA example sets:

YAML

lora_r: 64
lora_alpha: 128

Several model cookbooks use lora_r: 16 with lora_alpha: 32. Both keep alpha at twice the rank. See LoRA and QLoRA in the Halo docs.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL