QLoRA is LoRA applied to a base model whose frozen weights are loaded in 4-bit precision, which takes roughly a quarter of the memory of 16-bit. The adapter trains in higher precision on top.
How it works
QLoRA (quantized LoRA) was introduced in a 2023 paper. It stores the frozen base model in 4-bit, using two techniques:
- NF4 (4-bit NormalFloat). A 4-bit data type whose levels are spaced for weights that follow a normal distribution.
- Double quantization. The quantization constants are quantized too, which saves a little more. It is also called nested quantization.
In the forward pass the 4-bit weights are dequantized on the fly to a 16-bit compute type, usually bf16. Gradients flow through them into the LoRA matrices, which stay in 16-bit or 32-bit. Only the adapter is trained.
Trade-offs
- A model that does not fit one GPU for LoRA may fit for QLoRA.
- Steps are usually slower than plain LoRA, because of the dequantization.
- Quantization adds a small error to the base weights, and the adapter learns on top of it.
In Halo
QLoRA is the LoRA config plus three keys:
YAML
load_in_4bit: true
bnb_4bit_quant_type: nf4
use_bnb_nested_quant: trueThis peaks at 7.9 GiB for a 4B model at batch 1 and 4k tokens, so it fits a 24 GB GPU. Across GPUs, QLoRA runs under data parallelism (DDP or FSDP) and context parallelism only. See LoRA and QLoRA in the Halo docs.
