What is QLoRA?

QLoRA is LoRA applied to a base model whose frozen weights are loaded in 4-bit precision, which takes roughly a quarter of the memory of 16-bit. The adapter trains in higher precision on top.

How it works

QLoRA (quantized LoRA) was introduced in a 2023 paper. It stores the frozen base model in 4-bit, using two techniques:

  • NF4 (4-bit NormalFloat). A 4-bit data type whose levels are spaced for weights that follow a normal distribution.
  • Double quantization. The quantization constants are quantized too, which saves a little more. It is also called nested quantization.

In the forward pass the 4-bit weights are dequantized on the fly to a 16-bit compute type, usually bf16. Gradients flow through them into the LoRA matrices, which stay in 16-bit or 32-bit. Only the adapter is trained.

Trade-offs

  • A model that does not fit one GPU for LoRA may fit for QLoRA.
  • Steps are usually slower than plain LoRA, because of the dequantization.
  • Quantization adds a small error to the base weights, and the adapter learns on top of it.

In Halo

QLoRA is the LoRA config plus three keys:

YAML

load_in_4bit: true
bnb_4bit_quant_type: nf4
use_bnb_nested_quant: true

This peaks at 7.9 GiB for a 4B model at batch 1 and 4k tokens, so it fits a 24 GB GPU. Across GPUs, QLoRA runs under data parallelism (DDP or FSDP) and context parallelism only. See LoRA and QLoRA in the Halo docs.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL