Quantization

Quantization stores a model's weights in fewer bits, for example 8 or 4 instead of 16, so the model takes less memory. It trades a small loss of accuracy for a model that fits on smaller hardware.

How it works

Weights are trained as 16-bit or 32-bit floating-point numbers. Quantization maps them onto a much smaller set of values: each weight becomes a short code, and a scale shared by a block of weights turns the codes back into numbers. A 4-bit model takes about a quarter of the memory of a 16-bit one.

Fewer bits mean more rounding error: usually small at 8 bits, larger at 4 and worth measuring on your own task.

Where it is used

  • During training. QLoRA loads the frozen base model in 4-bit and trains a LoRA adapter on top. The quantized weights are never updated, which is how a large model fits on a small GPU.
  • After training. A finished model is quantized for serving, so that it fits a smaller GPU.

Quantization is not mixed precision, which computes in 16-bit floats and goes no lower.

In Halo

Halo ships a tool that writes a block-scaled compressed-tensors checkpoint in mxfp8, mxfp4 or nvfp4:

Bash

halo run quantize-to-lowp --input_dir <bf16-model-dir> --output_dir <nvfp4-out-dir> --format nvfp4

On a mixture-of-experts checkpoint you are about to serve, it halves (fp8) or quarters (fp4) the bytes of the expert weights. The docs mark this export experimental. For training, the QLoRA example adds load_in_4bit: true. See Checkpoints.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL