# Quantization

Quantization stores a model's weights in fewer bits, for example 8 or 4 instead of 16, so the model takes less memory. It trades a small loss of accuracy for a model that fits on smaller hardware.

## How it works

Weights are trained as 16-bit or 32-bit floating-point numbers. Quantization maps them onto a much smaller set of values: each weight becomes a short code, and a scale shared by a block of weights turns the codes back into numbers. A 4-bit model takes about a quarter of the memory of a 16-bit one.

Fewer bits mean more rounding error: usually small at 8 bits, larger at 4 and worth measuring on your own task.

## Where it is used

- **During training.** [QLoRA](/glossary/qlora) loads the frozen base model in 4-bit and trains a [LoRA](/glossary/lora) adapter on top. The quantized weights are never updated, which is how a large model fits on a small GPU.
- **After training.** A finished model is quantized for serving, so that it fits a smaller GPU.

Quantization is not [mixed precision](/glossary/mixed-precision), which computes in 16-bit floats and goes no lower.

## In Halo

Halo ships a tool that writes a block-scaled `compressed-tensors` checkpoint in `mxfp8`, `mxfp4` or `nvfp4`:

```bash
halo run quantize-to-lowp --input_dir <bf16-model-dir> --output_dir <nvfp4-out-dir> --format nvfp4
```

On a [mixture-of-experts](/glossary/mixture-of-experts) checkpoint you are about to serve, it halves (fp8) or quarters (fp4) the bytes of the expert weights. The docs mark this export experimental. For training, the QLoRA example adds `load_in_4bit: true`. See [Checkpoints](/halo/docs/checkpoints#post-processing-tools).

