# Mixed precision (bf16)

Mixed precision runs most of training in 16-bit floating point instead of 32-bit, which halves the memory of those tensors and speeds up GPU math. For language models the usual 16-bit format is bf16 (bfloat16).

## How it works

32-bit floating point (fp32) is the classic format for neural-network math. A 16-bit tensor takes half the memory, and modern GPUs multiply 16-bit matrices much faster, which raises [training throughput](/glossary/training-throughput). Mixed precision runs the forward and backward passes in 16-bit and keeps 32-bit only where precision matters.

It is not [quantization](/glossary/quantization), which stores weights in 8 bits or fewer.

## Why bf16 and not fp16

Both formats are 16 bits wide and spend the bits differently:

| Format | Exponent bits | Fraction bits | Largest value |
| --- | --- | --- | --- |
| fp16 | 5 | 10 | 65,504 |
| bf16 | 8 | 7 | about 3.4e38, as in fp32 |

fp16 overflows easily: a value above 65,504 becomes infinity, and a very small gradient rounds to zero, so fp16 training needs loss scaling. bf16 has the range of fp32 and needs none. It keeps fewer significant digits, which training tolerates. That makes bf16 the usual choice for language models on GPUs that support it.

## In Halo

The parser sets `bf16: true` by default for every method. That also brings in the `AdamWBF16` optimizer wherever `optim` is left at a stock AdamW. The exception is replicated DDP (PyTorch's distributed data parallel). An explicit `fp16: true` suppresses the default, and the two cannot both be set. See [Writing a config](/halo/docs/configuration#what-the-parser-does-before-your-config-reaches-the-trainer).

