Mixed precision (bf16)

Mixed precision runs most of training in 16-bit floating point instead of 32-bit, which halves the memory of those tensors and speeds up GPU math. For language models the usual 16-bit format is bf16 (bfloat16).

How it works

32-bit floating point (fp32) is the classic format for neural-network math. A 16-bit tensor takes half the memory, and modern GPUs multiply 16-bit matrices much faster, which raises training throughput. Mixed precision runs the forward and backward passes in 16-bit and keeps 32-bit only where precision matters.

It is not quantization, which stores weights in 8 bits or fewer.

Why bf16 and not fp16

Both formats are 16 bits wide and spend the bits differently:

FormatExponent bitsFraction bitsLargest value
fp1651065,504
bf1687about 3.4e38, as in fp32

fp16 overflows easily: a value above 65,504 becomes infinity, and a very small gradient rounds to zero, so fp16 training needs loss scaling. bf16 has the range of fp32 and needs none. It keeps fewer significant digits, which training tolerates. That makes bf16 the usual choice for language models on GPUs that support it.

In Halo

The parser sets bf16: true by default for every method. That also brings in the AdamWBF16 optimizer wherever optim is left at a stock AdamW. The exception is replicated DDP (PyTorch's distributed data parallel). An explicit fp16: true suppresses the default, and the two cannot both be set. See Writing a config.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL