Flash Attention is a family of GPU kernels that compute exact attention without storing the full attention matrix. Memory grows linearly with sequence length instead of with its square, and steps run faster.
How it works
Standard attention in a transformer scores every pair of tokens, which makes a matrix that grows with the square of the sequence length. Flash Attention computes the same result in small blocks that fit in the GPU's fast on-chip memory, and never writes the full matrix out. The backward pass recomputes the blocks it needs.
The output is exact, not an approximation. The method comes from a 2022 paper.
Why it matters
At long sequences the full attention matrix dominates memory. With a flash kernel a longer context length fits on the same GPU, and steps run faster because less data moves in and out of GPU memory.
Later versions are written for newer GPU generations, so the right one depends on the hardware. The usual fallback is PyTorch's built-in scaled dot-product attention (SDPA).
In Halo
Halo picks the backend when attn_implementation is unset. Supervised fine-tuning gets Flash Attention 4 on Blackwell GPUs and Flash Attention 3 on Hopper, with Flash Attention 2 as the fallback. Gemma 4 has no flash path and runs on SDPA, and some families need attn_implementation: sdpa set by hand. Methods that train on padded batches, such as preference tuning and classification, default to SDPA. The supported matrix gives the rules per family.
