# Flash Attention

Flash Attention is a family of GPU kernels that compute exact attention without storing the full attention matrix. Memory grows linearly with sequence length instead of with its square, and steps run faster.

## How it works

Standard attention in a [transformer](/glossary/transformer) scores every pair of tokens, which makes a matrix that grows with the square of the sequence length. Flash Attention computes the same result in small blocks that fit in the GPU's fast on-chip memory, and never writes the full matrix out. The backward pass recomputes the blocks it needs.

The output is exact, not an approximation. The method comes from a 2022 [paper](https://arxiv.org/abs/2205.14135).

## Why it matters

At long sequences the full attention matrix dominates memory. With a flash kernel a longer [context length](/glossary/context-length) fits on the same GPU, and steps run faster because less data moves in and out of GPU memory.

Later versions are written for newer GPU generations, so the right one depends on the hardware. The usual fallback is PyTorch's built-in scaled dot-product attention (SDPA).

## In Halo

Halo picks the backend when `attn_implementation` is unset. Supervised fine-tuning gets Flash Attention 4 on [Blackwell](/glossary/blackwell-vs-hopper) GPUs and Flash Attention 3 on Hopper, with Flash Attention 2 as the fallback. Gemma 4 has no flash path and runs on SDPA, and some families need `attn_implementation: sdpa` set by hand. Methods that train on padded batches, such as preference tuning and classification, default to SDPA. The [supported matrix](/halo/docs/supported-matrix#attention-backends) gives the rules per family.

