Flash Attention & Attention Backends

load_distributed_model (halo/model_loading/parallel/loader.py) auto-detects the backend per GPU and model: flash_attention_4 on Blackwell, flash_attention_3 on Hopper, flash_attention_2 otherwise. Pass attn_implementation only to override it. The detection line names only that GPU default, which the model-level checks below may move; the run's start log names the backend the policy was built with (Attention:).

FA2 is the detector's terminal default, with no availability probe — every shipped image carries it, and a host without it raises at model build. A route from FA4 to FA2 for a model FA4 cannot train (below) does probe, and refuses on such a host, naming flex_attention and eager. Context Parallelism hard-requires flash-attn and raises ImportError when it is missing.

Why tiling attention is the standard fix for the model's most memory-bound op: GPU Training Theory §6.

Supported backends

BackendGPUCPGptOss
flash_attention_4Blackwell SM100+ (CuTe DSL)YesYes — every sink policy (neutralized, live-frozen via learnable_sink, trainable via the sink-gradient rescale)
flash_attention_2Ampere+YesYes (sinks reset; live sinks rejected)
flash_attention_3Hopper SM90 onlyYesYes (sinks neutralized; live sinks need an s_aux build, trainable sinks rejected)
flex_attentiontorch ≥2.5No (auto-switched to the arch's flash kernel: FA4 / FA3 / FA2, below)Yes (LSE renorm; FSDP2 NaN fix auto-applied)
sdpaallNoYes when sinks reset (reset → dropped sink column is exp(dtype.min)=0; loader flips _supports_sdpa). Rejected when sinks are kept live.
eagerallNoYes

GPU support: A100 (SM80) = FA2/flex/sdpa. H100/H200 (SM90) = FA2/FA3/flex/sdpa, no FA4. B200 (SM100) / B300 (SM103) = FA2/FA4/flex/sdpa, no FA3.

The CP column above is what reject_unsupported_ulysses_model (halo/distributed/context_parallel/model_rules.py) accepts as a config label; it is not which kernel runs. CP's attention call goes through _flash_attn_func (halo/distributed/context_parallel/ulysses_attention.py), which ignores that label and probes in a fixed order: the architecture's preferred_flash_impl (halo/models/attention/selection.py: FA3 on Hopper (SM90), FA4 on Blackwell (SM100+)), then FA2.

Both architecture probes precede FA2 because every image ships FA2; an FA2 probe placed first would resolve on every device and leave the arch-matched kernel unreachable.

The one exception is model-level: _flash_attn_func takes an allow_fa4 flag, resolved per wrapper from supports_fa4_backward, so GLM-4.7-Flash runs FA2 on Blackwell under CP in every run (Model-specific handling); Qwen3.5/3.6 and Qwen3-Next, whose FA4 a deterministic run also refuses, take no CP. The probe overrides the configured label, so that flag is the only protection there. preferred_flash_impl is also what the automatic detection and the loader's CP label read, so CP and the config-level choice can never disagree about a device.

The order matters because CP calls the non-varlen forward, the path FA2 serves worst on both architectures.

On B300 (Halo 1.1.0 or earlier), CP on FA4 measures 1.2–3× FA2 at ≥32k tokens with equal peak memory, the margin widening with sequence length (+19% at 32k, +63% at 128k on gpt-oss-20b ep8+cp8; +197% on dense Qwen3-4B ep1+cp8 at 128k). FA2 is faster only below ~16k, where per-call overhead dominates.

Automatic detection

_detect_attention_impl (halo/models/attention/selection.py) takes preferred_flash_impl, which reads only the compute-capability major:

  1. major >= 10 (Blackwell SM100/SM103): flash_attention_4 if flash_attn.cute imports, else flash_attention_2. A build that is present but unusable warns before degrading.
  2. major == 9 (Hopper): flash_attention_3 if flash_attn_interface (else flash_attn_3) provides the varlen kernel, else flash_attention_2.
  3. Anything older, or any exception: flash_attention_2.

The detection above, the sink probe and the FA4 warm-up resolve kernels through flash_kernel(attn_impl, varlen=...): one module table in transformers' dispatch order (FA3 through flash_attn_interface first), FA4 imported only after ensure_fa4_kernel_cache_env. A build missing the entry point reads as absent. Context parallelism resolves its kernel through flash_kernel too, in preferred_flash_impl's order under its FA4 flag, then FA2 (above); the trainable-sink rescale imports flash_attn.cute directly.

The major >= 10 test also admits consumer Blackwell (SM 12.x, RTX 50-series), where flash_attn.cute's SM100 kernels do not run. That is not a supported target — the images build for B200/B300 and H100/H200.

resolve_attn_implementation then applies the model-level overrides below. One is precision-driven: in fp32, any flash_attention* is redirected to flex_attention when the model has sinks or an attention softcap and sdpa otherwise — flash kernels are half-precision only. Context Parallelism in fp32 raises, since CP runs attention through FlashAttention unconditionally.

The Blackwell image ships both FA2 (top-level flash_attn, inherited from the base image) and FA4 (flash-attn-4 4.0.0b37, the flash_attn.cute submodule) under one namespace; FA4 is the production default. Profiling benchmarks leave --attn_implementation unset, so they run FA4 on Blackwell too. transformers dispatches flash_attention_4 natively through flash_attn.cute (causal, sliding-window, sinks via s_aux, softcap, varlen). FA4 serves a softcap on the dense call only: its varlen kernels are not validated with one, so the varlen entry point refuses a softcapped call (Gemma 2 27B, below). 4.0.0b37 is beta — validate accuracy-sensitive runs. Hopper keeps FA2 + FA3.

FA4 vs FA2 vs SDPA on Blackwell

The tables in this section and the backend benchmark are from Halo 1.1.0 or earlier, before FA4 4.0.0b37. The 4.0.0b37 figures quoted under Model-specific handling are from 1.2.0.

FA2 is an SM80-style kernel, untuned for Blackwell. FA4 is the Blackwell-native CuTe DSL kernel: 3.6–3.9× faster than FA2 on the isolated attention kernel (microbench B2×S8192×H32×D128: fwd 2.71→0.70 ms, 3.9×; fwd+bwd 10.9→3.0 ms, 3.6×). Both match an fp32 SDPA reference to ~2e-3 (bf16 floor) across MHA/GQA and head_dim 64/128.

The end-to-end step win is a function of attention's share of the step, which grows with sequence length (attention is O(seq²), the rest ~O(seq)). Qwen3-4B-Instruct-2507, batch 1, GC on, Liger on, tok/s/GPU on one B300 (every cell on the same device), mean of two runs, Blackwell image. Runs repeat within 1.1%, except flex at 4,096 (7.9%):

SeqLenFA4FA2SDPAflexFA4/FA2
4,09623,61419,92025,73320,1471.19×
16,38423,13912,28223,11216,1671.88×
32,76817,5047,49717,04710,6072.33×

At 4k batch-1 the step is overhead-bound, so FA4 stays close to FA2 (1.19×); the gap opens with length and reaches 2.3× at 32k. SDPA (Blackwell-tuned cuDNN kernel) leads FA4 by 9% at 4k and is within 3% of it from 16k, so SDPA is a fine fallback for plain dense models. flex trails both at every length; it ties FA2 at 4k (within its own 7.9% run spread) and beats it from 16k.

On sparse MoE expert work and the wait for the busiest rank take most of the step, so the kernel speedup is only +17% end-to-end. gpt-oss-20b, EP=8, seq 16,384, batch 1, GC on, 8× B300 (Blackwell image, median of 2): FA4 12,382 vs FA2 10,627 tok/s/GPU (1.17×); the FA4 arm is the s16384 row of the ep8 sequence sweep in Throughput Benchmarks. FA4 is the production default for gpt-oss SFT; SDPA also dispatches once sinks are reset but lacks FA4's native sink/sliding-window/softcap handling.

EP/long-context throughput tables live in Throughput Benchmarks, which already run the FA4 default.

Model-specific handling

load_distributed_model applies these automatically.

Nemotron-H takes FA4 on both towers, or FA2 where no FA4 build imports. The text tower calls 32 query heads over 2 KV heads at head_dim 128, causal, through the varlen kernel across packed documents and padded rows. The Radio vision tower calls 16 heads at head_dim 80, non-causal, through the varlen kernel across the images of a batch. The warm-up compiles both towers' calls, as it does any non-causal tower on FA4. At 8,192 text tokens a call costs 2.2 ms on FA4 and 6.3 ms on FA2, and at 13,322 vision tokens a block under checkpointing 7.3 ms and 18.8 ms. kernels/test_nemotron_attention_kernels.py holds both calls against eager through the real modules.

GptOss (attention sinks):

  1. Disables transformers' auto-fallback to vllm-flash-attn3 (ensure_gpt_oss_local_flash, run once when halo/models/attention/gpt_oss_sinks.py is imported, which every toolkit load path does first). transformers reroutes GptOss FA2 and FA3 requests to kernels-community/vllm-flash-attn3: SM90-only binaries fetched through the kernels package, which the images do not ship. Every GptOss build in the process therefore keeps the local flash build, a reward-model scorer's or a sentence-transformers backbone's included.

  2. Applies the run's sink policy (halo/models/attention/gpt_oss_sinks.py). The policies — neutralized, live and frozen, trainable — and what each refuses are on GPT-OSS → Attention sinks.

  3. Variable-length RL (async GRPO) runs FA4 via a per-row dense forward (_dense_last_hidden_state), for exact RoPE positions and rank-uniform collective counts (rows_forward_densely).

    The dense kernel is seqlen-invariant, so the path forwards each row on its own, trimmed to its real contiguous span, with attention_mask=None: RoPE positions [0, len), log-probs bit-identical to the unpadded forward. FA4 and the row-position families are where the trim is mandatory; rows_forward_densely also takes it on any attention path at per_device_train_batch_size: 1, for the cost alone (Chunked log-probs).

    Per-rank forward counts are all-reduced to a max and padded with tied dummy forwards, and every row — the padding included — stays connected to the loss, so the forward and backward collective sequences stay in lockstep. A disconnected row would let autograd prune its backbone backward and desync the FSDP/EP grad collectives across ranks.

Fallback chain for sinks models: requested → flash_attention_4 (added as a candidate when the sinks are live) → flash_attention_3 (Hopper, where its build imports) → flash_attention_2 → flex_attention → SDPA (only when sinks are reset) → eager (select_supported_attn_implementation); under live sinks every sink-dropping candidate is filtered out.

With sinks reset for SFT (the default), an explicit attn_implementation='sdpa' is kept — install_sdpa_for_reset_sinks flips _supports_sdpa because the dropped sink column is exp(dtype.min)=0.

With live sinks select_supported_attn_implementation admits only the sink-carrying impls — FA4 (learnable_sink), FA3 with s_aux (Hopper), flex_attention, eager — and raises on flash_attention_2 and sdpa (why). The Docker Hopper FA3 build exposes no s_aux, so live-sinks runs on Hopper use flex_attention or eager.

Capability is read from the installed kernel's signature (_attn_impl_handles_sinks), probing the module transformers actually dispatches (FA3 via flash_attn_interface first, then flash_attn_3), so a flash-attn upgrade that adds sink support is picked up automatically.

The probe reads the varlen entry point (flash_kernel's default): transformers builds its per-argument capability map from _flash_varlen_fn alone and applies it to the dense call too, so a sink argument only the dense signature accepts would never be forwarded.

The resolver takes the run's SinksPolicy and records its reset verdict on the config instance, so the nested EP/CP/lazy-loader re-validations (which pass no policy) enforce the same matrix. A from-scratch build resolves as live sinks: it has no pretrained sinks to neutralize.

Gemma4 (5 full-attention layers at global_head_dim=512): FA2/FA3/FA4/cuDNN-SDPA all reject head_dim>256 (FA2 cap 256, cuDNN cap 128 on cu13, FA4's SM100 kernel overflows tensor memory and asserts in flash_fwd_sm100); math SDPA materializes [B, heads, S, S] (64 GB/layer at 32k → OOM).

The loader therefore redirects any FlashAttention impl — auto-detected or caller-supplied — to SDPA for any model whose widest head exceeds 256 (has_head_dim_beyond_flash, off the per-layer head_dim), and on SDPA patch_sdpa_for_wide_heads forces mem-efficient SDPA (the only backend supporting head_dim=512) and sets transformers.integrations.sdpa_attention.use_gqa_in_sdpa → False for manual KV repeat. On that path alone (HALO_FLEX_SLIDING=0), Gemma4 32k EP=8 peaks at ~155 GB/rank (Halo 1.1.0 or earlier). On CUDA the model itself is built with sdpa_flex_sliding, which moves the sliding and short-context global layers off that kernel (Gemma 4).

Qwen3.5 / Qwen3.6 / Qwen3-Next train on FA4's head_dim-256 path, in every script: its general SM100 forward kernel and its dedicated head_dim-256 backward, for a 256-wide query/key head (uses_fa4_dedicated_head_dim, which counts MLA's qk_nope_head_dim + qk_rope_head_dim too; transformers' flash path pads a narrower value head up to that width). Qwen3.5's partial-rotary torch.cat hands that path q/k strided, which it reads as they are (below). The dedicated backward's deterministic mode is not validated here, so a run that asks for one (FLASH_ATTENTION_DETERMINISTIC=1, which full_determinism sets as the config is parsed, ahead of SFT's row plan and the model load) falls back from FA4 to SDPA on a head_dim-256 model FA4 otherwise trains (supports_fa4_backward), to FA2 under Context Parallelism, the kernel the Ulysses path then runs, and to FA2 on SFT rows that hold several documents (below).

SFT rows that hold several documents (packing, padding_free) load FA2 where FA4 gains little on them or would leave them to SDPA (default_training_rows_attn_implementation in halo/entrypoints/model_setup.py). On the GatedDeltaNet families that holds when attn_implementation is unset and the GPU's pick is FA4, deterministic run or not (keyed on uses_fa4_dedicated_head_dim): FA4 runs packed rows only 2.9% faster at 16k and 4.6% at 32k (Qwen3.5-35B-A3B EP8 packed SFT on B300, FA4 4.0.0b37), below the 5% gain a default change needs, so a pinned FA4 keeps them outside full_determinism (the request logs that reason and that pin). Under full_determinism it holds on every model the run demotes off FA4 (is_fa4_demoted_by_determinism), a pinned attn_implementation: flash_attention_4 included: SDPA runs such a row as one dense [L, L] mask with K/V repeated per query head (a one-step packed Qwen3.5-35B-A3B EP8 run: 367 s on SDPA, 162 s on FA2). FA2 consumes the cu_seq_lens the collator already emits. Unpacked rows keep FA4, 1.18× FA2 at 16k and 1.33× at 32k on the same model and FA4 release (one round), and padded rows of one document keep a deterministic run's SDPA.

Qwen3.5's partial-rotary torch.cat leaves q/k head-major before the flash transpose, MLA hands FA4 a value sliced off a fused K/V tensor, the Ulysses all-to-all at batch 1 hands it q, k and v each sliced off the one buffer it receives (token stride their summed width), and on every family a forward with the KV cache on leaves k/v head-major. FA4's head_dim-256 path reads those strided q/k/v as they are (Dao-AILab/flash-attention#2666): tests/gpu/kernels/test_fa4_hd256_strided_inputs.py holds its output and gradients to FA2's error against fp64 on those layouts, and test_fa4_hd256_family_parity.py per head_dim-256 family. The dedicated head_dim-256 backward asserts against softcap and sliding-window attention, so the resolver keeps Gemma 2 and Gemma 3 at head_dim 256 off it, as it keeps GLM-4.7-Flash (next paragraphs).

Gemma 2 / Gemma 3 at head_dim 256 (2B/9B, 1B/4B/12B): auto-fall back from FA4 to FA2 (fa4_dedicated_kernel_refusal), ahead of the deterministic-backward fallback, so a full_determinism run lands on FA2 too. FA4 trains a 256-wide query/key head only through its dedicated SM100 2-CTA backward kernel, which asserts on a softcap (Gemma 2's attn_logit_softcapping) and on a window (Gemma 3's sliding layers, once a row is longer than the window). The demotion reads the query/key width (uses_fa4_dedicated_head_dim, MLA's qk_nope_head_dim + qk_rope_head_dim included), the softcap and the declared layer_types, not the family: the hd128 27B sizes keep FA4, whose general kernel takes both, and a head_dim-256 model with neither keeps FA4 too.

GLM-4.7-Flash (GLM-4 MoE Lite): multi-head latent attention at a 192 + 64 = 256-wide query/key head and a 256-wide value is kept on FA2 on Blackwell as a precaution, in every run, a pinned flash_attention_4, a deterministic run and Context Parallelism included (fa4_dedicated_kernel_refusal, keyed on a kv_lora_rank latent at that width, has_latent_attention, not on the family). The precaution rests on a training result on FA4 4.0.0b16, not on a kernel defect: on its dedicated kernel (contiguous q/k/v) a run's step-2 gradient norm measured about 4x FA2's, a heavy tail in about 20 tokens of one document, while the kernel matched FA2 on that run's own tensors. No run measures the shipped 4.0.0b37 on this model, so the demotion stays. Latent attention at other widths (Mistral4's 128-wide query/key) keeps FA4's general kernel.

Gemma 2 27B (head_dim 128) on varlen rows: a softcap stays off FA4's varlen kernels (fa4_varlen_refusal): they are not validated with one. Every training batch but one unpadded row reaches them. Packed and padding-free rows hold several documents, and transformers unpads any batch holding a padding token, so more than one row per batch reaches it too (rows_reach_varlen_kernel in halo/data/collators/collator_rules.py). The resolver does not see the rows, so the scripts decide before the load. With the YAML pinning nothing, SFT, self-distillation and the embedding backbone the parallel loader loads request FA2 for a softcapped model on such rows (default_training_rows_attn_implementation, a pre-processed corpus judged as it was baked; an fp32 run takes flex_attention instead), and the padded scripts request FA2 (Padded workloads, below). One unpadded row per batch (per_device_train_batch_size: 1, which TrainingArguments defaults to 8, and no packing) keeps FA4's dense kernel, which takes the softcap: the SFT collators hand a softcapped model's one-document rows no varlen set on FA4 (One-document batches). Evaluation batches count the same way: a run that evaluates (eval_strategy other than no) at per_device_eval_batch_size above 1, which TrainingArguments defaults to 8, takes FA2 too. A reference takes the policy's request, so it runs FA2 beside a policy moved there, and a softcapped teacher moves the padded scripts' shared request as a softcapped student does. A pinned FA4 the load would keep on such rows is refused with the remedy before any weight loads (reject_fa4_varlen_training) in SFT, self-distillation, teacher distillation and the embedding parallel-loader path. Any softcapped call that still reaches FA4's varlen entry point, a pinned FA4 on the other padded scripts among them, raises there in the forward, before the kernel launches (install_fa4_varlen_softcap_refusal, which apply_family_patches installs on every FA4 load of a softcapped model).

Attention logit softcapping (Gemma 2): SDPA is refused for any config setting attn_logit_softcapping, requested or defaulted, because transformers' SDPA path never reads the softcap and would train a different model without a word. The validator falls back to FA3 on Hopper, then FA2, then flex_attention (which applies the softcap as a score modifier), then eager; in fp32 it skips both flash kernels. FA3, the arch's own kernel, comes first on Hopper; a requested FA3 carries the softcap and stays.

Ling 2.0 and Ring (bailing_moe, bailing_moe_linear): remote code on transformers v4's Llama flash template, whose table, support flag (_supports_flash_attn_2, which v5 ignores) and kernel call know only flash_attention_2. patch_remote_llama_flash_template (halo/models/attention/remote_flash_attention.py), run on every remote modeling module the remote-class funnel loads, makes the auto-selected FA4/FA3 build and run varlen over packed documents; the file's own SDPA and eager masks ignore position_ids, so packing is refused there (Bailing → Attention and packing).

Ling 3.0 (bailing_hybrid): the file has no flash attention class, only the v4 support flag, so every flash label, and an unset one outside the padded scripts' default (below), is refused at model build; set attn_implementation: sdpa. Its MLA mask ignores position_ids as Ling 2.0's SDPA path does, so packing is refused and Ling 3.0 trains on padded batches.

DeepSeek-V4 (deepseek_v4): eager-only. head_dim=512 exceeds every FlashAttention kernel's 256 cap, SDPA drops the learnable sink column, and the CSA/HCA compressors concatenate KV entries after the mask is built (no BlockMask resize), so transformers marks all non-eager backends unsupported. resolve_attn_implementation forces attn_implementation="eager" for any requested backend (the family spec's eager_only_attention). There is no varlen path under eager, so the collator factory rejects padding_free (see Padding-Free Collator). See DeepSeek-V4.

Qwen3.8-Flash-Next (qwen4_exp): the class declares no flash or flex support, so the load takes SDPA. Its sparse (QSA) layers run Halo's block-sparse kernel under the default HALO_QSA_KERNEL=auto where it can run, and the tiled path (masked SDPA per 128-query tile, memory linear in the row) elsewhere, always under full_determinism (Qwen3.8-Flash-Next).

PaliGemma / PaliGemma 2 (prefix LM): image and prompt tokens attend bidirectionally and the trained tokens causally, a mask transformers builds from token_type_ids (0 prefix, 1 suffix). transformers also sets the text config's use_bidirectional_attention whenever a checkpoint leaves it unset, as every published one does, so each decoder attention module is built non-causal and attends to every token wherever no mask is materialized. No flash kernel reads token_type_ids, so the resolver never selects one for the family (has_prefix_lm_attention in declares_flash_attention), auto-detected or pinned: PaliGemma takes SDPA, PaliGemma 2's softcap rules SDPA out and it takes flex_attention; a pinned flex or eager stands. The VLM collators write the token_type_ids that put each row's trained tokens in the causal suffix (SFT → Vision-language models), and every training load makes the decoder causal where no mask overrides it (apply_prefix_lm_causal_decoder), so a batch without them scores causally.

GPT-J, GPT-Neo, Falcon and Bark: their modeling files list one flash class, FA2's, so resolve_attn_implementation refuses any other flash label, unset included, naming flash_attention_2. attention_dropout: transformers hands only FA2 its dropout_p, so a config with a non-zero value is refused on FA3 and FA4 rather than trained without it. Live sinks are read off the built model, under any family: verify_live_sinks_carried refuses a backend that drops them (Granite SWA on FA2 or FA3).

Padded workloads: the scripts that forward right-padded batches — DPO / SMPO / KTO, teacher distillation, reward modeling, classification, and all three GRPO scripts (offline, online, environmental) — share one default for an unset attn_implementation (default_padded_attn_implementation; a pinned value wins). On Blackwell they request nothing and take the loader's own pick, FA4 where it applies and every demotion above where it does not: at head_dim 256, FA4 for Qwen3.5/3.6, Qwen3-Next and Gemma 1, FA2 for Gemma 2/3 and GLM-4.7-Flash, never flash for PaliGemma (above). A remote-code checkpoint keeps SDPA there, since its class is invisible to the validator (Bailing's refuses every flash label at build). A softcapped one at any width, the policy's or a teacher's, moves the request off the loader's pick too, since its padded batches would reach FA4's varlen kernels: a softcapped policy requests FA2, which trains the softcap (SDPA where no FA2 build imports), and a softcapped teacher beside a policy without one leaves the request at SDPA, which the teacher's softcap turns into FA2 at its load. Every other GPU requests SDPA, which a softcap turns into FA3 on Hopper. Teacher distillation decides one request on both the student's and the teacher's checkpoint, and the DPO/KTO and GRPO references take the policy's request. The online and environmental GRPO scripts, under packed_microbatch_tokens and a YAML pinning nothing, plans its rows as packed ones instead (plan_row_attention, packing=True, evaluation batch 1): FA2 for GatedDeltaNet decoders on Blackwell and under full_determinism, FA3 on Hopper, the loader's pick otherwise; with reset_sinks: false it keeps the loader's pick, as FA2 carries no sinks. A resolved backend that is not varlen flash is refused at construction (Packed micro-batches).

Every one of them takes that default only under reset_sinks: true, and offline GRPO and SMPO also skip it under CP, leaving an unset label to the loader's FlashAttention selection, which Ulysses requires. With live gpt-oss sinks (reset_sinks: false) the default drops and the model config passes through untouched, since SDPA drops the sink column and would be rejected outright. SFT keeps the auto-selected FA4 (FA2 for GLM-4.7-Flash, for a softcapped model and on the head_dim-256 GatedDeltaNet families' multi-document rows, above): a packed batch takes its varlen path on the set the packing collator builds, whose max_length is a host int. On FA4 the GRPO scorers forward each row densely, trimmed to its real span (rows_forward_densely, under GptOss above).

Measured on Halo 1.1.0 or earlier, 8× B300, with benchmark_smpo_ep.py (the gpt-oss-20b rows from a separate run set; Blackwell image; auto = FA4; mean of 2 runs). Its rows are padded close to --seq, so a batch with heavy padding is not measured. The SMPO collator emits no input_ids, so the callback's tokens/s is a padded-length estimate; the table gives the step-time speedup of auto over SDPA:

modelseqauto vs SDPA
gpt-oss-20b ep840961.36×
gpt-oss-20b ep881921.42×
qwen3-30b-a3b ep840961.17×
qwen3-30b-a3b ep881921.14×
qwen3-8b ep1 (dense)40961.26×

FA4 wins every row, on MoE and dense alike.

Context Parallelism: an explicit flex_attention label auto-switches to the kernel _flash_attn_func runs (preferred_flash_impl under supports_fa4_backward: flash_attention_4 on Blackwell, flash_attention_3 on Hopper, else flash_attention_2) since Ulysses can't dispatch flex; see Supported backends. SDPA/eager unsupported.

Installation

Flash Attention is not in pyproject.toml — it must be built from source against your PyTorch ABI. The prebuilt images already carry it; this is for a bare host.

Bash

# FA2 — restrict FLASH_ATTN_CUDA_ARCHS to your target arch (the default
# "80;90;100;110;120" builds 5 archs, ~5x compile time).
FLASH_ATTN_CUDA_ARCHS=100 MAX_JOBS=16 NVCC_THREADS=4 \
  pip install --no-cache-dir --no-build-isolation flash-attn

# FA3 (Hopper only)
git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention/hopper && pip install --no-cache-dir --no-build-isolation .

On Hopper (sm_90) under CUDA 13.2, ptxas hangs on the 24 flash_fwd_split_hdim*_*_sm80.cu kernels. The Docker Hopper build replaces them with throw-stubs (docker/training/flash_attn_split_stubs_hopper.cpp) and rewrites flash_api.cpp's two /*num_splits*/ 0 arguments to 1, asserting both: with dropout 0 the occupancy heuristic behind them picks split-K whenever batch*heads*ceil(seq_q/64) is under ~`0.8*2*SMs` (an unpadded row of a few hundred tokens at batch 1, and a decode step under GQA, whose varlen call takes split-K too), and the dense and varlen forwards then run the one-split kernel instead. The backward never uses split-K. Building FA2 for Hopper outside Docker on CUDA 13.2 needs the same two changes (or CUDA 13.1).

Import FA3 from flash_attn_interface (import flash_attn_3 succeeds but does not guarantee a functional build). transformers auto-detects FA3 through is_flash_attn_3_available(), which reads the flash_attn_interface package metadata; the Docker Hopper build writes that dist-info and a flash_attn_3 re-export so both import forms resolve; flash_kernel reads flash_attn_interface first, as transformers dispatches.

After a PyTorch upgrade the .so is ABI-incompatible (undefined symbol: _ZNK3c106SymInt...) — rebuild with --force-reinstall. If pip pulls a different torch as a build dependency, reinstall torch~=2.11.0 from the cu130 index.

Usage

A YAML config can pin the backend (attn_implementation: flash_attention_2); omit the field for auto-detection plus the model-specific handling above.

Mask construction on flash. Every flash load installs patch_transformers_packed_sequence_probe (halo/models/attention/backend_setup.py, from apply_family_patches). Without it, transformers' causal and sliding-window mask builders test position_ids for packed documents whenever no padding mask is passed (find_packed_sequence_indices, one device read per mask at the start of every forward), and flash_attention_mask discards the packed mask that test feeds. The skip reads the mask interface off the registry per model, so an SDPA, eager or flex model in the same process keeps the probe its packed mask needs.

Gradient checkpointing with EP. EP MoE layers go through enable_scoped_gradient_checkpointing (halo/models/gradient_checkpointing.py), which enables GC and installs the recompute scope its recompute replays the DeepEP dispatch from. Enabling GC by any other route leaves EP without that scope and raises.

EP, CP and every MoE force use_reentrant=True (a configured False is overridden with a warning; a MoE routes in its recompute, and non-reentrant checkpointing rejects the routing shapes that change when a nondeterministic attention recompute flips a near-tie pick). Pipeline parallelism is the exception: a reentrant forward runs under no_grad, where FSDP2 registers no pre-backward hooks, so PP stages run non-reentrant and refuse use_reentrant: true (PP rejected combinations). Under FSDP2, a layer the reentrant backward recomputes more than once gets its post-backward completed by the trainer (Data Parallelism), and a tower block behind a frozen patch embedding takes an input that carries its gradient (Tower blocks).

Backend benchmark

Forward+backward on B300 (SM103), seq 16,384, GC on, BF16: SDPA and FA4 tie for the lead on both Qwen3-8B dense (11,080 tok/s/GPU) and Qwen3-30B-A3B MoE (9,870), with flex at 1.23–1.33× their step time, FA2 at 1.42–1.57×, and eager at 6.55× on dense while the MoE OOMs. Memory is identical across working backends — weights + activations dominate, not the kernel.

Bash

torchrun --nproc_per_node=1 \
    tests/gpu/profiling/benchmark_attention_implementations.py \
    [--model qwen3-30b-a3b --gradient_checkpointing] [--seq 8192] [--backends flash_attention_2 sdpa]

Choosing a backend

Every row is what the loader picks on its own; the reason for each redirect is in Model-specific handling.

ScenarioBackend
Blackwell trainingflash_attention_4
Hopper trainingflash_attention_3 (else FA2 when not installed)
No flash-attnFA2 is still requested and the model build raises — set attn_implementation: sdpa yourself (CP then unavailable)
GptOssFA4 on Blackwell, FA3 on Hopper
Qwen3.5 / Qwen3.6 / Qwen3-NextFA4 on Blackwell (head_dim-256 path), padded scripts included; under full_determinism → SDPA; FA2 for SFT packing / padding_free rows in either case
GLM-4.7-Flash→ FA2 on Blackwell in every run, CP included (kept there as a precaution, above)
Gemma 2 / Gemma 3 at head_dim 256→ FA2 on Blackwell (FA4's head_dim-256 kernel takes no softcap or window); never SDPA under a softcap
Gemma 2 27B (head_dim 128)FA4 at one unpadded row per batch, training and evaluation; → FA2 on packed, padding-free or padded rows (a softcap stays off FA4's varlen kernels)
Padded scripts (DPO, SMPO, KTO, RM, classification, distillation, GRPO), unpinnedthe loader's pick on Blackwell (FA2 for a softcapped policy; SDPA for remote code and for a softcapped teacher, which lands on FA2); SDPA elsewhere, which a softcap turns into FA3 on Hopper
Gemma4→ sdpa_flex_sliding on CUDA: FlexAttention on the sliding layers, matmul attention on the head_dim-512 global layers, past a score-memory budget mem-efficient attention per packed document (Gemma 4)
Ling 2.0 · RingFA4 on Blackwell, FA3 on Hopper (patch_remote_llama_flash_template)
Ling 3.0no fallback: the flash label fails the model build — set attn_implementation: sdpa
Lagunathrough remote code → SDPA with a warning, and a pinned flash label raises: its flash call never receives the sliding window (remote_code_flash_loses_window). In-library classes: FA4 on Blackwell, FA3 on Hopper
LFM2-VL→ SDPA (its SigLIP2 tower declares no flash support, and declares_flash_attention reads each sub-model's class)
DeepSeek-V4→ eager
GLM-5 Next · Inkling→ SDPA (upstream declares no flash support)
Step-3.7 Flash→ SDPA (upstream declares no flash support), built as sdpa_flex_sliding on CUDA: FlexAttention on the sliding layers and on the full layers' packed or padded rows
Qwen3.8-Flash-Next→ SDPA (upstream declares no flash support); its sparse layers run the block-sparse kernel (HALO_QSA_KERNEL=auto) where it can run, the tiled QSA elsewhere (full_determinism, non-CUDA, unsupported dtype or shape)
PaliGemma · PaliGemma 2→ SDPA · → flex_attention (prefix LM: no flash kernel reads its token_type_ids mask, above)
Context ParallelismFA3 on Hopper, FA4 on Blackwell — FA2 for GLM-4.7-Flash

Known issues

flex_attention + FSDP2 NaN gradients (GptOss). Loss starts normally then collapses to 0 with NaN grad_norm. transformers wraps flex_attention in torch.compile(dynamic=False) (WrappedFlexAttention); the compiled backward produces NaN when GptOss sinks (plain nn.Parameter) interact with FSDP2's DTensor-wrapped LSE during post-attention renorm.

patch_flex_attention_compile() (halo/models/attention/backend_setup.py) bypasses the compile wrapper. load_distributed_model applies it whenever flex is the resolved backend and either EP is on (the compile path also deadlocks with EP's all-to-all on seq-length recompiles) or the model is a sinks model running with reset_sinks: true.

The bypassed flex runs uncompiled: torch's unfused path holds the full fp32 [B, heads, L, L] score matrix per attention call, about 16 GiB at GPT-OSS's 64 query heads and an 8,192-token row, so under EP memory bounds the row length, as it does on eager (bf16 scores, about half).

flex_attention gradient overflow on dense Qwen3. flex produces ~6e18 grad norms on Qwen3-0.6B/8B, even single-GPU without distributed wrapping; losses are fine but overflow is a stability risk. Use the auto-detected flash backend for training, not flex. Upstream PyTorch/transformers issue.

FA4 first-use compile under multi-rank parallelism. FA4 JIT-compiles each kernel on first use (~10 s) mid-forward. A cache-miss compile on one rank stalls it while peers race ahead to the next collective (a TP o_proj all-reduce, an EP DeepEP dispatch), desyncing the group into a deadlock.

warm_attention_kernels (halo/model_loading/parallel/whole_model.py, called at the end of load_distributed_model, of the frozen reference/teacher loader, of the embedding SentenceTransformer load and of a trainer's path-string model load, resolve_trainer_model) runs the rank-local compiles on every rank behind one fence (barrier_on_exit), so the training loop only hits warm kernels: warm_fa4_kernels (halo/models/attention/backend_setup.py) when FA4 is active, for each attention layer's own call (below; MLA's nope + RoPE query/key width) and each non-causal vision tower's, and warm_flex_sliding_kernels when the model runs sdpa_flex_sliding (below). It is a no-op at world_size == 1, where no peer waits on a compiling rank and each kernel compiles at its first call. Tree packing warms the kernel variants it adds behind the same fence (warm_tree_attention_kernels). The FA4 and tree warm-ups are best-effort: a failure logs a warning on that rank, which then compiles at first use. Only a failure that reaches the fence, a FlexAttention warm-up's today, raises on every rank.

It covers both entry points — dense flash_attn_func and varlen flash_attn_varlen_func over a two-document cu_seqlens — for each distinct call the model's attention modules make, each in the variants it can reach: windowless always (a row no longer than the window drops it), sliding-window where the layer has one, the layer's logit softcap where it passes one (Gemma 2; FA4 hashes the value into the kernel key), on the dense call only: a softcap stays off FA4's varlen kernels (above), so the warm-up launches no varlen call on a softcapped layer; and a learnable sink sized to the layer's heads when the model carries sinks. Each call passes the deterministic flag transformers passes its kernels (FLASH_ATTENTION_DETERMINISTIC as set at the load, read by flash_attention_deterministic), which FA4's backward keys its kernel on; the Ulysses CP call passes the same flag (Context Parallelism → Requirements). Both warm-ups read that call off each module (attention_calls in halo/models/attention/layout.py): its own query heads, KV heads from its GQA group count, the query/key width (MLA's qk_head_dim), the value width (MLA's v_head_dim: FA4 warms the value at the query/key width, which transformers' flash path pads it up to, FlexAttention at its own), the window it passes (its config's where the module keeps none: Mistral, Phi-3) and its softcap. FA4 keys a kernel on the query/KV head ratio, so a per-layer head count gives each layer type its own kernel: Step-3.7's 64 full / 96 sliding heads, whose global config.num_attention_heads raises, Laguna's 48 / 72, whose global count is the full layers', and Inkling's sliding layers. tests/cpu/models/test_attention_warmup_geometry.py runs a training step of every roster model that declares flash support through transformers' flash path after the warm-up, with and without FLASH_ATTENTION_DETERMINISTIC=1, and checks that its forward and backward dispatch no kernel the warm-up skipped; tests/gpu/kernels/test_attention_warmup_per_layer_heads.py trains reduced Step-3.7, Laguna and Gemma 2 models after the warm-up (Gemma 2 on its single-document rows) and checks that no FA4 JIT cache and no FlexAttention graph gains an entry, and that no FA4 varlen call carrying a softcap, the warm-up's included, reaches the kernel.

The on-disk cache (ensure_fa4_kernel_cache_env in halo/kernels/jit_cache.py, anchored on HF_HOME) makes the compile a one-time cost across processes. The same call turns on CUTE_DSL_ENABLE_TVM_FFI where the environment leaves it unset, and pins Liger's RoPE to Triton, whose CuTe DSL kernel is wrong under it (Liger → Kernel implementation). The DeepEP ElasticBuffer dispatch/combine timeout is long (100 s) and absorbs residual skew. FA2 is precompiled and unaffected.

FA4 varlen max_seqlen (packed / masked batches). A packed or attention-masked batch unpads to a cu_seqlens varlen FA4 call. transformers' _process_flash_attention_kwargs coerces the 0-dim max_seqlen tensor to int only under tracing; in eager it forwards a fresh CUDA tensor each step. On Blackwell FA4 reads no tensor bound on the host, and it compiles its head_dim-256 varlen kernels one way for a host-int bound (a rectangular grid) and another for a tensor one (a flat grid), so the bound's type picks the variant that runs.

patch_transformers_flash_varlen_int_seqlen (halo/models/attention/backend_setup.py, applied by apply_family_patches on every FA4 load) coerces max_seqlen_{q,k} to int at that boundary, so every call runs the int variant warm_fa4_kernels compiles, at one .item() per call on a masked batch. A packed flash row arrives with the varlen set already built on the host, by the packing collator (Collators) or, under pipeline parallelism, by each stage per microbatch (PP packing), and so does an SFT batch of one unpadded document (One-document batches), so its max_seqlen is an int from the start and no layer reads it back; the coercion serves the masked batches and rows that reach attention without the set. warm_fa4_kernels warms the varlen backward too, so the first packed step is already compiled.

A padded batch on any flash backend derives its unpad set (token indices, cu_seqlens, max length) once per mask, not in every layer: patch_transformers_flash_unpad_memo (halo/models/attention/backend_setup.py, applied by apply_family_patches on every flash load) keeps it on the mask tensor, keyed on its version counter, so a forward derives it once per mask kind and the checkpoint recompute not at all.

Blackwell notes

vLLM on B200: set VLLM_ATTENTION_BACKEND=FLASH_ATTN where FlashInfer JIT-fails on SM 10.0. See Online GRPO.

Masked matmul attention

GLM-5 Next's DSA layers attend over a boolean [B, 1, S, S] mask holding about 1,800 to 1,900 keys per query, at most 2,051, which flash kernels refuse, so stock SDPA runs them on torch's mem-efficient kernel on Blackwell, a general one that scores every key of the row, and on cuDNN on Hopper, the one architecture where cuDNN takes this head width. By default on Blackwell, apply_family_patches replaces the sdpa slot of modeling_glm5_next's own AttentionInterface instance, the local override transformers documents, with masked_matmul_attention (halo/models/attention/masked_matmul.py). The config still names sdpa, so the mask, selection, warm-up and layout code read what they always did, and a tree registry installed later wraps this instance.

  • The forward runs SDPA's mem-efficient kernel, torch._scaled_dot_product_efficient_attention, one batch row and one tile of queries at a time, over the tile's keys and its part of the mask as an additive bias. Called directly, it takes no backend flag. Grad is off inside the autograd function, so it records no backward of its own, and it returns the tile's output and log-sum-exp. A query with no key gets a zero output and a zero log-sum-exp.
  • The backward replaces that kernel's, rebuilding each tile's fp32 scores and applying the closed form FlashAttention and tree sparse attention use: d_scores = probs * (g @ v^T - sum(g * out)) * scale, d_q = d_scores @ k, d_k += d_scores^T @ q, d_v += probs^T @ g, with d_k and d_v summed in fp32 by the GEMM itself. One Triton launch per tile (halo/kernels/masked_softmax.py) takes the scores, g @ v^T, the mask, the saved log-sum-exp and sum(g * out) to 16-bit probs and d_scores, as separate element-wise passes over every score would run at memory bandwidth; warm_attention_kernels compiles it at load, so no rank compiles mid-step. A frozen input skips its products. It saves q, k, v, the mask, the output and the log-sum-exp, no scores.
  • Tiles and keys. SCORES_PER_TILE bounds a tile at 2 GiB of fp32 scores, and a tile holds up to 6 GiB in the backward. Each tile scores keys only through the last one any of its queries selected, read off the mask, so no mask can lose a key and a causal one crops to the tile's end. Tile widths and key counts are multiples of 8 (GEMM_ALIGNMENT), up to at most 7 masked keys, as a bf16 GEMM runs 3–10× slower when either dimension of its second operand is off one: the keys are N in q @ k^T and g @ v^T and inner in d_q, the queries inner in d_k and d_v. A row's last tile stays unaligned, which costs under 1%.
  • What goes to SDPA: anything off CUDA, dropout, output_attentions, no mask or a non-boolean or per-head one, a KV cache (q_len != kv_len), differing query, key and value shapes, a head width off a multiple of 8, a DTensor or other tensor subclass, differing dtypes outside autocast, a dtype the kernel has no build for, and calls carrying position_bias or a paged cache. The vision tower reads the same module name and passes no mask. Under autocast the inputs take its dtype, as SDPA's do.

HALO_DSA_MATMUL=0 keeps stock SDPA. tests/cpu/models/test_masked_matmul_attention.py holds the backward to SDPA in fp32 and to rounding in fp64, over a stand-in for the CUDA-only forward kernel, on dense, causal, pool-shaped and padded masks at several tile budgets, with negative controls on the normalizer term and the crop, and a tiny GLM-5 Next to stock on loss and every gradient. tests/gpu/kernels/test_masked_matmul_attention.py holds the real kernel to stock and to an fp32 reference.

Against torch's mem-efficient kernel on real masks, on one B300 at 64 heads of 256 in bf16 (scripts/profiling/benchmark_dsa_attention.py), a forward, its recompute and a backward run 4.95× faster at 7,678 tokens and 5.13× at 15,659. The forward takes 16.6 ms against 25.4 and 65.9 ms against 105.6, the backward 15.1 ms against 188.4 and 61.0 ms against 777.8. Against an fp32 reference the output and d_v errors match and the d_q and d_k errors are about 18% lower. A call's peak above its inputs is 7.34 GiB against 1.54 and 8.65 GiB against 3.36, which leaves a training run's peak where it was (GLM-5 Next → Throughput).

Block-sparse attention (Qwen3.8-Flash-Next)

A QSA layer's query attends the block_topk compression blocks its indexer picked, each compress_ratio consecutive keys of its document, and the keys after its last complete block. halo/kernels/block_sparse_attention.py runs that as one Triton program per query token and K/V head, which loads the head's whole query group, loops over the selected blocks in key tiles and takes the tail as one more iteration of the same loop. Softmax and the output accumulate in fp32, and the backward adds dK and dV into fp32 buffers with atomics, so it is not deterministic.

  • What it reads. The selection's block ids ([B, N, block_topk], -1 unused, about 1 KB per token) and spans, a (first, visible) pair per query (8 B per token) derived from the row's documents: the document's first unpadded key and how many keys the query sees. A query's valid slots are a prefix, min(visible // compress_ratio, block_topk) long, so no per-query key list exists. The recompute of a checkpointed layer attends the cached selection, spans included.
  • Shapes (query heads, K/V heads, head width, ratio, top-k) come from the module and its config. The head width must be a power of two of at least 16, the ratio must divide and stay under the key tile, and a K/V head's query group, rounded up to a power of two, times the head width may not exceed 16,384 fp32 accumulator elements (the register budget), and the key and value tiles, with the fp32 copy of the value tile, must fit the device's shared memory; block_sparse_unsupported returns the reason otherwise.
  • One compile per layout. The row length, batch, strides and top-k are runtime arguments that Triton does not specialize, so one compile serves every pack length. finalize_loaded_model warms the forward and backward for each layer layout, in bf16 and fp16, and for both block-id dtypes, rank-local and without a barrier, through the warm-up the family patch registered (halo/models/kernel_warmups.py); a variant it did not cover logs a warning at its first use. The key tile, warps and stages per SM major are the LAUNCH_CONFIGS table (Blackwell's row; any other major takes DEFAULT_LAUNCH_CONFIG). A layout whose kernel fails to build at warm-up (Triton OutOfResources or a compile error) runs the tiled path under auto and raises under on.
  • HALO_QSA_KERNEL. auto (the default) takes the kernel where it can run and the tiled path elsewhere, off keeps the tiled SDPA path, on runs the kernel or raises ValueError where it cannot (not CUDA, a dtype other than bf16 or fp16, an unsupported shape, full_determinism). The tiled path stays the reference for the kernel's tests. On the real checkpoint it takes the LoRA step at 16k packed tokens from 28.95 s to 18.38 s (1.57×) at an unchanged peak, and torch's mem-efficient backward, 22% of the off step's kernel time, disappears from the profile (Flash-Next kernel defaults).
  • Indexer. The selection scores many queries per matmul: HALO_QSA_INDEXER_MIB bounds a chunk of whole 128-query tiles by its fp32 scores. Operands stay fp32 as transformers', and a query whose k-th and next block tie is re-scored alone, so the exactly-tied selection does not depend on the chunk.

LoRA on q_proj, k_proj, v_proj and o_proj is plain autograd around the kernel's function, which returns dQ, dK and dV. Attention runs inside one stage and issues no collective, so FSDP2, EP, ETP and PP stages see only a different autograd function; tensor and context parallelism stay refused for the family. tests/cpu/models/test_qwen4_exp_qsa_kernel_dispatch.py holds the kernel's key set to the tiled mask and transformers' dense selection and the fp64 gather reference to the tiled path; tests/gpu/kernels/test_block_sparse_attention.py holds the kernel to that reference, to the tiled path and to its own compile count.

Sliding-window and wide-head layers on SDPA

On CUDA, a model whose run resolves to sdpa and that has sliding-window layers (has_sliding_window_layers: Step-3.7, Gemma 4, and Cohere2 MoE or Laguna's in-library classes on a pinned sdpa) or a head wider than 256 (has_head_dim_beyond_flash; Gemma 4) is built with sdpa_flex_sliding (halo/models/attention/sdpa_flex_sliding.py), which apply_family_patches returns to every loader it patches: the policy loader, the frozen reference/teacher loader, the reward-model scorer, and a pinned-backend embedding SentenceTransformer load or trainer path-string load. Plain sdpa stays where every call would carry what the kernels here are not given: sinks models (GptOss), a config with attn_logit_softcapping (Gemma 2), and a class transformers does not declare backend-compatible (is_backend_compatible(); Inkling, whose every call carries a dense relative bias) or does not map at all (remote code, whose attention need not dispatch through the registry). HALO_FLEX_SLIDING=0 opts out.

  • Masked causal layers (every sliding layer, and a full layer no wider than 256 under a packed or padded mask; a query/key or value head narrower than 16, which Inductor's FlexAttention kernel does not take, stays on SDPA) run compiled FlexAttention under a BlockMask derived from the dense mask transformers built, so causality, the window, packed-document isolation and padding carry over and the tiles outside the window and across documents are skipped. Under SDPA the same layers compute every tile of the dense mask. Each query's allowed keys in such a mask are one contiguous run, so the mask reduces to the first and last key per query (key_intervals, read 256 query rows at a time), and the kernel's mask function compares against those two [Q] tensors: no index over the [Q, K] grid (whose flattened offset outgrows int32 past about 46k tokens) and no copy of the dense mask is kept. A mask that allows some query a non-contiguous set of keys goes to SDPA. The block masks are built once per mask tensor (every sliding layer of a forward, and its gradient-checkpoint recompute, gets the same one) and are freed with it.
  • One compiled shape family. Each batch row runs alone, padded to a multiple of 128, always under a mask and with autocast off inside the kernel. Under autocast the inputs first take the autocast dtype, as SDPA's do, so fp32 parameters under bf16 autocast, whose fp32 rotary embedding promotes the queries and keys beside bf16 values, still run the warmed bf16 graph and the tuned tiles. A call either records a graph with every input requiring grad (an input left frozen, as by a LoRA that adapts q_proj alone, joins as a detached leaf) or runs under no_grad. The kernel therefore compiles twice, a training graph and a forward-only one, for any mix of lengths, batch sizes, grad modes and trained projections. warm_flex_sliding_kernels compiles both for each call the model's attention modules make on FlexAttention (head count, read off each module since Step-3.7's and Laguna's vary by layer, query/key and value widths, window and scaling, which Dynamo guards on), so no rank compiles mid-run, and FlexAttention takes only that call: the module's heads, its query/key and value widths, its scaling, and a mask over exactly the call's queries and keys. Any other call runs SDPA, with a one-time warning. Under tensor parallelism a call carries the rank's share of the heads, which the module does not declare, so a TP run on a pinned sdpa (Cohere2 MoE, or a dense sliding-window model on HF-native TP) keeps SDPA on these layers. Deterministic-algorithms mode is a guard too, and HF's Trainer turns it on after the load under full_determinism, so each rank would compile again at its own first long masked call while its peers wait in the next collective. Where the warm-up ran (world size > 1), every trainer therefore refuses full_determinism at construction (reject_full_determinism_after_warmup); with HALO_FLEX_SLIDING=0 the model runs plain SDPA and takes it. The tuned tiles (head_dim 256, 16-bit inputs, SM100+) do not depend on the length, since a length-dependent choice is itself a recompile guard; other shapes use FlexAttention's defaults. The contract exists because Dynamo guards on a batch of one, a tile-multiple length, grad mode, each input's dtype and requires_grad, the autocast state and whether a block mask carries a mask tensor: a mid-run compile stalls the other ranks in the next collective, and past Dynamo's recompile limit (8) FlexAttention runs unfused over the full score matrix.
  • Short masked calls (a query no longer than the window, or than one 128-token tile) go to SDPA: no tile would be skipped. A full layer over one unpadded document gets no mask and keeps SDPA's causal kernel.
  • Causal global layers run matmul attention where SDPA has only its mem-efficient kernel (head_dim above 256, or the flash backend disabled process-wide, as the Gemma 4 loader does) and the scores saved for backward fit MATMUL_GLOBAL_BUDGET_BYTES (2 GiB). The logits stay fp32, as in mem-efficient SDPA: the score GEMM takes the 16-bit inputs and writes fp32, at 16-bit GEMM speed. At Gemma 4's global shape (bf16, scaling 1.0, RMS-normed 512-wide heads, unit norm weights, 1,024 tokens) the output's error against fp64 is 1.25x mem-efficient SDPA's, and the query and key gradients' error is half of SDPA's; bf16 logits would put the output's at 18x. Under autocast the GEMM takes the autocast dtype, as SDPA does. With no mask the layer attends as SDPA does: causally from the first key, or every key for a single query or an explicit is_causal=False. The budget is per layer: without gradient checkpointing every global layer holds its share. At 4,096 tokens a Gemma 4 26B-A4B global layer saves 1.5 GiB against 0.13 GiB on mem-efficient SDPA, about 7 GiB more across its 5 global layers.
  • Packed global layers past the budget with heads wider than 256 run mem-efficient attention per document. Where every query of the mask attends exactly its own document, from the document's first token up to itself (what transformers builds for a packed row), the documents' boundaries over every batch row (causal_documents, read once per mask tensor) go to the varlen form of the kernel SDPA runs these heads on (_efficient_attention_forward/_backward with cu_seqlens, causal from each document's first query and key), which visits each document's own tiles only. The KV heads are repeated per query group, as SDPA repeats them, and nothing compiles. Any other mask (a padded query, a bidirectional span, a window) goes to SDPA, whose mem-efficient kernel computes every tile of the dense mask.
  • Everything else goes to the implementation registered as sdpa: bidirectional modules, dropout, float masks, output_attentions, and calls carrying sinks (s_aux), a softcap or a position bias.

tests/gpu/kernels/test_flex_masked_layers.py checks a tiny Step-3.7 on packed and padded rows, and a tiny Gemma 4 with 512-wide global layers past the budget on packed rows, against the SDPA model (loss and every gradient, the path counted, a padded row's global layers left on SDPA); at Step-3.7's and Gemma 4's real attention shapes, the output and input gradients against a per-document fp64 reference within 1.5x mem-efficient SDPA's error on the dense mask; packed documents bitwise isolated from each other; and no compile after the warm-up, a call carrying a tensor-parallel rank's share of the heads (left on SDPA) included. tests/cpu/models/test_attention_warmup_geometry.py holds the warm-up to that contract at Dynamo's level on CPU, tracing the kernel on Dynamo's eager backend (Inductor's own guards stay the GPU tests') under error_on_recompile: on tiny Step-3.7, Laguna and Mistral no packed or single-document row, trained or forward-only, traces a graph after the warm-up, and each row runs FlexAttention on exactly the layers it masks (every layer of a packed row, the sliding ones of a single document). tests/cpu/models/test_flex_sliding_masked_layers.py checks the resolver over the tiny roster, that the warm-up's calls are the ones a packed forward makes and each is the call its module declares, that FlexAttention refuses any other call (a rank's share of the heads, another value width or scaling, a mask over other queries or keys), that a packed full layer's block masks hold exactly its tiles, that only a document-causal mask yields documents, and that every tiny model it serves, dense Mistral (a window on the config alone), Cohere2 (NoPE full layers) and OLMo 3 (q/k norm) included, reproduces the SDPA model's loss and every gradient on a packed row and a right-padded batch longer than the window, every layer on uncompiled FlexAttention reading each block mask as the compiled kernel does. tests/gpu/kernels/test_sdpa_flex_sliding.py checks Gemma 4 and Mistral (a window on every layer) against the SDPA model, loss and every gradient, on packed rows longer than the window; rows of ~47.5k tokens against SDPA; that one warm-up leaves no compile for later lengths, batch sizes, masks, grad modes, trained projections and autocast, and none for a Gemma 4 forward and backward, fp32 parameters under bf16 autocast included; that the block masks are freed with their mask; that the global layers' error at Gemma 4's shape stays within 1.5x mem-efficient SDPA's, their saved bytes match the budget's count, their unmasked calls match SDPA, and the score GEMM follows autocast; and that full_determinism is refused after the warm-up.

Source on GitHubUpdated 18 November 2026