Padding-Free Flash Attention Collator

Concatenates variable-length sequences into one flattened tensor and emits cu_seq_lens so a varlen Flash Attention kernel skips padding compute. These collators are SFT-only: select_data_collator (halo/data/collators/factory.py) picks them when padding_free: true. Mutually exclusive with packing. Incompatible with Context and Pipeline Parallelism.

SMPO also supports padding_free (not under PP), but flattens the batch itself in _forward_padding_free (halo/trainers/preference/smpo.py) rather than using these collators. Padded rows pay full GEMM cost; the M-dimension argument is in GPU Training Theory §2.

SMPO's flattened row isolates its documents too: attention through position_ids, and LFM-2, GatedDeltaNet and Nemotron-H's Mamba through seq_idx / cu_seq_lens segment markers built by the same gate these collators use (halo/models/segment_markers.py).

Implementations live in halo/data/collators/packing.py:

  • DataCollatorWithFlattening — padding-free, no completion masking.
  • DataCollatorWithFlatteningAndCompletionMask — padding-free + completion-only masking (train only on assistant responses).

Configuration

padding_free, packing, packing_strategy and eval_packing come from TRL's SFTConfig; the completion-masking fields (train_on_completions_only, assistant_message_template) from SFTScriptArguments:

YAML

padding_free: true
attn_implementation: "flash_attention_2"   # a varlen FA backend (FA2, FA3 or FA4)
packing: false                              # mutually exclusive with padding_free
train_on_completions_only: true
assistant_message_template: "<|im_start|>assistant\n"

select_data_collator gates on the model's resolved _attn_implementation, not on a family list. padding_free is rejected outside flash_attention_2 / _3 / _4: the cu_seq_lens it exists to emit go unread there, so it buys nothing at all. The SFT script runs the same gate (reject_dense_attention_rows) before the model load, against the backend the loader will resolve for the checkpoint.

That catches every model the loader routes off Flash Attention: DeepSeek-V4 (eager-only), Gemma 4 (head_dim=512), the families whose upstream class declares no flash support (Ling 3.0, GLM-5 Next, Step-3.7 Flash, Inkling, Qwen3.8-Flash-Next), PaliGemma and PaliGemma 2 (prefix LMs, whose token_type_ids mask no flash kernel reads), and, outside SFT, a pinned flash_attention_4 on a head_dim-256 model under full_determinism, which falls back to SDPA (SFT's multi-document rows load FA2 there, pinned or not). The full map is in Flash Attention. A softcapped model (Gemma 2) runs both on FA2 on Blackwell, since a softcap stays off FA4's varlen kernels; a pinned flash_attention_4 the load keeps (Gemma 2 27B) is refused there.

Use packing or padded batches there: packing raises tokens per row on any kernel, so off a varlen backend it only warns. gpt-oss, the Bailing family (Ling 2.0, Ring, Ling 3.0) and GLM-5 Next are the exceptions. Their forwards never pass the packed position_ids into mask construction, so the row would run as one dense causal sequence with documents attending across each other, and packing off a varlen backend is rejected there. DeepSeek-V4 rejects packing too, on every backend: its compressors cut KV windows at row indices, so a later packed document reads the row's first windows. A model whose linear-attention layers no segment marker reaches (Ring's Lightning Attention-2) refuses packing and padding_free on every backend, flash included, unless the run sets allow_cross_document_linear_attention: true; Ling 3.0 (no flash class, and its SDPA mask attends across documents) and GLM-5 Next's flat rows stay refused either way (Document isolation).

Both collators emit position_ids that reset per document; whether a family's forward turns those into an isolating mask is per-family, and the exceptions are tabulated in Document isolation under packing. Where it does, the only difference is cost: a non-varlen backend materializes a dense [L, L] mask instead of consuming cu_seq_lens.

Both collators flatten the mini-batch into a single row — padding-free by construction, packing via flatten_packed_batch — so L is the whole batch's token count either way (the summed real tokens for padding-free, up to per_device_train_batch_size × max_length for packing), and at equal tokens per step the two masks are the same size.

Packing's mask is bounded by max_length only at per_device_train_batch_size: 1, so keep the batch at 1 there and scale with gradient_accumulation_steps.

The collator emits input_ids, labels, and position_ids (reset per sequence) shaped [1, total_tokens], plus the varlen kwargs cu_seq_lens_q/cu_seq_lens_k (1-D int32, length num_seqs + 1) and max_length_q/max_length_k (Python ints). The varlen FA kernel reads cu_seq_lens to attend only within each sequence. seq_idx joins them under return_seq_idx, which select_data_collator sets for the families whose conv, Mamba or linear-attention mixers read it (LFM-2, GatedDeltaNet, Nemotron-H).

Benchmark results

Throughput is real (non-padding) tokens/s/GPU — attention_mask.sum(), not padded element count — set via include_num_input_tokens_seen="non_padding" so all three modes share one real-token basis.

Model: Qwen3-30B-A3B (128 experts, top_k=8), FA2. Hardware: 2× B300 (SM103), EP=2, Blackwell image, Halo 1.1.0 or earlier. Data: max_length=4096, avg ≈ 1024 tokens (~75% padding waste). Setup: batch_size=2/GPU, GC on, 8 steps / 3 warmup.

Modetokens/s/GPU (real)Step timevs Standard
Standard9222.08s1.0×
Packing13,3410.59s14.5×
Padding-Free4,1780.46s4.5×

Peak memory is 117–120 GiB across all three (weights + optimizer states dominate; packing's is ~3 GiB above the other two). At ~75% padding waste, packing wins by filling each max_length block with ~4 real sequences and cutting the step count. Padding-free strips the padding via varlen FA, so it processes only real tokens per step at lower memory than packing, while keeping per-sample boundaries (no cross-sequence attention).

The table uses --attn_implementation flash_attention_2. FA4 is also valid: FA4 + packing + EP is safe (warm_attention_kernels compiles the FA4 kernels behind a barrier at load — see Flash Attention). With --attn_implementation unset (the default), the benchmark auto-detects FA4 on Blackwell (tests/common/benchmark_args.py).

Parallelism compatibility

CollatorEPTPCPPP
DataCollatorWithFlattening (+ …AndCompletionMask)YesYesNoNo
DataCollatorWithPacking (+ DataCollatorForCompletionOnlyLMWithPacking)YesYesNoYes
DataCollatorForCausalLMWithPaddingYesYesYesYes

CP requires fixed-length sequences for collective synchronization: padding-free (variable-length output) and packing (the Ulysses CP attention path has no per-document boundaries, so packed documents would attend across each other) are both rejected by select_data_collator when use_context_parallel=True — use the standard padded collator with pad_to_multiple_of=cp_size (Context Parallelism).

PP rejects padding-free: its flattened width varies every step, while the pipeline's point-to-point buffers freeze after the first. Packing runs under PP at one packed row per microbatch, each stage rebuilding the segment markers its conv / linear-attention layers read, except where such layers read no boundary at all: GLM-5 Next and Inkling in the PP roster (PP packing).

When to use each

  • avg > 80% of max_length — any collator (all within ~1%); use standard.
  • avg << max_length — packing (14.5× at ~75% waste; packing's own cross-sequence padding overhead is 1–5%).
  • Need CP — standard padding only; both packing and padding-free are rejected.
  • Want to skip padding FLOPS without cross-sequence boundaries — padding-free.

Running benchmarks

Bash

# All 3 collator modes (variable-length, ~75% padding waste) — table above
torchrun --nproc_per_node=2 \
    tests/gpu/profiling/benchmark_collators.py --model qwen3-30b-a3b --ep 2 --seq 4096 \
    --batch_size 2 --attn_implementation flash_attention_2

# Higher padding waste (avg 10% of max_length)
torchrun --nproc_per_node=2 \
    tests/gpu/profiling/benchmark_collators.py --model qwen3-30b-a3b --ep 2 --seq 4096 --avg_ratio 0.1

# Single mode
torchrun --nproc_per_node=2 \
    tests/gpu/profiling/benchmark_collators.py --model qwen3-30b-a3b --ep 2 --mode packing

# Fixed-length comparison (collators equivalent)
torchrun --nproc_per_node=2 \
    tests/gpu/profiling/benchmark_sft_ep.py --model qwen3-30b-a3b --ep 2 --seq 16384

Source on GitHubUpdated 18 November 2026