Collators

Collators turn tokenized samples into batches: padding, sequence packing, completion-only label masking, and per-sequence position IDs. They live in halo/data/collators/. factory.py selects among the SFT collators of padded.py and packing.py after collator_rules.py refuses the option combinations none can serve; all of them share the span resolver and label rule in halo/data/completion_spans.py and the turn terminators in halo/data/turn_markers.py.

The per-method leaves (smpo.py, vlm.py, self_distill.py, classification.py, offline_grpo.py, vlm_preference.py) are built by their own trainers and scripts. On a prefix-LM model (PaliGemma) the VLM collators of vlm.py and self_distill.py write token_type_ids from the labels, so every trained token sits in the causal suffix, refuse a chat template that renders <bos> (the processor puts one after each row's last image) and never train the newline the processor appends to each row (SFT → Vision-language models). Every VLM collator hands the processor one image list per row (run_vlm_processor): Gemma 3 and Gemma 4 processors read a flat list as one row's images and refuse a batch of several rows. fixed_shape.py wraps any of them for a consumer whose buffer shapes freeze on the first batch.

CollatorPackingPadding-FreeCompletion MaskingPosition IDs
DataCollatorForCompletionOnlyLMNoNoYesNo
DataCollatorWithPackingYesNoNoreset per sequence
DataCollatorForCompletionOnlyLMWithPackingYesNoYesreset per sequence
DataCollatorWithFlatteningNoYesNoreset per sequence
DataCollatorWithFlatteningAndCompletionMaskNoYesYesreset per sequence
DataCollatorForCausalLMWithPaddingNoNoNoNo
DataCollatorForLanguageModelingWithVarlenNoNoNoNo

DataCollatorForCausalLMWithPadding is the SFT CP-mode padded route; it preserves precomputed labels (the parent DataCollatorForLanguageModeling rebuilds them from input_ids and crashes on ragged ones). DataCollatorForLanguageModelingWithVarlen is TRL's own SFT collator handing a one-document batch its varlen set (One-document batches).

You rarely construct these directly — select_data_collator() picks one from your YAML config.

Collator classes

Constructor notes that matter when you build one directly:

  • response_prompt_template (str or List[int], the assistant-header template) is required for DataCollatorForCompletionOnlyLM, DataCollatorForCompletionOnlyLMWithPacking and DataCollatorWithFlatteningAndCompletionMask; the flattening one refuses None at construction. An empty template raises — it would match every position and train on all tokens.
  • tokenizer is required for every collator except DataCollatorWithFlattening, which defaults it to None and never uses it.
  • pad_to_multiple_of is set to cp_size under CP; the masked-label fill value is -100 throughout.

DataCollatorForCompletionOnlyLM — masks loss on everything except assistant completion tokens (user/system tokens get ignore_index). Extends DataCollatorForLanguageModeling. If response_prompt_template is not found in a sequence, that sequence is dropped from the loss.

DataCollatorWithPacking — packs multiple sequences per sample. Position IDs reset at each boundary; labels at position_ids == 0 are set to -100 so the model is not penalized for predicting the next document's first token across a boundary.

It drops the dense attention_mask (packed rows store none; coordinated_pack leaves it out) so Flash Attention runs its varlen kernel over per-document cu_seq_lens, handed over by the collator (below) or derived from the resetting position IDs, and flattens the packed mini-batch to a single [1, total_tokens] row of real tokens only — transformers' packed-sequence detection only engages at batch size 1, so a [B>1, L] batch would silently fall back to dense-causal attention.

On a flash run (the resolved backend select_data_collator reads) it also hands attention that varlen set on every multi-document row: cu_seq_lens_q/cu_seq_lens_k and host-int max_length_q/max_length_k, the values transformers would derive from the position IDs. No attention layer then re-derives them — a packed-row check, a nonzero and a max length read back, each a device sync, per layer — and a gradient-checkpointing recompute, reentrant or not, re-runs attention on the same set. A one-document row follows One-document batches. PP's kept rows carry none: the schedule splits only batch-dim tensors, so each pipeline stage builds the set, and the segment markers its conv / linear-attention layers read, per microbatch instead (PP packing).

On packed Qwen3-8B SFT under FA4 and FSDP2 (8× B300, s4096, 1.2.0, five rounds) the set takes a step from 310 host syncs to 24 and runs 9.5% faster (13,141 → 14,391 tok/s/GPU), step-1 and step-2 loss identical.

Inter-row padding is dropped in the flatten. A pad carries position 0, so every kept pad would be its own varlen segment, and the FA4 backward pays a fixed per-segment cost.

Where pads must survive (a fixed-shape batch), the tail's position IDs are a ramp restarting every PAD_TAIL_SEGMENT_CHUNK (256) tokens, keeping it a handful of no-op segments.

Whether the packed documents actually stay isolated is per family, not universal; see Document isolation under packing.

Where the attention path is isolated, the remaining difference between backends is cost. The non-varlen path builds a dense [L, L] mask over the flattened row, so L is the whole batch's token count (up to per_device_train_batch_size × max_length) and the mask exhausts memory at long context. select_data_collator warns there, telling you to keep per_device_train_batch_size at 1.

It detects packed data via the seq_lengths field: TRL's pack_dataset() emits it for bfd and bfd_split, and coordinated_pack adds it to wrapped rows, one entry per piece of a source document, cut at document ends and at the row edge and summing to the row length. A document cut by the row edge is two pieces in two rows, each isolated from its row's other documents. The first token of a piece has no earlier token in its row to be predicted from, so its label is masked like any document's first token, a continuation's included; under completion-only masking a continuation piece holds no assistant marker and trains nothing. A continuation also restarts its position_ids at 0, so it sees none of the prefix; under completion-only masking prefer bfd_split over wrapped. Rows without seq_lengths (a corpus baked by an earlier release) collate as one document, so concatenated documents attend across each other; the collator warns once per instance when it sees such rows.

Online packing (packing: true at training time) and the offline prepare_dataset.py pipeline both accept all three strategies (see Pre-Processing).

DataCollatorForCompletionOnlyLMWithPacking — packing plus completion masking; masks each packed sub-sequence independently, falls back to standard completion masking for non-packed input.

DataCollatorWithTreePacking — packed rows of prefix-sharing trees (tree_packing: true). Built by the SFT script, not the factory; see Tree Packing.

DataCollatorWithFlattening — padding-free. Flattens the batch into one [1, total_tokens] sequence and emits Flash Attention varlen kwargs. Output keys: input_ids, labels, position_ids (reset per sequence), cu_seq_lens_q/cu_seq_lens_k, max_length_q/max_length_k. The first token of every sub-sequence is label-masked (the causal shift cannot predict it from the previous document).

DataCollatorWithFlatteningAndCompletionMask — padding-free plus completion masking; the lowest-overhead collator for variable-length SFT trained on assistant responses only.

One-document batches

Without the set, an unpacked batch-1 row costs every flash attention layer a packed-row check (_is_packed_sequence, a device sync), again in each checkpoint recompute, before transformers runs its dense kernel: two per layer and step under gradient checkpointing. Where select_data_collator gets emit_varlen_one_document=True (the SFT script passes it), a batch that is one unpadded document (an unpacked batch-1 row, a one-document packed row, a wrapped row) carries the set too: cu_seq_lens_q/k [0, S] and host-int max_length_q/k S, what transformers derives from its positions (single_row_varlen_kwargs in halo/models/segment_markers.py). Attention then runs the varlen kernel over one segment and no layer checks the row. The set is all the collator adds: every other tensor of the batch is the one the route builds without it. With neither packing nor completion masking the factory returns DataCollatorForLanguageModelingWithVarlen, TRL's own SFT collator handing the set, built with the pad_token_id and pad_to_multiple_of TRL builds its default with; without the opt-in it returns None there and TRL builds its own.

The set is handed on FA2, FA3 and FA4 only, and not to a model whose GatedDeltaNet layers read it too (Qwen3.5/3.6, Qwen3-Next: the delta rule would take its varlen path) or whose attention FA4's varlen kernels refuse (fa4_varlen_refusal: Gemma 2's softcap, whose one-document rows keep FA4's dense kernel); supports_one_document_varlen_set in collator_rules.py is the rule. Never under CP (Ulysses passes its own cu_seqlens) or PP (each stage builds the set of its packed microbatches), and not for teacher distillation, whose batch also reaches the teacher. A batch of several rows or a padded one keeps transformers' mask path, and a multi-document packed row the set of its documents. A remote-code forward declaring no **kwargs raises on the set at the first step, as it does on a packed row's.

Offline GRPO

OfflineGRPODataCollatorWithPadding and, under CP, OfflineGRPOCPDataCollatorWithPadding are built by the trainer, not this factory: Offline GRPO.

Factory: select_data_collator()

halo/data/collators/factory.py picks the collator from config flags and validates compatibility:

Python

from halo.data.collators.factory import select_data_collator

collator = select_data_collator(
    tokenizer=tokenizer,
    padding_free=True,
    train_on_completions_only=True,
    assistant_message_template="<|im_start|>assistant\n",
)

Flags, keyword-only: train_on_completions_only (required: the run's own setting), padding_free=False, packing=False, assistant_message_template=None, pad_to_multiple_of=None, use_context_parallel=False, train_on_last_assistant_only=False, model_config=None, per_device_train_batch_size=1, keeps_packed_rows=False, allow_cross_document_linear_attention=False, per_device_eval_batch_size=0, emit_varlen_one_document=False (One-document batches). With packing above batch 1 the collator merges the mini-batch's packed rows into one row (document isolation and throughput unchanged), and the factory warns: raise max_length instead when a longer pack is the intent. keeps_packed_rows suppresses that warning; only the PP seam sets it.

Pass model_config. It is what feeds resolve_eos_token_ids(tokenizer, model_config); without it the collator falls back to the tokenizer's EOS alone and mis-masks any family whose turn terminators live in config.eos_token_id (GLM-4). The SFT script passes it. train_on_last_assistant_only=True requires train_on_completions_only=True and raises otherwise.

ConfigurationCollator
CP + packingrejected — the Ulysses CP attention path has no per-document boundaries, so packed documents would attend across each other
PP + padding_freerejected — the flattened width varies per step while the pipeline's P2P buffers are frozen after the first
PP + packing on a model whose conv / linear-attention layers read no boundary at all (KDA, Lightning Attention-2, Inkling's short convolutions)rejected, whatever allow_cross_document_linear_attention says — no marker reaches those layers, so they cross the kept rows' documents; the segment markers GatedDeltaNet and LFM-2 ShortConv read each stage rebuilds (PP packing)
CP + completion maskingDataCollatorForCompletionOnlyLM (CP mode)
CPDataCollatorForCausalLMWithPadding (a label-preserving DataCollatorForLanguageModeling subclass in padded.py)
padding_free + completion maskingDataCollatorWithFlatteningAndCompletionMask
padding_freeDataCollatorWithFlattening
packing + completion maskingDataCollatorForCompletionOnlyLMWithPacking
packingDataCollatorWithPacking
completion masking onlyDataCollatorForCompletionOnlyLM
noneDataCollatorForLanguageModelingWithVarlen where one-document batches take the set, else None (TRL builds its own)

YAML fields (packing, packing_strategy, padding_free, eval_packing come from TRL's SFTConfig; train_on_completions_only and assistant_message_template from ConversationRenderArguments, train_on_last_assistant_only from SFTScriptArguments, both in halo/configs/sft.py):

YAML

padding_free: false
packing: true
train_on_completions_only: true
assistant_message_template: "<|start_header_id|>assistant<|end_header_id|>\n\n"
train_on_last_assistant_only: false

The SFT script defaults train_on_completions_only: true; assistant_message_template has no default and must be the tokenizer's exact assistant-header encoding (the ChatML form above for Qwen; see Constraints) — a missing or non-rendering marker raises at startup.

For a pre-processed dataset the SFT script derives the flags from the data, not the config: packing follows metadata.packed (offline-packed chunks still need the packing collator's per-document position IDs), and completion masking is disabled at runtime because the baked labels are authoritative — see SFT Dataset Pre-Processing.

After building its collator a script calls disable_trl_dataset_prep (halo/entrypoints/data_setup.py): it sets dataset_kwargs: {skip_prepare_dataset: true} and clears packing and padding_free on the TRL config, since TRL reads both again at collator selection and would swap its own collator in over rows it never tokenized. A custom script must call it too; a user dataset_kwargs or dataset_text_field is refused at startup (reject_trl_dataset_prep_args).

Document isolation under packing

Packing only pays off if document B's logits do not move when document A's content changes. That drift is measured per family (0 = isolated) and the verdict is not universal: the collator emits correct position_ids everywhere, but a family's forward has to carry them into its mask, and a non-attention mixer carries state along the sequence regardless of any mask.

The on-policy GRPO trainers' packed micro-batches (packed_microbatch_tokens) flatten several rows into one varlen call under these same rules, applied with no opt-in: a family whose rows cross documents (the rows below marked refused, and the families allow_cross_document_linear_attention would admit for SFT), compressed-KV layers, and attention that reads a token's row index are refused at construction, since a crossing RL step corrupts the policy gradient without a trace (Packed micro-batches).

FamilyAttention layersOther layers
Qwen3 (dense + MoE), GLM-4 MoE Lite, Laguna, Gemma 4, and dense Llama, Qwen2, Mistral, Mixtral, OLMo3, Cohere2, Gemma 1, 2 and 3, and PaliGemmaisolated on their supported backends—
Inklingisolatedfour depthwise causal convs per layer cross by construction — the modeling reads seq_idx but its call sites never forward kwargs, so no emission can reach them; packing and padding_free are refused unless allow_cross_document_linear_attention
Mistral4isolated — flash needs patch_mistral4_flash_packed_position_ids (upstream drops position_ids before the attention interface); dense clean to grouped-GEMM reduction noise—
GPT-OSSisolated on flash only; eager/SDPA/flex leak — the model's mask kwargs omit position_ids, so the packed row runs as one dense causal sequence—
DeepSeek-V4 (eager-only)isolated on the training path (use_cache=False; a live cache suppresses the packed mask)the CSA/HCA compressors cut KV windows at row indices and judge their causality by position, so every document after the first attends the row's first windows — packing and padding_free are refused
Qwen3.5 / 3.6, Qwen3-NextisolatedGatedDeltaNet reads seq_idx (conv) + cu_seq_lens_q (delta rule): the collators and SMPO's padding-free forward emit both for the family, so its boundaries reach the kernels — but only the fla / causal-conv1d fast paths consume them (both installed in the images). The torch fallbacks take neither, so a multi-document row would cross in the conv and the scan — the factory refuses packing / padding_free, and SMPO its padding_free, unless transformers' own fast-path predicates hold
Qwen3.8-Flash-Nextisolated — its sparse (QSA) layers run Halo's tiled kernel per seq_idx document, and a packed row without seq_idx is refused (Qwen3.8-Flash-Next)GatedDeltaNet as Qwen3.5; the per-layer embedding's n-gram history and dilated conv restart at every seq_idx document start
Bailing / LingLing 2.0, Ring: isolated on flash only (patch_remote_llama_flash_template); eager/SDPA leak — the remote file builds its mask from attention_mask alone (Bailing/Ling). Ling 3.0: flash refused, MLA on SDPA leaks the same way, so Ling 3.0 cannot packLing 3.0 KDA and Ring's Lightning-Attention-2 linear layers — cross. The KDA op accepts a cu_seqlens kwarg, but the model forward never threads kwargs down to it, so there is no reachable boundary parameter — the same class of crossing as Inkling's convs
GLM-5 Nextboth layer types receive the 2D padding mask create_recurrent_attention_mask builds, so no packed-boundary parameter reaches the DSA attention either — flat packing crosses, so it is refused; tree packing isolates it (the DSA forward reads the plan's path links)KDA linear attention — crosses under flat packing, the same unreachable-boundary class as Bailing/Ling; isolated under tree packing (conv and scan segmented by the plan)
LFM-2isolatedShortConv is isolated exactly: the collators and SMPO's padding-free forward emit seq_idx for this family and both conv paths honor it
Zayaisolated — flash needs patch_zaya_packed_document_isolationCCA convolution + delayed-value recurrence — isolated exactly by the same patch, which restarts both at every document start (Zaya)
Cohere2 MoEisolated on every backend, no patch needed — the forward feeds position_ids into mask construction and through layer kwargs to the attention interface—
Nemotron-Hisolated on FA4, FA2 and SDPAthe Mamba conv and scan are isolated exactly: the collators and SMPO's padding-free forward emit seq_idx for the family, and the toolkit's scan runs each document from a zero state (Nemotron-H)
Step-3.7 Flashisolated on SDPA and eager, no patch needed — the forward feeds position_ids into both mask constructions (full and sliding); bit-exact through dense layers, MoE layers add expert-summation reduction noise (~1e-7 fp32). Training path only: a live cache suppresses the packed mask (use_cache=False, as DeepSeek-V4)—

These families are refused outright in select_data_collator. The checks read the text sub-config too, so a composite (VLM) wrapper is covered:

  • GPT-OSS, Bailing / Ling (Ling 2.0, Ring, Ling 3.0) and GLM-5 Next, dense_packing_leaks on their MoEFamilySpec, read by has_dense_packing_leak in halo/models/segment_markers.py: packing on a non-varlen backend raises (reject_dense_attention_rows, which the SFT script also runs before the model load, against the backend the loader will resolve). Use a flash backend or turn packing off; Ling 3.0 and GLM-5 Next have no flash backend.
  • DeepSeek-V4, or any model whose layer_types declare a compressed-KV layer (the "compressed" kind in halo/models/attention/layout.py: compressed_sparse_attention, heavily_compressed_attention): packing and padding_free raise on every backend (reject_compressed_kv_rows). Positions restart per document but the compressor windows are cut at row indices, so a later document reads another document's compressed KV as its own context. Train with padded batches. The SFT script runs the same check before the model load, prepare_dataset.py before an offline --pack_sequences pass, and the SFT trainer for a packing / padding-free collator handed to it directly (reject_multi_document_collator, keyed on the same two collator roots as the CP gate), as does SMPO for its padding_free forward.

Which families get which segment markers, and the kernel refusal below, live in halo/models/segment_markers.py, shared by the collators and SMPO's padding-free forward. LFM-2's ShortConv is matched by family; GatedDeltaNet by the layers a config declares, under any model_type: linear_attention in layer_types on a config carrying GatedDeltaNet's linear_num_key_heads / linear_num_value_heads split. Kimi Delta Attention spells its layers linear_attention too but declares a single linear_num_heads, and gets no markers.

A model with GatedDeltaNet layers carries one more refusal, about markers that would be emitted but not read. transformers selects their segment-aware linear-attention kernels at modeling-import time via is_causal_conv1d_available() / is_flash_linear_attention_available(), which require the package installed, fla >= 0.2.2, and a CUDA-capable torch.

The torch fallbacks it takes otherwise drop seq_idx and cu_seq_lens_q, so conv and recurrent state cross document boundaries while attention stays isolated, invisible in the loss. SFT's packing and padding_free and SMPO's padding_free are refused unless those same predicates hold, so the refusal cannot disagree with the kernels actually selected (the production images satisfy them).

"Crosses by construction" applies where the mixer has no per-document boundary parameter at all (Inkling's convs). Such a family's rows are refused as unmarked linear attention's are, below, and the same opt-in admits them: their short convs carry each document into the next, so packed rows train on mixed context.

A linear-attention layer no segment marker reaches is refused unless the run opts in: find_unmarked_sequence_mixer (halo/models/segment_markers.py) reads hybrid layer groups (layer_group_size above 1: Ring's Lightning Attention-2, Ling 3.0's KDA) or linear_attention layer types (GLM-5 Next's KDA) outside the families that read cu_seq_lens, and the sequence mixer a family spec names in unmarked_sequence_mixer (Inkling's short convolutions). SFT's packing / padding_free (before the model load and again at collator selection) and SMPO's padding_free raise on every backend through reject_unmarked_sequence_mixer_rows, flash included, where the softmax layers isolate and the loss shows nothing. So does the SFT trainer for a packing / padding-free collator handed to it directly (reject_multi_document_collator), under the opt-in the factory records on the collators it builds; a tree collator is left to its plan. allow_cross_document_linear_attention: true (SFTScriptArguments, SMPOConfig) trains such rows anyway, with a warning: those layers' recurrent state then carries from each document of a row into the next. It admits no GLM-5 Next row: the family runs only dense backends, where its DSA layers read only the padding mask (table above), so the SFT script (before the model load) and the collator factory refuse its flat packing before the opt-in is read, and its padding_free needs a varlen kernel it does not have. Tree packing skips this check and refuses the opt-in: its plan segments the linear-attention layers of every family it admits (Tree Packing).

Distillation takes no packed rows: the teacher script passes packing=False and self-distillation refuses packing and padding_free.

A family outside the table above and the family specs gets a warning on every backend (packing_isolation_is_measured, halo/models/segment_markers.py): BERT- and BART-family decoders, OPT and BioGPT leak on dense backends, and GPT-Neo and Moshi on flash too.

Constraints

  • packing and padding_free are mutually exclusive. Packing pads concatenated sequences to max_length; padding-free flattens to one variable-length tensor. Pick packing for short sequences, padding-free for long/variable ones.

  • Both want a varlen Flash Attention backend. The gate is the model's resolved _attn_implementation against VARLEN_ATTN_IMPLEMENTATIONS (flash_attention_2/3/4, halo/models/attention/selection.py), not a model-family list.

    Off it, padding_free raises (its cu_seq_lens kwargs have no consumer) and packing warns (it pays a dense [L, L] mask per layer). A family whose dense path leaks documents makes packing raise too (Document isolation), and DeepSeek-V4 (eager-only: head_dim 512 exceeds FA's 256 cap) refuses both. Gemma 4 lands on the warning; its shipped config packs at batch 1. See Padding-Free Collator.

  • A softcapped model (Gemma 2) does not train on FA4's varlen call, whose kernels are not validated with a softcap. Packed, padding-free and padded rows (more than one per batch: transformers unpads any batch holding a padding token) all reach it (rows_reach_varlen_kernel). Left unset, attn_implementation takes FA2 there; a pinned flash_attention_4 the load keeps (Gemma 2 27B; the resolver moves a head_dim-256 pin to FA2) is refused before the load and again on the resolved backend by select_data_collator (reject_fa4_varlen_training). See Flash Attention.

  • Padding-free is incompatible with Context Parallelism, and packing is rejected under CP. Under CP use the padded collator, which sets pad_to_multiple_of=cp_size.

  • Completion masking requires assistant_message_template matching the tokenizer's exact assistant-header encoding; unmatched sequences are dropped from the loss.

    A marker the chat template never renders is refused by select_data_collator, the self-distillation collators and the VLM collator: it would mask every label and the run would train zero tokens at loss ≈ 0. The text probe renders both a plain and a thinking assistant turn, so a marker gpt-oss harmony emits only under thinking passes it and is caught per batch instead.

    select_data_collator and the VLM collators also refuse assistant turns that end in no turn terminator (reject_untrainable_completions): they render two exchanges through the chat template and mask them under the route's span policy (packed and padding-free rows run the packed policy, the rest the padded one), so a template borrowed from another family (a Gemma template on PaliGemma, whose turns close with <end_of_turn> while its eos is <eos>) is refused until eos_token names the token it closes a turn with. A template that ends a turn at the next role marker (GLM's) passes through the model config's terminator set. The self-distillation policy runs a terminator-less span to the row end, so it passes there. The packed and padding-free routes still accept a terminator-less template (their policy trains the final turn), and the offline bake and tree packing do not go through this probe, so the refusal holds for padded rows.

EOS, pad, and the response template

The turn terminator is the resolved terminator set, not just tokenizer.eos_token_id. A turn ends at the first token in the union of the model config's eos_token_id (scalar or list; text_config.eos_token_id for VLM/composite configs) and the tokenizer's EOS (resolve_eos_token_ids, halo/data/turn_markers.py). This is required for templates that delimit turns with role markers instead of a per-turn <|endoftext|> — GLM-4 ends an assistant turn with <|user|> or <|observation|> and lists both in config.eos_token_id.

A distinct pad token is deliberately not a terminator: on a right-padded row whose turn lacks a real terminator, the first trailing pad would close the span and train the model to predict pad. Such a turn falls to the warn-and-mask path instead.

assistant_message_template should be the assistant role token alone when the model prefixes a thinking marker (GLM's <|assistant|></think> vs <|assistant|><think>…, Qwen3's <think> blocks). The full template "<|assistant|></think>" matches only the non-thinking render and silently drops every thinking turn from the loss; the role token marks every assistant turn and yields the identical span for non-thinking turns.

The completion-only collators locate each turn at its role token and unmask through to that turn's first terminator, bounded by the next response start. A truncated turn therefore yields an empty span rather than borrowing a later turn's terminator.

The packed and padding-free routes add a global end-of-sequence fallback, where the last position substitutes only when the sequence has no terminator at all. The padded route has no fallback: on a padded row the last position is pad, so a terminator-less turn is masked out entirely.

One span resolver, three named policies. Every completion mask locates its spans with resolve_completion_spans in halo/data/completion_spans.py, under one of three declared policies: PADDED_SPAN_POLICY (the padded route, described above), PACKED_SPAN_POLICY (the same plus the global end-of-sequence fallback the packed and padding-free routes take) and SELF_DISTILL_SPAN_POLICY (the span starts after the marker and runs to the first terminator anywhere after it, falling back per span to the row end).

All three enter it through one preamble, resolve_spans_or_warn: it applies the policy and — when the marker never matched, or every span closed at -1 — warns that the sequence trains zero tokens and returns the no-span verdict, a fully-masked row. One label rule then refills the spans, build_span_labels: it copies each span from input_ids or from labels already masked, and optionally restores each span's closing token from input_ids (the packed route's EOS rescue).

The padded collator, the offline label bake (prepare_dataset.py) and the self-distill / VLM label builder (build_training_labels) share one batch masker on top of it, mask_batch_to_completion_spans. It additionally takes an optional attention_mask confining the span search to a row's real tokens.

It also takes extra_ignore_token_ids (image tokens on the VLM path), masked after the span refill: the refill copies from input_ids, so masking them first hands every extra id inside a completion span back as a trainable target.

The packed and padding-free collators mask one sequence at a time instead, copying spans out of labels so a document boundary keeps its own masking. The bake takes its policy from the artifact it writes — PACKED_SPAN_POLICY when that artifact is packed — so a preprocessed dataset and the same YAML on a raw dataset train the same tokens for a turn whose terminator is missing.

When pad_token_id == eos_token_id (GLM-4 uses <|endoftext|> for both) completion masking is unaffected: the completion-only collators detect turn boundaries in input_ids, not labels. The full-label collators (DataCollatorWithPacking, DataCollatorForCausalLMWithPadding) restore the real EOS from input_ids at every real position after the parent collator's value-based masking (collate_preserving_precomputed_labels); offline-precomputed labels pass through untouched — their baked -100 spans are authoritative.

Source on GitHubUpdated 18 November 2026