SFT Dataset Pre-Processing

scripts/before_training/prepare_dataset.py converts raw conversation data into tokenized, optionally packed and sharded datasets ready for training. The SFT script scripts/training/sft.py (text or VLM, auto-detected) detects a pre-processed dataset via its metadata.json and skips tokenization, cutting startup latency on large runs.

Only SFT (text and VLM) supports offline pre-processing. DPO, SMPO, GRPO, and Classification use different data formats and tokenize during training. The script is single-process — run it with plain python, not torchrun; --num_proc sets HF map workers, not ranks. Its flags are the PreprocessingConfig fields (halo/data/preprocessed/metadata.py) plus the tool's own PrepareDatasetArguments, spelled like the training YAML keys; a YAML file of them works as the first argument, with --key=value overrides after it.

Pipeline: load (S3 / HuggingFace Hub / local) → tokenize (tokenizer or VLM processor) → pack (text SFT only, optional) → shard (Megatron-LM style, optional) → save (S3 / Hub / local).

The knobs baked into the rows are recorded in metadata.json and re-checked against the training config at startup. A disagreement raises when the training YAML states the knob — a runtime value cannot re-render already-baked rows. A knob left at its own default only warns: that config claims nothing about the artifact, and the prepared value is what trained. The rendered-template check below still applies.

Two groups are compared. The chat-render knobs are --conversation_field, --system_prompt, --model_supports_system_role false, --tools_field, --interleaved_thinking, --images_field, and, under completion masking, --assistant_message_template. The tokenizer mutations — --pad_token, --eos_token, --bos_token, --chat_template — change the ids themselves: an EOS override moves the completion-mask boundaries, and a template override re-renders every turn. --chat_template is recorded as resolved template text, so a path and the text it names compare equal.

The template the rows were rendered with is recorded too (rendered_chat_template, the processor's under --is_vlm), and training raises when the tokenizer or processor it ends with, whose template the checkpoint exports, carries another. Preparation applies --chat_template whatever template the tokenizer ships, while training applies chat_template over a shipped one only with force_chat_template: true and raises on a differing one without it, so a --chat_template bake trains with the same chat_template plus that flag. A stamp without rendered_chat_template is held to its recorded --chat_template outside --is_vlm, and to nothing otherwise.

The checked set is every PreprocessingConfig field outside _UNCOMPARED_FIELDS, so a new knob is compared by default, and a knob the metadata does not record warns instead. --mode text artifacts render no template and skip the render check entirely. A training config setting packing: true against an unpacked artifact warns — preprocessed rows are never packed at runtime. A model_name_or_path other than the prep --model_name_or_path only warns, unless the template the run exports differs from the rows' (which raises, see above), so train with the tokenizer the rows were baked with.

--mode is validated on the config, not only by the CLI's choices: an unvalidated typo like txt falls through every mode == "text" branch and chat-templates a raw pretraining corpus. A knob belonging to the other mode set to a non-default value (--system_prompt with --mode text, --text_field with --mode chat) is rejected rather than silently ignored; _KNOB_MODES names the mode that reads each mode-specific knob. PreprocessingConfig also refuses --pack_sequences with --is_vlm and an unknown --packing_strategy, so every refusal lands before the input is fetched.

Over-length rows are dropped, not truncated in chat and --is_vlm mode. --mode text instead truncates each document to --max_length (reserving the last slot for the appended EOS), unless --pack_sequences is set.

If dropping empties a split the script raises and names the usual cause (a wrong --conversation_field, or an --assistant_message_template that does not match the rendered assistant turn) rather than --max_length. The same two causes are caught again after the label bake: a probe of the first and last 64 rows raises when every label is -100, rather than shipping a dataset that trains zero tokens.

--tokenizer_backend gigatoken (optional extra halo[gigatoken]) encodes text→ids with the gigatoken Rust encoder — ~6× faster tokenization on UltraChat 200K with Qwen3-0.6B — while chat templating and decoding stay on the model's tokenizer (halo/model_loading/tokenizer_backend.py). Token IDs are verified against the HF tokenizer at startup; any divergence raises. VLM mode swaps the processor's inner tokenizer the same way, but image processing dominates there, so the win is small.

The same two values are also a training config field, tokenizer_backend (default hf). It resolves in configure_run_tokenizer, so every training method that tokenizes rows honors it with no preparation pass; embedding training rejects a non-hf value, since SentenceTransformer owns its own tokenization.

With --train_on_completions_only the completion-only labels are baked into the saved dataset under the span policy the artifact's own runtime collator applies (mask_batch_to_completion_spans): marker tokens trained, each turn's terminator bounded by the next marker start.

That is PADDED_SPAN_POLICY (no end-of-sequence fallback), and PACKED_SPAN_POLICY under --pack_sequences, where a turn with no terminator anywhere ends at the sequence end. Preprocessed and runtime training therefore see identical loss masks, image tokens included (masked after the span refill on both paths).

Baked labels are authoritative at training time. The SFT script disables runtime completion re-masking for preprocessed data: a re-mask would overwrite them, e.g. re-masking a packed chunk whose response marker landed in the previous chunk drops its trained tokens.

So train_on_completions_only in the training config cannot change them; it is validated instead, and a value disagreeing with the baked labels raises at startup. The collator only pads the stored labels (VLM uses PreprocessedVLMDataCollator).

Offline-packed data (--pack_sequences) still needs per-document position IDs at collation, so the SFT script selects the packing collator from metadata.packed — the collator choice follows the data, not the config flag; TRL-side packing stays off either way.

Usage

Bash

python scripts/before_training/prepare_dataset.py \
    --input "s3://bucket/raw/my_dataset" \
    --output "s3://bucket/preprocessed/my_dataset" \
    --model_name_or_path "Qwen/Qwen3-8B" \
    --max_length 8192 \
    --test_size 0.01 \
    --num_shards 64 \
    --pack_sequences \
    --conversation_field prompt \
    --assistant_message_template $'<|im_start|>assistant\n'

--conversation_field and --train_on_completions_only default to the training-side values (prompt, and masking on), so a dataset prepared with defaults is one the training defaults accept. Masking on means --assistant_message_template is required — pass the marker your model's chat template renders, or --train_on_completions_only false for a full-sequence loss. A full-sequence artifact stores no labels: its rows are the ones the runtime path collates, and the collator derives the labels from input_ids as it does there (a pad-valued token is masked, the turn-ending EOS kept where pad == eos).

--input accepts an S3 URI, a Hub ID (org/name, with optional :config and @split suffixes, e.g. HuggingFaceH4/ultrachat_200k@train_sft), or a local path. --output accepts an S3 URI, hf://org/name, or a local path (add --private for a private Hub dataset). Only the explicit hf:// prefix uploads to the Hub — unlike --input, a bare org/name output is a local directory, so a relative path such as preprocessed/my_dataset is never published. Read it back as ./preprocessed/my_dataset or an absolute path: a Hub-shaped source that names a local directory is refused as ambiguous (S3 Utilities → Dataset path formats).

The output holds two splits, train and test. A multi-split input must carry train; with no test split its validation split is baked as test. Any other split (validation next to test, an unsupervised split) is left out, with a warning that names it. To prepare one of those, point --input at it with an @split suffix.

VLM mode (--is_vlm; packing not supported, and the Qwen and Gemma 4 processors take the runtime VLM path instead) stores input_ids with vision placeholders expanded, attention_mask, labels (image tokens masked to -100), pixel_values (float16 bytes), pixel_values_shape, and the patch grid (image_grid_thw, or image_grid_hw on Nemotron-H).

A row carrying a video or audio part raises rather than drops, since no collator feeds one and a dropped row would leave an artifact that looks whole.

Two combinations raise and point at the runtime VLM path: --is_vlm with --tools_field or --interleaved_thinking (the offline render would diverge from the runtime one), and a processor emitting a vision key outside this fixed schema on an image row (mm_token_type_ids and the like — the model needs it at forward). The Qwen2.5-VL, Qwen3-VL, Qwen3.5/3.6, Qwen3.8-27B, Qwen3.8-Flash-Next and Gemma 4 processors all emit it, so their image rows train through the runtime VLM path and --is_vlm bakes only their text rows.

VLM sequences are never truncated, since cutting expanded placeholders while pixel_values keeps every patch desyncs text from vision. Over-length rows are dropped here instead — the runtime VLM collator raises on one, because a live batch cannot drop rows without desyncing DP ranks. --min_pixels/--max_pixels are part of the map cache key, so re-running at a new resolution does not reuse stale pixels.

Images may live inside the conversation's content parts or in a separate column, the shape hub VLM datasets ship (FineVision / the_cauldron / Docmatix). --images_field <column> merges that column into the messages exactly as the runtime VLM path does, so both routes bake the same rows.

The merge fills the conversation's image placeholders in order, else prepends the images to the first user turn. It requires --is_vlm; the text tokenization merges nothing.

An image column (images / image / pixel_values) that no --images_field names is refused. Every source column is dropped at tokenization, so it would otherwise be discarded in silence and the artifact would carry is_vlm: true over rows holding no pixels, which training reads as multimodal.

Parameters

ParameterDefaultDescription
--inputrequiredInput: S3 URI, Hub ID, or local path
--outputrequiredOutput: S3 URI, hf://org/name, or local path
--model_name_or_pathrequiredModel name/path for the tokenizer; completion-only masking also reads its config.json for the turn-terminator ids, so it must be the model checkpoint, not a bare tokenizer
--max_length8192Maximum sequence length
--trust_remote_code / --no-trust_remote_codeFalseExecute the checkpoint's own tokenizer/processor and config code when loading them — off, since --model_name_or_path may be a Hub repo; pass it for a remote-code family (Bailing/Ling, Laguna) you trust (Scripts)
--modechatchat = apply chat template to --conversation_field (SFT); text = raw-text causal-LM for (continued) pre-training (tokenize --text_field, append EOS per document). See Pre-training.
--text_fieldtext(mode=text) column holding raw text
--append_eosTrue(mode=text) the per-document EOS that preserves document boundaries; false skips it
--conversation_fieldpromptConversation field name (mode=chat). Matches the training-side default
--system_promptNoneSystem prompt to prepend
--tools_fieldNoneDataset field holding a tool list (list of dicts or JSON string), forwarded into apply_chat_template — must match the tools_field used at training time. Rejected with --is_vlm
--interleaved_thinkingoffKeep thinking blocks in non-final assistant turns (clear_thinking=False), matching the live path's interleaved_thinking. Rejected with --is_vlm
--model_supports_system_roleTruefalse merges the system prompt into the first user message (models without a system role)
--train_on_completions_onlyTrue in --mode chat, False in --mode textMask user turns; train only on assistant responses. The chat default matches the training side and requires --assistant_message_template; asking for masking under --mode text is rejected (raw documents have no assistant turns)
--assistant_message_templateNoneTemplate marking start of assistant response
--pack_sequencesFalsePack multiple sequences (text SFT only). Refused where training would refuse the rows: compressed-KV layers (DeepSeek-V4), a family with no flash backend whose dense forward leaks (GLM-5 Next), and sequence mixers no boundary reaches (Ring, Ling 3.0, Inkling) unless --allow_cross_document_linear_attention
--allow_cross_document_linear_attentionFalseNeeds --pack_sequences. Bakes rows for a model whose mixers receive no document boundary; training them takes the same opt-in
--packing_strategybfdNeeds --pack_sequences. TRL strategy: bfd (best-fit-decreasing, drops overflow), bfd_split (carries overflow into later examples) or wrapped (concatenate-and-chunk, cutting a document at the row edge; seq_lengths keeps the documents of a row apart)
--is_vlmFalseVLM mode (stores pixel_values and their patch grid, image_grid_thw or image_grid_hw; refuses an image row whose processor emits another vision key)
--images_fieldNone(--is_vlm) column holding the row's image(s) for datasets that keep them outside the conversation; merged into the messages like the runtime path. An image column named by nothing is refused
--min_pixels / --max_pixelsNoneVLM image pixel bounds; refused without --is_vlm and on a processor that does not resize by pixel count (Gemma 4)
--num_shards11 writes an unsharded dataset that trains at any data-parallel size. Above 1 see Sharded loading
--num_procHALO_DATASET_NUM_PROCProcesses for dataset map. With neither set, every CPU the process may use but two, since the tool runs alone (HALO_DATASET_NUM_PROC)
--tokenizer_backendhfhf = the model's tokenizer; gigatoken = Rust bulk encoder (see above)
--test_sizeNoneTest split fraction (e.g. 0.01), cut from train. Refused when the input already carries a test/validation split — point --input at the train split alone. Without it (and without a test or validation split in the input) the output is train-only: an unsharded one trains with the first 100 train rows warned in as a placeholder test split, and --num_shards above 1 is refused before tokenization, since training rejects a sharded dataset with no test split
--pad_token / --eos_token / --bos_tokenNoneOverride pad / EOS / BOS token. Recorded in metadata.json and re-checked at training time
--chat_templateNoneOverride chat template (Jinja2 string, or a path to a .jinja/.jinja2/.j2 file), applied whatever template the tokenizer ships. The resolved text is recorded and re-checked; training applies it over a shipped template only with force_chat_template: true
--overwriteFalseReplace an existing output. Only the local writer keeps the previous dataset intact through a failed publish (see below)
--dry_runFalsePrint the planned output paths and exit before the tokenizer or input dataset load — it validates nothing about the data
--privateFalseMake Hub dataset private. The hf:// upload and every load read the ambient credentials (HF_TOKEN or the stored login)
--verboseFalseLog at DEBUG

Output structure

--num_shards 1 (the default) writes a DatasetDict.save_to_disk directory with no shard index; every rank loads it whole and the DataLoader splits the rows by data-parallel rank:

Text

s3://bucket/preprocessed/dataset/
├── metadata.json              # Preprocessing config and stats
├── dataset_dict.json
├── train/                     # Arrow files + dataset_info.json + state.json
└── test/                      # only when the input has a test/validation split or --test_size is set

--num_shards > 1 writes per-split shards and a shard index:

Text

s3://bucket/preprocessed/dataset/
├── metadata.json              # Preprocessing config and stats
├── train/
│   ├── shard_index.json       # Shard mapping for distributed loading
│   ├── shard_0000/            # Arrow dataset shards
│   └── ...
└── test/
    ├── shard_index.json
    └── shard_0000/

metadata.json records the preprocessing config and stats; the authoritative field list is the PreprocessedDatasetMetadata dataclass in halo/data/preprocessed/metadata.py.

Its version stamp is compared on load: a stamp this build does not read raises IncompatiblePreprocessedDataset naming the version, rather than being swallowed into a silent fall-back to the raw (re-tokenizing) path. Unknown fields at a stamp this build does read raise the same way (a diverged build). A metadata.json carrying no preprocessed key is somebody else's file; the dataset is treated as raw with a warning.

One rank per input-filesystem scope runs the detection probe, so its answer is the scope's. Only a confirmed absence reads as raw: no file, an S3 404, a Hub repo without it (offline, a cache without it). A stamp that does not parse or a Hub it cannot reach raises on every rank instead. The one tolerated failure is an S3 outage with no local mirror of the stamp, read as raw with a warning: an absence is never mirrored, and a warm-cache raw dataset loads without S3.

Each split's shard_index.json (ShardIndex, halo/data/preprocessed/shard_index.py) carries the same version stamp and is held to it the same way: a stamp this build does not read, or a field it does not know, raises the same IncompatiblePreprocessedDataset (defined in that leaf, so a loader can refuse without importing the writer).

The index decides which shards each rank loads, so a diverged one read with this build's field meanings hands ranks the wrong rows.

An index listing zero shards for train raises too, instead of handing every rank an empty split; the same on a non-train split warns, since a tiny eval split that does not reach every rank is legitimate.

metadata.json is also the completion marker — is_preprocessed_dataset keys on it — so it is written last and a publish never leaves it beside an incomplete payload. A local output is baked once, straight into a sibling of the destination (<output>.halo-staging, on its filesystem), and swapped in by rename, leaving the previous dataset intact if the bake fails; an existing output without --overwrite is refused before the bake. An S3 or Hub output is baked into a TMPDIR temp directory and uploaded from it.

Each tokenize, filter, label and pack step writes a full copy of the corpus. The tool writes them to a scratch directory under HF_DATASETS_CACHE, removed when the bake ends, so peak disk there is those copies and nothing stays behind. The directory is named after --output, so a rerun to the same output first removes the copies a killed bake left there. Completion-only labels are stored at the integer width of input_ids (int32).

A map's callable is pickled once per map, not per worker task (_PickledOnceCallable, halo/data/pipeline/coordinated_map.py), so a 190-worker bake of HuggingFaceH4/ultrachat_200k takes 69 s as chat rows and 50 s as packed text.

The pack runs on the --num_proc workers too. TRL packs each 1000-row batch on its own, so every worker packs whole batches and the artifact holds the same rows at any worker count. Packed rows store no attention_mask (the packing collator builds its own and drops it), so a packed full-sequence text artifact holds input_ids and seq_lengths alone, about 4 bytes per token.

Hub and S3 --overwrite both delete the marker first and re-upload it after the payload, so an interrupted upload reads as raw (a loud failure) rather than as a silently truncated preprocessed dataset. The S3 writer never clears the prefix up front: it overwrites objects in place and sweeps the ones the new tree does not carry only after the upload completed.

Training does not read a Hub-published dataset in place: load_dataset on a saved tree returns its sidecar files as rows, so a preprocessed dataset named by its Hub id (org/name) is refused at startup. Download it first (hf download org/name --repo-type dataset --local-dir <dir>) and point dataset: at the directory.

Sharded loading

Point the training config at the dataset (dataset: "s3://my-bucket/preprocessed/my_dataset"); each data-parallel rank then loads only its own shards. How ranks split them, the num_shards >= data_parallel_size requirement and what a starved rank does: Distributed Data Loading.

Size --num_shards to the largest DP degree you train at; a multiple of it balances evenly. Example: 64 GPUs across 8 nodes (DP 64, no CP/TP) → --num_shards 64.

Source on GitHubUpdated 18 November 2026