Dataset Formats for Training Scripts
Required columns per method. For Async GRPO with Environments see Async GRPO with Environments — prompt plus an answer column only where the environment grades against one; for Distillation, teacher and self-distillation use the SFT format (self-distillation adds a privileged-answer field) and online SDPG uses the GRPO prompt/answer format.
Sources are S3, HuggingFace Hub (including the :config and @split suffixes), or local paths — see
S3 Utilities for resolution rules and caching.
Splits and combining
A dataset must have a train split; loading raises otherwise (with an @split suggestion). With no
test split a validation split is read as the test split; with neither, the toolkit splits off
test_size from train, or falls back to the first 100 samples.
Setting test_size on an already-split dataset (a test split, or a validation read as one)
re-splits it: train and test are concatenated and re-divided, destroying the curated split. Pin the
split you want with @split to keep test_size splitting only that one (openai/gsm8k:main@train).
A sharded or pre-processed dataset carries the split decided at preparation time; test_size there
is ignored with a warning. A pre-processed dataset also fixed its rows, so a dataset_ratio below 1 raises.
Combine multiple datasets by passing lists; dataset_ratio is the per-dataset fraction kept:
YAML
dataset:
- "s3://my-bucket/chat/v1/train"
- "HuggingFaceH4/ultrachat_200k"
dataset_ratio:
- 1.0
- 0.5The pipeline loads each source, filters rows whose conversation field is None or [],
ratio-subsets, normalizes the schema, and concatenates.
A source that ships neither a test nor a validation split contributes training rows only. The
corpus test split comes from the sources that ship one, and only a corpus where none does falls back
to a placeholder cut from its own train rows (set test_size to carve a held-out split from every
source instead).
_normalize_dataset_schema (halo/data/sources/corpus.py) keeps the intersection over train and
test together: a column missing from any dataset, or whose feature type differs across them, is
dropped, with no fixed allowlist.
The declared render columns are the exception. A declared conversation_field must be in every
entry: a source without it raises, naming that source. tools_field may be missing from some entries,
which get a null-filled column of the carrying entry's type, so a mixed corpus keeps its tools. A
declared column the intersection loses anyway (a type mismatch across the entries, which no fill can
bridge) raises rather than rendering the whole corpus without it.
Which rows a ratio keeps is drawn without replacement from numpy.random.default_rng(seed)
(PCG64) — seed 42, +1 for the test split and stepped per list entry, so every rank selects the
same rows with no exchange. Entries are subset independently and concatenated, with no cross-entry
dedup: the same source listed twice is drawn twice, and every row both draws keep lands twice.
The dataset fingerprint records the seed and the resulting sizes, never the row set, so a change of
RNG API, algorithm or per-split seed would swap the training corpus under an unchanged fingerprint.
tests/cpu/data/test_dataset_ratio_seed.py pins the exact selection to keep that a deliberate edit.
Required columns by method
Every message is {"role": "system"|"user"|"assistant", "content": str} (or multimodal content for
VLM, below).
conversation_field must name a column the dataset actually carries: the loader checks the
train/test schemas at load and raises, naming the available columns. Without it a typo silently
no-ops the empty-conversation filter and surfaces much later as a KeyError inside the tokenizer map.
The check follows the script, not the YAML. SFT, both distillation scripts and the three
prompt-rendering GRPO scripts (offline, online, environmental) declare a conversation column, each with
its own default (messages for teacher distillation, prompt elsewhere), so a dataset without it
raises whether or not the YAML names one. Scripts that render no conversation (preference, reward,
classification, embedding) declare none and skip the check.
tools_field is checked the same way for a single dataset. Across a dataset: list it raises only
when no source carries the column and warns per source otherwise — a tool-use corpus concatenated
with plain chat is a legitimate shape, and only the rows of the sources without the column render
toolless. Pre-processed datasets skip both checks; their rows are already tokenized.
SFT
JSON Lines
{"prompt": [{"role": "user", "content": "What is machine learning?"}, {"role": "assistant", "content": "Machine learning is a subset of AI..."}]}Multi-turn: include all turns in the single prompt list.
YAML
conversation_field: "prompt" # conversation column (default "prompt")
system_prompt: null # optional system prompt to prepend
model_supports_system_role: true # if false, merge system into first user message
train_on_completions_only: true # train only on assistant responses
train_on_last_assistant_only: false # train only on the LAST assistant message
assistant_message_template: "<|start_header_id|>assistant<|end_header_id|>\n\n" # completion maskingPipeline: load → filter empties → apply chat template → tokenize (sequences over max_length are dropped) → optional packing. Datasets load with keep_in_memory=False, so Arrow files are memory-mapped, not copied into RAM.
Offline pre-processing (tokenization, packing, sharding) is optional, see SFT Dataset Pre-Processing.
SFT-VLM
Messages carry text and images:
JSON Lines
{"prompt": [{"role": "user", "content": [{"type": "image", "image": "<image_data>"}, {"type": "text", "text": "What's in this image?"}]}, {"role": "assistant", "content": "A sunset over the ocean."}]}The image field accepts: base64 data URI (data:image/png;base64,...), file path, PIL Image, or raw bytes. So do the transformers chat format's spellings of the part: path, base64, and url for a local path, file:// or a data: URI. An http(s) URL is refused, since training fetches nothing. Text-only messages keep the plain "content": str form. A conversation stored as a JSON string parses the same as on the text path.
Hub datasets that store images in a separate column instead train via images_field (the column is injected into the first user turn), and the paired {user, assistant} turn shape used by the HuggingFaceM4 texts column is normalized to role/content messages — see SFT — Vision-language models.
A run takes the VLM data path only when both halves agree (is_vlm_run_across_ranks, halo/data/vlm_detection.py): the checkpoint is multimodal, and the run declares image data. The model class follows the checkpoint alone, so a natively-multimodal checkpoint carrying text-only rows loads its multimodal class but trains through the text pipeline, packing included.
A run declares images three ways, any one of which is enough: images_field names the column; the dataset carries an images, image or pixel_values column, which the map then reads as if named; or the conversation_field column embeds {"type": "image"} content parts. The preference scripts have no conversation_field, so DPO declares by column only.
The embedded-parts declaration is read from the column's Arrow schema: the parts struct is typed over the whole split, so an image payload field (image, path, base64 or url) — or a datasets.Image feature under any name — declares the run no matter which rows carry images. A conversation column stored as JSON strings has no struct to read and falls back to a probe of each split's first row.
The verdict is agreed across ranks (all-reduce MAX — any image-declaring rank wins), because on a pre-sharded corpus every rank probes a disjoint slice: its own first row, and the Arrow schema datasets inferred for its own shard set. A split verdict would pair one arm's barrier against the other's store wait, so the probe spends one collective per call whatever the answer.
The checkpoint half is is_vlm_model (halo/models/introspection/modality.py), judged on the config that probe_checkpoint (halo/model_loading/parallel/model_source.py) reads once, on every rank, at the pinned revision: a model_type registered under AutoModelForImageTextToText whose mapped class does not end in ForCausalLM (mistral4 and fuyu are registered there but decode text), or a config carrying a vision_config. This covers natively-multimodal models (Qwen3.5/3.6, Gemma3/4, Mistral3) with no name list. A config transformers knows decides both ways: a model_type present in CONFIG_MAPPING that matches neither test is text-only, and the name is never consulted.
The name-substring heuristic (-vl, vl-, vl2, llava, vision, pixtral, molmo, idefics, paligemma, cogvlm, minicpm-v, qwen3.5-, qwen3.6-) runs only when the config cannot load (offline, or a raw weights dir without config.json) or its model_type is unregistered (remote code). The hints match mid-word — a path containing revision reads as a VLM — so pass an explicit config where the path is uncontrolled.
For a JSON-string conversation column, the first-row fallback cannot see a mixed dataset whose probed rows are text-only: that run resolves to the text path and then fails loud at the first image row.
Every text renderer — SFT, preference prep (DPO/SMPO), Bradley-Terry reward prep, the GRPO generation-prompt renderer, generation eval, offline GRPO's templater and classification — refuses a conversation carrying an image part (reject_image_content) rather than expanding it into placeholder tokens with no pixels behind them, which would train vision placeholders as text in silence. The escape is an image column on the dataset, or offline --is_vlm preprocessing. The VLM path refuses an image part with no payload, and every path refuses video, audio and image_url parts, which no collator hands a processor. The text renderers refuse any other non-text part too (input_image, input_text): by template it renders as a placeholder with no data behind it, or vanishes with no error. A thinking part passes, as the Mistral Small 4 and Command A+ templates write it as their reasoning block. A text checkpoint refuses image data of any declaration rather than drop it.
Limitations: packing and padding-free training are NOT supported on the VLM path (images cannot be packed). Offline pre-processing uses --is_vlm, which the Qwen and Gemma 4 processors' image rows do not fit, so those train through this runtime path; see SFT pre-processing.
Preference (DPO/SMPO)
DPO and SMPO share the format below; full history goes in prompt.
JSON Lines
{"prompt": [{"role": "user", "content": "Capital of France?"}], "chosen": [{"role": "assistant", "content": "Paris, the largest city in France..."}], "rejected": [{"role": "assistant", "content": "France capital is Paris I think."}]}Hub shape variants normalize to this contract automatically (normalize_preference_row, halo/data/pipeline/preferences.py): a plain-string prompt becomes a user turn; chosen/rejected that repeat the prompt turns (ultrafeedback_binarized, the Tulu-3 mixtures) have that prefix stripped; a missing prompt column (Skywork-Reward style) is extracted as the shared leading span of chosen/rejected.
Completions are rendered as template(prompt + completion) minus the rendered-prompt prefix, so strict chat templates (Qwen3.5) work and prompt + chosen always reconstructs the full conversation exactly.
DPO requires a reference model (DPO → Reference model). Supports EP and TP; CP not supported (
concatenated_forwardneeds full sequences).SMPO is reference-model-free. Supports EP and TP for both modalities, CP for text only.
VLM mode (any multimodal model; the trainer normalizes rows itself, so hub-shape variants work) takes the same rows plus an optional
images/imagecolumn.DataCollatorForVLMSMPOprocesses images at collation, and padding-free is text-only. See SMPO — Vision-language.
Vision DPO/KTO. An images/image column routes to TRL's vision collators, and the rows must ALREADY be contract-shaped (prompt = message list, chosen/rejected = continuation-only) — the hub-shape normalization above runs only on the text path. Modality routing keys on the dataset, so a natively-multimodal model with text-only preference data trains through the normal text pipeline.
Reference-model shapes for vision rows: DPO → Vision-language.
Offline GRPO
JSON Lines
{"prompt": [{"role": "user", "content": "Write a haiku about programming."}], "completions": [[{"role": "assistant", "content": "Code flows like water..."}], [{"role": "assistant", "content": "print hello world"}]], "rewards": [0.95, 0.23]}- Variable group sizes allowed (1 to 1000+ completions per prompt).
- Provide one reward per completion. A length mismatch raises
ValueErrornaming the offending row, rather than silently truncating to the shorter list. - Each group's rewards become group-relative advantages via
advantage_method(defaultquantile_norm).
Config in OfflineGRPOConfig (halo/configs/grpo_offline.py); defaults: Offline GRPO → Configuration.
Full-fine-tune KL runs need a finite, unsharded dataset (Offline GRPO → Reference model).
Classification
Single-label uses a string; multi-label uses a list. The row carries a prompt conversation, or a raw text column named by text_field (e.g. imdb's text), which is wrapped as one user turn:
JSON Lines
{"prompt": [{"role": "user", "content": "Best purchase ever!"}], "label": "positive"}
{"text": "A romantic comedy about time travel.", "label": ["comedy", "romance", "sci-fi"]}Label-set rules, -1 included: Classification → Dataset.
YAML
dataset: "path/to/classification/dataset"
text_field: text # only when rows have no 'prompt' conversation
max_length: 2048CP not supported — needs pooled representations from the complete sequence.
Reward modeling
Same format as preference data. There is no pre-tokenization pass: the rows go straight to
DistributedRewardTrainer (over TRL's RewardTrainer), which chat-templates and tokenizes
chosen/rejected itself and trains the Bradley-Terry loss. Implicit-prompt datasets (shared
leading turns inside chosen/rejected, e.g. Skywork-Reward) therefore work with no prompt
column. Rows over max_length are filtered by the trainer, not truncated. tools_field is aliased onto the tools column the trainer renders.
CP not supported — pooled representations are incompatible with sequence splitting.
Dataset maps
dataset_num_proc sets the map/filter worker processes: Halo's field on the SMPO, classification,
distillation and offline GRPO configs, TRL's own on the configs taken from TRL. Unset, the toolkit's
maps run the toolkit default, not one worker; how that default is derived and how ranks share it:
HALO_DATASET_NUM_PROC.
Trainer-side map/filter callables must be module-level functions taking their state through
fn_kwargs, never bound methods. At num_proc > 1 datasets ships the callable to worker
processes through dill, which pickles a bound method's whole self: the model, and under EP the
DeepEP/NCCL process groups (unpicklable, so the map dies mid-pass).
reject_self_capturing_fn enforces this at coordinated_map / coordinated_filter. Passing
tunables through fn_kwargs also puts them in the cache key, which values read off self never
reach.
At num_proc > 1 both seams pickle the callable once per call, its fn_kwargs bound, and every
worker task carries those bytes. datasets otherwise pickles the callable into each task one after
another in the calling process, whatever the dataset size: about 1.2 s of dill per task for a captured
Qwen3 tokenizer, 3.6 s for Gemma 4. The key reads the bare callable, so every rank names the same cache
file whatever worker count it maps with.
The key (halo/data/pipeline/map_cache_key.py) hashes the callable's source plus the source of every
first-party module-level function it calls by bare name, so editing such a helper re-maps. A helper
reached as a module attribute or a method stays out of the key, and editing it reuses stale rows: call
helpers by bare name from a mapped function, a functools.partial of one, or a builder closure.
Those two helpers carry their own rank ordering and must never be nested in a main-first block — see Filesystem Handling.
Chat templates
YAML
chat_template: "jinja-templates/qwen3/qwen3-instruct.jinja" # file path or inline Jinja string
force_chat_template: true # needed over a template the model ships, or the run raises
pad_token: "<|endoftext|>"
eos_token: "<|im_end|>"
bos_token: null
added_special_tokens:
- "<special1>"
model_supports_system_role: false # merge system message into first user messageBundled templates live under jinja-templates/<family>/: stripped training templates
(*-instruct.jinja single-turn, *-multiturn.jinja/*-chat.jinja multi-turn) plus verbatim
upstream *-native.jinja templates (system messages, tools, thinking) for Qwen3, Qwen3.5/3.6,
GLM and Ling. GPT-OSS has gpt-oss-harmony.jinja, upstream's with two deviations,
and gpt-oss-multiturn.jinja, the SFT form that terminates every assistant turn. Pin a native
template whenever the training render must match serving byte-for-byte.
The template lands on the tokenizer and, for a VLM run, on its processor as well
(adopt_tokenizer_chat_template): a processor renders and saves with a chat_template of its own,
so the processor is what decides both the VLM rows and the exported chat_template.jinja. The same
holds for prepare_dataset.py --is_vlm and patch_vocab.py --chat_template.
Special-token ownership
Rendered chat-template text tokenizes through tokenize_rendered
(halo/data/chat_render.py), which probes each tokenizer once — tokenizing "" with
add_special_tokens=True — to learn what its post-processor actually adds. bos_token alone is
unreliable: gpt-oss and Bailing declare a nominal BOS their post-processor never emits, and Zaya
appends a trailing <|im_end|> instead of prepending BOS.
- A rendered leading BOS is stripped only when the post-processor re-adds it, so specials appear exactly once and no BOS is forced onto a tokenizer that never emits one (gemma-4's template-emitted BOS survives).
- Tokenizer-appended trailing specials stay on training rows, unless the render already ends with the same terminator sequence (Zaya) — then the duplicate is stripped.
- They are always stripped from generation prompts (
for_generation=True), so a prompt never ends with a turn terminator.
This seam covers SFT (runtime and offline), classification, reward-model scoring (encode_for_scoring),
teacher/self-distillation, and generation-eval prompts. The SMPO and offline-GRPO prompt paths use
the same probe to prepend BOS only when the post-processor owns it.
render_generation_prompt — the prompt stage the online/async GRPO scripts share — applies
the same rule to the rendered TEXT it returns, and measures max_prompt_length with the generation
tokenization the row will really carry.
Source on GitHubUpdated 18 November 2026