# Dataset Formats for Training Scripts

Required columns per method. For Async GRPO with Environments see [Async GRPO with Environments](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo.md) — `prompt` plus an `answer` column only where the environment grades against one; for [Distillation](https://whitecircle.com/halo/docs/agent-docs/training-methods/distillation.md), teacher and self-distillation use the SFT format (self-distillation adds a privileged-answer field) and online SDPG uses the GRPO prompt/answer format.

Sources are S3, HuggingFace Hub (including the `:config` and `@split` suffixes), or local paths — see
[S3 Utilities](https://whitecircle.com/halo/docs/agent-docs/data/s3-utilities.md) for resolution rules and caching.

## Splits and combining

A dataset must have a `train` split; loading raises otherwise (with an `@split` suggestion). With no
`test` split a `validation` split is read as the test split; with neither, the toolkit splits off
`test_size` from train, or falls back to the first 100 samples.

Setting `test_size` on an **already-split** dataset (a `test` split, or a `validation` read as one)
re-splits it: train and test are concatenated and re-divided, destroying the curated split. Pin the
split you want with `@split` to keep `test_size` splitting only that one (`openai/gsm8k:main@train`).

A sharded or pre-processed dataset carries the split decided at preparation time; `test_size` there
is ignored with a warning. A pre-processed dataset also fixed its rows, so a `dataset_ratio` below 1 raises.

Combine multiple datasets by passing lists; `dataset_ratio` is the per-dataset fraction kept:

```yaml
dataset:
  - "s3://my-bucket/chat/v1/train"
  - "HuggingFaceH4/ultrachat_200k"
dataset_ratio:
  - 1.0
  - 0.5
```

The pipeline loads each source, filters rows whose conversation field is `None` or `[]`,
ratio-subsets, normalizes the schema, and concatenates.

A source that ships neither a `test` nor a `validation` split contributes training rows only. The
corpus test split comes from the sources that ship one, and only a corpus where none does falls back
to a placeholder cut from its own train rows (set `test_size` to carve a held-out split from every
source instead).

`_normalize_dataset_schema` (`halo/data/sources/corpus.py`) keeps the intersection over train and
test together: a column missing from any dataset, or whose feature type differs across them, is
dropped, with no fixed allowlist.

The **declared** render columns are the exception. A declared `conversation_field` must be in every
entry: a source without it raises, naming that source. `tools_field` may be missing from some entries,
which get a null-filled column of the carrying entry's type, so a mixed corpus keeps its tools. A
declared column the intersection loses anyway (a type mismatch across the entries, which no fill can
bridge) raises rather than rendering the whole corpus without it.

**Which rows a ratio keeps** is drawn without replacement from `numpy.random.default_rng(seed)`
(PCG64) — seed `42`, `+1` for the `test` split and stepped per list entry, so every rank selects the
same rows with no exchange. Entries are subset independently and concatenated, with no cross-entry
dedup: the same source listed twice is drawn twice, and every row both draws keep lands twice.

The dataset fingerprint records the seed and the resulting sizes, never the row set, so a change of
RNG API, algorithm or per-split seed would swap the training corpus under an unchanged fingerprint.
`tests/cpu/data/test_dataset_ratio_seed.py` pins the exact selection to keep that a deliberate edit.

> [!WARNING]
> **A list disables sharded and pre-processed loading**
>
> Every entry of a `dataset:` list is loaded **fully replicated** on every rank — passing DP
> rank/size would double-shard a sharded entry — and the pre-processed-dataset probe only runs for
> a single string path. Point `dataset:` at one path to use a pre-processed or sharded dataset.

## Required columns by method

| Method | Script | Columns | Types | Config field |
|--------|--------|---------|-------|--------------|
| SFT | `scripts/training/sft.py` | `prompt` | `List[Dict]` | `conversation_field` (default `"prompt"`) |
| SFT-VLM | `scripts/training/sft.py` | `prompt` | `List[Dict]` (multimodal) | `conversation_field` |
| DPO | `scripts/training/preference/dpo.py` | `chosen`, `rejected` (+ optional `prompt`) | all `List[Dict]` | hardcoded |
| SMPO | `scripts/training/preference/smpo.py` | `chosen`, `rejected` (+ optional `prompt`) | all `List[Dict]` | hardcoded |
| KTO | `scripts/training/preference/kto.py` | `prompt`, `completion`, `label` | `List[Dict]`/`str`, bool | `completion_field`, `label_field` |
| Offline GRPO | `scripts/training/grpo/offline.py` | `prompt`, `completions`, `rewards` | `List[Dict]`, `List[List[Dict]]`, `List[float]` | hardcoded |
| Classification | `scripts/training/classification.py` | `prompt` or `text_field`, `label` | `List[Dict]` or `str`, `str`/`List[str]` | `text_field` (label hardcoded) |
| Reward Model | `scripts/training/reward/reward_model.py` | `chosen`, `rejected` (+ optional `prompt`, `images`) | all `List[Dict]` | `images_field` |

Every message is `{"role": "system"|"user"|"assistant", "content": str}` (or multimodal content for
VLM, below).

`conversation_field` must name a column the dataset actually carries: the loader checks the
`train`/`test` schemas at load and raises, naming the available columns. Without it a typo silently
no-ops the empty-conversation filter and surfaces much later as a `KeyError` inside the tokenizer map.

The check follows the *script*, not the YAML. SFT, both distillation scripts and the three
prompt-rendering GRPO scripts (offline, online, environmental) declare a conversation column, each with
its own default (`messages` for teacher distillation, `prompt` elsewhere), so a dataset without it
raises whether or not the YAML names one. Scripts that render no conversation (preference, reward,
classification, embedding) declare none and skip the check.

`tools_field` is checked the same way for a single dataset. Across a `dataset:` list it raises only
when **no** source carries the column and warns per source otherwise — a tool-use corpus concatenated
with plain chat is a legitimate shape, and only the rows of the sources without the column render
toolless. Pre-processed datasets skip both checks; their rows are already tokenized.

## SFT

```jsonl
{"prompt": [{"role": "user", "content": "What is machine learning?"}, {"role": "assistant", "content": "Machine learning is a subset of AI..."}]}
```

Multi-turn: include all turns in the single `prompt` list.

```yaml
conversation_field: "prompt"             # conversation column (default "prompt")
system_prompt: null                      # optional system prompt to prepend
model_supports_system_role: true         # if false, merge system into first user message
train_on_completions_only: true          # train only on assistant responses
train_on_last_assistant_only: false      # train only on the LAST assistant message
assistant_message_template: "<|start_header_id|>assistant<|end_header_id|>\n\n"  # completion masking
```

Pipeline: load → filter empties → apply chat template → tokenize (sequences over `max_length` are dropped) → optional packing. Datasets load with `keep_in_memory=False`, so Arrow files are memory-mapped, not copied into RAM.

Offline pre-processing (tokenization, packing, sharding) is optional, see [SFT Dataset Pre-Processing](https://whitecircle.com/halo/docs/agent-docs/data/dataset-preparation.md).

## SFT-VLM

Messages carry text and images:

```jsonl
{"prompt": [{"role": "user", "content": [{"type": "image", "image": "<image_data>"}, {"type": "text", "text": "What's in this image?"}]}, {"role": "assistant", "content": "A sunset over the ocean."}]}
```

The `image` field accepts: base64 data URI (`data:image/png;base64,...`), file path, PIL Image, or raw bytes. So do the transformers chat format's spellings of the part: `path`, `base64`, and `url` for a local path, `file://` or a `data:` URI. An `http(s)` URL is refused, since training fetches nothing. Text-only messages keep the plain `"content": str` form. A conversation stored as a JSON string parses the same as on the text path.

Hub datasets that store images in a separate column instead train via `images_field` (the column is injected into the first user turn), and the paired `{user, assistant}` turn shape used by the HuggingFaceM4 `texts` column is normalized to role/content messages — see [SFT — Vision-language models](https://whitecircle.com/halo/docs/agent-docs/training-methods/sft.md#vision-language-models).

A run takes the VLM data path only when **both** halves agree (`is_vlm_run_across_ranks`, `halo/data/vlm_detection.py`): the checkpoint is multimodal, and the run declares image data. The model class follows the checkpoint alone, so a natively-multimodal checkpoint carrying text-only rows loads its multimodal class but trains through the text pipeline, packing included.

A run declares images three ways, any one of which is enough: `images_field` names the column; the dataset carries an `images`, `image` or `pixel_values` column, which the map then reads as if named; or the `conversation_field` column embeds `{"type": "image"}` content parts. The preference scripts have no `conversation_field`, so DPO declares by column only.

The embedded-parts declaration is read from the column's **Arrow schema**: the parts struct is typed over the whole split, so an image payload field (`image`, `path`, `base64` or `url`) — or a `datasets.Image` feature under any name — declares the run no matter which rows carry images. A conversation column stored as JSON strings has no struct to read and falls back to a probe of each split's first row.

The verdict is **agreed across ranks** (all-reduce MAX — any image-declaring rank wins), because on a pre-sharded corpus every rank probes a disjoint slice: its own first row, and the Arrow schema `datasets` inferred for its own shard set. A split verdict would pair one arm's barrier against the other's store wait, so the probe spends one collective per call whatever the answer.

The checkpoint half is `is_vlm_model` (`halo/models/introspection/modality.py`), judged on the config that `probe_checkpoint` (`halo/model_loading/parallel/model_source.py`) reads once, on every rank, at the pinned revision: a `model_type` registered under `AutoModelForImageTextToText` **whose mapped class does not end in `ForCausalLM`** (mistral4 and fuyu are registered there but decode text), or a config carrying a `vision_config`. This covers natively-multimodal models (Qwen3.5/3.6, Gemma3/4, Mistral3) with no name list. A config transformers **knows** decides both ways: a `model_type` present in `CONFIG_MAPPING` that matches neither test is text-only, and the name is never consulted.

The name-substring heuristic (`-vl`, `vl-`, `vl2`, `llava`, `vision`, `pixtral`, `molmo`, `idefics`, `paligemma`, `cogvlm`, `minicpm-v`, `qwen3.5-`, `qwen3.6-`) runs only when the config cannot load (offline, or a raw weights dir without `config.json`) or its `model_type` is unregistered (remote code). The hints match mid-word — a path containing `revision` reads as a VLM — so pass an explicit config where the path is uncontrolled.

For a JSON-string conversation column, the first-row fallback cannot see a mixed dataset whose probed rows are text-only: that run resolves to the text path and then **fails loud at the first image row**.

Every text renderer — SFT, preference prep (DPO/SMPO), Bradley-Terry reward prep, the GRPO generation-prompt renderer, generation eval, offline GRPO's templater and classification — refuses a conversation carrying an image part (`reject_image_content`) rather than expanding it into placeholder tokens with no pixels behind them, which would train vision placeholders as text in silence. The escape is an image column on the dataset, or offline `--is_vlm` preprocessing. The VLM path refuses an image part with no payload, and every path refuses video, audio and `image_url` parts, which no collator hands a processor. The text renderers refuse any other non-text part too (`input_image`, `input_text`): by template it renders as a placeholder with no data behind it, or vanishes with no error. A `thinking` part passes, as the Mistral Small 4 and Command A+ templates write it as their reasoning block. A text checkpoint refuses image data of any declaration rather than drop it.

Limitations: packing and padding-free training are **NOT supported** on the VLM path (images cannot be packed). Offline pre-processing uses `--is_vlm`, which the Qwen and Gemma 4 processors' image rows do not fit, so those train through this runtime path; see [SFT pre-processing](https://whitecircle.com/halo/docs/agent-docs/data/dataset-preparation.md).

## Preference (DPO/SMPO)

DPO and SMPO share the format below; full history goes in `prompt`.

```jsonl
{"prompt": [{"role": "user", "content": "Capital of France?"}], "chosen": [{"role": "assistant", "content": "Paris, the largest city in France..."}], "rejected": [{"role": "assistant", "content": "France capital is Paris I think."}]}
```

Hub shape variants normalize to this contract automatically (`normalize_preference_row`, `halo/data/pipeline/preferences.py`): a plain-string `prompt` becomes a user turn; chosen/rejected that repeat the prompt turns (ultrafeedback_binarized, the Tulu-3 mixtures) have that prefix stripped; a missing `prompt` column (Skywork-Reward style) is extracted as the shared leading span of chosen/rejected.

Completions are rendered as `template(prompt + completion)` minus the rendered-prompt prefix, so strict chat templates (Qwen3.5) work and `prompt + chosen` always reconstructs the full conversation exactly.

- **DPO** requires a reference model ([DPO → Reference model](https://whitecircle.com/halo/docs/agent-docs/training-methods/preference/dpo.md#reference-model)). Supports EP and TP; **CP not supported** (`concatenated_forward` needs full sequences).

- **SMPO** is reference-model-free. Supports EP and TP for both modalities, CP for text only.

    VLM mode (any multimodal model; the trainer normalizes rows itself, so hub-shape variants work)
    takes the same rows plus an optional `images`/`image` column. `DataCollatorForVLMSMPO` processes
    images at collation, and padding-free is text-only. See
    [SMPO — Vision-language](https://whitecircle.com/halo/docs/agent-docs/training-methods/preference/smpo.md#vision-language).

**Vision DPO/KTO.** An `images`/`image` column routes to TRL's vision collators, and the rows must ALREADY be contract-shaped (prompt = message list, chosen/rejected = continuation-only) — the hub-shape normalization above runs only on the text path. Modality routing keys on the **dataset**, so a natively-multimodal model with text-only preference data trains through the normal text pipeline.

Reference-model shapes for vision rows: [DPO → Vision-language](https://whitecircle.com/halo/docs/agent-docs/training-methods/preference/dpo.md#vision-language).

## Offline GRPO

```jsonl
{"prompt": [{"role": "user", "content": "Write a haiku about programming."}], "completions": [[{"role": "assistant", "content": "Code flows like water..."}], [{"role": "assistant", "content": "print hello world"}]], "rewards": [0.95, 0.23]}
```

- Variable group sizes allowed (1 to 1000+ completions per prompt).
- Provide one reward per completion. A length mismatch raises `ValueError` naming the offending row, rather than silently truncating to the shorter list.
- Each group's rewards become group-relative advantages via `advantage_method` (default `quantile_norm`).

Config in `OfflineGRPOConfig` (`halo/configs/grpo_offline.py`); defaults: [Offline GRPO → Configuration](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/offline.md#configuration).

Full-fine-tune KL runs need a finite, unsharded dataset ([Offline GRPO → Reference model](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/offline.md#reference-model)).

## Classification

Single-label uses a string; multi-label uses a list. The row carries a `prompt` conversation, or a raw text column named by `text_field` (e.g. imdb's `text`), which is wrapped as one user turn:

```jsonl
{"prompt": [{"role": "user", "content": "Best purchase ever!"}], "label": "positive"}
{"text": "A romantic comedy about time travel.", "label": ["comedy", "romance", "sci-fi"]}
```

Label-set rules, `-1` included: [Classification → Dataset](https://whitecircle.com/halo/docs/agent-docs/training-methods/classification.md#dataset).

```yaml
dataset: "path/to/classification/dataset"
text_field: text   # only when rows have no 'prompt' conversation
max_length: 2048
```

**CP not supported** — needs pooled representations from the complete sequence.

## Reward modeling

Same format as preference data. There is **no** pre-tokenization pass: the rows go straight to
`DistributedRewardTrainer` (over TRL's `RewardTrainer`), which chat-templates and tokenizes
`chosen`/`rejected` itself and trains the Bradley-Terry loss. Implicit-prompt datasets (shared
leading turns inside `chosen`/`rejected`, e.g. Skywork-Reward) therefore work with no `prompt`
column. Rows over `max_length` are filtered by the trainer, not truncated. `tools_field` is aliased onto the `tools` column the trainer renders.

**CP not supported** — pooled representations are incompatible with sequence splitting.

## Dataset maps

`dataset_num_proc` sets the map/filter worker processes: Halo's field on the SMPO, classification,
distillation and offline GRPO configs, TRL's own on the configs taken from TRL. Unset, the toolkit's
maps run the toolkit default, not one worker; how that default is derived and how ranks share it:
[`HALO_DATASET_NUM_PROC`](https://whitecircle.com/halo/docs/agent-docs/reference/configuration-reference.md#environment-variables).

Trainer-side map/filter callables must be **module-level functions** taking their state through
`fn_kwargs`, never bound methods. At `num_proc > 1` `datasets` ships the callable to worker
processes through dill, which pickles a bound method's whole `self`: the model, and under EP the
DeepEP/NCCL process groups (unpicklable, so the map dies mid-pass).

`reject_self_capturing_fn` enforces this at `coordinated_map` / `coordinated_filter`. Passing
tunables through `fn_kwargs` also puts them in the cache key, which values read off `self` never
reach.

At `num_proc > 1` both seams pickle the callable once per call, its `fn_kwargs` bound, and every
worker task carries those bytes. `datasets` otherwise pickles the callable into each task one after
another in the calling process, whatever the dataset size: about 1.2 s of dill per task for a captured
Qwen3 tokenizer, 3.6 s for Gemma 4. The key reads the bare callable, so every rank names the same cache
file whatever worker count it maps with.

The key (`halo/data/pipeline/map_cache_key.py`) hashes the callable's source plus the source of every
first-party module-level function it calls by bare name, so editing such a helper re-maps. A helper
reached as a module attribute or a method stays out of the key, and editing it reuses stale rows: call
helpers by bare name from a mapped function, a `functools.partial` of one, or a builder closure.

Those two helpers carry their own rank ordering and must never be nested in a main-first block — see
[Filesystem Handling](https://whitecircle.com/halo/docs/agent-docs/data/filesystem-handling.md#coordination-primitives).

## Chat templates

```yaml
chat_template: "jinja-templates/qwen3/qwen3-instruct.jinja"  # file path or inline Jinja string
force_chat_template: true                                 # needed over a template the model ships, or the run raises

pad_token: "<|endoftext|>"
eos_token: "<|im_end|>"
bos_token: null
added_special_tokens:
  - "<special1>"

model_supports_system_role: false   # merge system message into first user message
```

Bundled templates live under `jinja-templates/<family>/`: stripped training templates
(`*-instruct.jinja` single-turn, `*-multiturn.jinja`/`*-chat.jinja` multi-turn) plus verbatim
upstream `*-native.jinja` templates (system messages, tools, thinking) for Qwen3, Qwen3.5/3.6,
GLM and Ling. GPT-OSS has `gpt-oss-harmony.jinja`, upstream's with [two deviations](https://whitecircle.com/halo/docs/agent-docs/models/gpt-oss.md#chat-template),
and `gpt-oss-multiturn.jinja`, the SFT form that terminates every assistant turn. Pin a native
template whenever the training render must match serving byte-for-byte.

The template lands on the tokenizer and, for a VLM run, on its processor as well
(`adopt_tokenizer_chat_template`): a processor renders and saves with a `chat_template` of its own,
so the processor is what decides both the VLM rows and the exported `chat_template.jinja`. The same
holds for `prepare_dataset.py --is_vlm` and `patch_vocab.py --chat_template`.

### Special-token ownership

Rendered chat-template text tokenizes through `tokenize_rendered`
(`halo/data/chat_render.py`), which probes each tokenizer once — tokenizing `""` with
`add_special_tokens=True` — to learn what its post-processor actually adds. `bos_token` alone is
unreliable: gpt-oss and Bailing declare a nominal BOS their post-processor never emits, and Zaya
appends a trailing `<|im_end|>` instead of prepending BOS.

- A rendered leading BOS is stripped only when the post-processor re-adds it, so specials appear
  exactly once and no BOS is forced onto a tokenizer that never emits one (gemma-4's
  template-emitted BOS survives).
- Tokenizer-appended trailing specials stay on training rows, unless the render already ends with
  the same terminator sequence (Zaya) — then the duplicate is stripped.
- They are always stripped from generation prompts (`for_generation=True`), so a prompt never ends
  with a turn terminator.

This seam covers SFT (runtime and offline), classification, reward-model scoring (`encode_for_scoring`),
teacher/self-distillation, and generation-eval prompts. The SMPO and offline-GRPO prompt paths use
the same probe to prepend BOS only when the post-processor owns it.

`render_generation_prompt` — the prompt stage the online/async GRPO scripts share — applies
the same rule to the rendered TEXT it returns, and measures `max_prompt_length` with the generation
tokenization the row will really carry.
