Datasets

A dataset is a list of records. Conversation columns hold lists of {"role": "system"|"user"|"assistant", "content": "..."} messages. One SFT record:

JSON Lines

{"prompt": [{"role": "user", "content": "What is machine learning?"}, {"role": "assistant", "content": "Machine learning is ..."}]}

Columns per method

MethodRequired columnsTypes
SFT (incl. VLM)promptList[Dict]
DPO / SMPOchosen, rejected (+ optional prompt)List[Dict] (prompt may be str)
Rewardchosen, rejected (+ optional prompt)List[Dict] (prompt may be str)
KTOprompt, completion, labelList[Dict], List[Dict]/str, bool
Offline GRPOprompt, completions, rewardsList[Dict], List[List[Dict]], List[float]
Classificationprompt or text_field, labelList[Dict]/str, a label or a list of labels (strings or ints)
Online GRPOprompt, answerstr/List[Dict], str
Async GRPO with environmentsprompt (+ answer where the environment grades one)str/List[Dict], str

These column names can be renamed in the config:

  • SFT: conversation_field
  • Classification: text_field
  • KTO: completion_field, label_field
  • Online and async GRPO: prompt_field, answer_field

The prompt/chosen/rejected and offline GRPO column names are fixed. A preference pair without a prompt column gets the turns both sides share extracted as the prompt.

Where data can come from

dataset takes one of:

  • a HuggingFace Hub ID (repo@split, repo:config);
  • a local path: JSON/JSONL, Parquet, Arrow, CSV, or a save_to_disk directory;
  • an S3 path (s3://my-bucket/key), with your own AWS credentials.

A bare org/name is read as a Hub ID, so write a local directory as ./data/corpus or an absolute path; an org/name that also names a local directory is refused as ambiguous (Dataset path formats ↗).

Pass a list to mix sources. dataset_ratio is the fraction of each source to keep, not a mixing weight:

YAML

dataset:
  - HuggingFaceH4/ultrachat_200k@train_sft
  - s3://my-bucket/chat/v1/train
dataset_ratio: [1.0, 0.5]

Mixing sources has side effects:

  • Only columns common to every source survive. Every source must carry the conversation column your config names, or the load raises naming the source. The tools column it names is null-filled where a source lacks it.
  • Columns whose types differ between sources are dropped. This is the usual reason a field goes missing.
  • A list loads fully on every rank. A sharded entry loads whole on every rank, and a pre-processed dataset is detected only at a single path. Point dataset at a single path to use either.

Preprocess large corpora once

Tokenizing and packing a large corpus on the fly costs startup time on every run. Do it once, offline:

Bash

halo run prepare-dataset -- --help

The tool tokenizes, optionally packs (--pack_sequences), and shards the corpus, and can push the result to the Hub or S3. Four rules decide whether the output trains:

  • --max_length is stored with the data and must equal the training config's max_length, or the load raises, since rows are never re-truncated.
  • --num_shards above 1 must be at least your data-parallel size. The default 1 trains at any size.
  • --mode text tokenizes raw text for pre-training; the default chat renders the conversation through the chat template.
  • The training run must end with the template the rows were rendered with, since the checkpoint exports the run's: after a --chat_template bake, set the same chat_template plus force_chat_template: true, or the load raises.

--tokenizer_backend gigatoken swaps in a faster Rust encoder whose IDs are checked against the HF tokenizer at startup; training configs take the same choice through tokenizer_backend. Every flag and the packing strategies: Pre-Processing ↗.

Data for the RL environments

The code-contests environment reads a prepared pool, not a plain dataset:

  1. halo run compact-code-tests compacts a test corpus into one row per problem, capping the tests in each suite.
  2. halo run prepare-code-dataset builds the pool. With --push_bands it also pushes one config per rating band: the <repo>:<band> most shipped examples name in dataset:. --config_name pushes one named config instead, which is how stage 1 of the curriculum builds its pool.

Pool format, build flags and bands: Code Contests datasets ↗.

Full schema (multimodal content included), collators and the S3 utilities: Dataset Formats ↗ · S3 Utilities ↗.

Source on GitHubUpdated 18 November 2026