Datasets
A dataset is a list of records. Conversation columns hold lists of
{"role": "system"|"user"|"assistant", "content": "..."} messages. One SFT
record:
JSON Lines
{"prompt": [{"role": "user", "content": "What is machine learning?"}, {"role": "assistant", "content": "Machine learning is ..."}]}Columns per method
These column names can be renamed in the config:
- SFT:
conversation_field - Classification:
text_field - KTO:
completion_field,label_field - Online and async GRPO:
prompt_field,answer_field
The prompt/chosen/rejected and offline GRPO column names are fixed. A
preference pair without a prompt column gets the turns both sides share
extracted as the prompt.
Where data can come from
dataset takes one of:
- a HuggingFace Hub ID (
repo@split,repo:config); - a local path: JSON/JSONL, Parquet, Arrow, CSV, or a
save_to_diskdirectory; - an S3 path (
s3://my-bucket/key), with your own AWS credentials.
A bare org/name is read as a Hub ID, so write a local directory as
./data/corpus or an absolute path; an org/name that also names a local
directory is refused as ambiguous
(Dataset path formats ↗).
Pass a list to mix sources. dataset_ratio is the fraction of each source to
keep, not a mixing weight:
YAML
dataset:
- HuggingFaceH4/ultrachat_200k@train_sft
- s3://my-bucket/chat/v1/train
dataset_ratio: [1.0, 0.5]Mixing sources has side effects:
- Only columns common to every source survive. Every source must carry the conversation column your config names, or the load raises naming the source. The tools column it names is null-filled where a source lacks it.
- Columns whose types differ between sources are dropped. This is the usual reason a field goes missing.
- A list loads fully on every rank. A sharded entry loads whole on every rank,
and a pre-processed dataset is detected only at a single path. Point
datasetat a single path to use either.
Preprocess large corpora once
Tokenizing and packing a large corpus on the fly costs startup time on every run. Do it once, offline:
Bash
halo run prepare-dataset -- --helpThe tool tokenizes, optionally packs (--pack_sequences), and shards the
corpus, and can push the result to the Hub or S3. Four rules decide whether the
output trains:
--max_lengthis stored with the data and must equal the training config'smax_length, or the load raises, since rows are never re-truncated.--num_shardsabove 1 must be at least your data-parallel size. The default1trains at any size.--mode texttokenizes raw text for pre-training; the defaultchatrenders the conversation through the chat template.- The training run must end with the template the rows were rendered with, since
the checkpoint exports the run's: after a
--chat_templatebake, set the samechat_templateplusforce_chat_template: true, or the load raises.
--tokenizer_backend gigatoken swaps in a faster Rust encoder whose IDs are
checked against the HF tokenizer at startup; training configs take the same
choice through tokenizer_backend. Every flag and the packing strategies:
Pre-Processing ↗.
Data for the RL environments
The code-contests environment reads a prepared pool, not a plain dataset:
halo run compact-code-testscompacts a test corpus into one row per problem, capping the tests in each suite.halo run prepare-code-datasetbuilds the pool. With--push_bandsit also pushes one config per rating band: the<repo>:<band>most shipped examples name indataset:.--config_namepushes one named config instead, which is how stage 1 of the curriculum builds its pool.
Pool format, build flags and bands: Code Contests datasets ↗.
Full schema (multimodal content included), collators and the S3 utilities: Dataset Formats ↗ · S3 Utilities ↗.
Source on GitHubUpdated 18 November 2026