Fine-tuning dataset formats

Fine-tuning dataset formats depend on the training method. Supervised fine-tuning reads conversations, preference tuning reads a chosen and a rejected answer, and reinforcement learning reads prompts with scores or checkable answers.

What a record looks like

A dataset is a list of records, usually stored as JSONL (one JSON object per line) or Parquet. A conversation is a list of messages, each with a role (system, user or assistant) and its content:

JSON Lines

{"prompt": [{"role": "user", "content": "Explain gravity."}, {"role": "assistant", "content": "Gravity is ..."}]}

The file holds plain messages. The chat template turns them into tokens when training starts.

What each method reads

The shapes are common across tools. The column names are the ones Halo reads.

MethodTrains onColumns
Supervised fine-tuningconversationsprompt
Preference tuning (DPO, SMPO)a chosen and a rejected answerchosen, rejected, optional prompt
Preference tuning (KTO)answers labeled good or badprompt, completion, label
Offline GRPOcompletions you already scoredprompt, completions, rewards
Online GRPOprompts with a verifiable answerprompt, answer

In Halo

dataset accepts a Hugging Face Hub ID, a local path (JSON/JSONL, Parquet, Arrow, CSV, or a save_to_disk directory) or an S3 path. Some column names are configurable, such as conversation_field for supervised fine-tuning. Datasets gives the columns and types method by method.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL