Preference Optimization
Three methods learn from comparisons and KTO from per-row binary labels. Pick by the shape of the rows you have.
Other shapes: several scored completions per prompt → Offline GRPO; generation during training → Online GRPO or Async GRPO with Environments; one good completion per prompt → SFT.
Dataset format
The three pairwise methods share one format, all fields list[dict] messages:
JSON Lines
{"prompt": [{"role": "user", "content": "What is the capital of France?"}], "chosen": [{"role": "assistant", "content": "Paris is the capital of France."}], "rejected": [{"role": "assistant", "content": "France is in Europe."}]}prompt is the conversation up to divergence; chosen and rejected are the competing completions. Reward modeling also reads implicit-prompt datasets. KTO's unpaired shape is on its own page. Full contract: Dataset Formats.
Parallelism
All four run EP, TP, ETP, EP+TP and PP (DPO and KTO precompute-only). SMPO alone declares CP support: it CP-aggregates its per-sequence log-prob sums, while DPO and KTO run TRL's CP-unaware loss path and the reward head needs the whole sequence to pool. Full matrix: Trainer Compatibility.
Launch
Bash
torchrun --nproc_per_node=8 scripts/training/preference/smpo.py \
examples/preference/gptoss/smpo-gptoss-20b-tulu3-prefmix-ep.yaml \
--expert_parallel_size=8halo launch smpo <config> --nproc 8 builds the same line; the other methods are dpo, kto and reward/reward-model.
Source on GitHubUpdated 18 November 2026