# Preference Optimization

Three methods learn from comparisons and KTO from per-row binary labels. Pick by the shape of the rows you have.

| Row shape | Method | Why |
|---|---|---|
| `prompt` / `chosen` / `rejected` | [SMPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/preference/smpo.md) | No reference model; the margin loss stops once a pair separates |
| `prompt` / `chosen` / `rejected` | [DPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/preference/dpo.md) | Reference-based; 15 TRL loss types, combinable |
| `prompt` / `completion` / `label` | [KTO](https://whitecircle.com/halo/docs/agent-docs/training-methods/preference/kto.md) | Unpaired thumbs-up/down feedback |
| `prompt` / `chosen` / `rejected` → scorer | [Reward modeling](https://whitecircle.com/halo/docs/agent-docs/training-methods/reward-modeling.md) | A Bradley-Terry score head for rejection sampling or RL |

Other shapes: several scored completions per prompt → [Offline GRPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/offline.md); generation during training → [Online GRPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/online.md) or [Async GRPO with Environments](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo.md); one good completion per prompt → [SFT](https://whitecircle.com/halo/docs/agent-docs/training-methods/sft.md).

## Dataset format

The three pairwise methods share one format, all fields `list[dict]` messages:

```jsonl
{"prompt": [{"role": "user", "content": "What is the capital of France?"}], "chosen": [{"role": "assistant", "content": "Paris is the capital of France."}], "rejected": [{"role": "assistant", "content": "France is in Europe."}]}
```

`prompt` is the conversation up to divergence; `chosen` and `rejected` are the competing completions. Reward modeling also reads implicit-prompt datasets. KTO's unpaired shape is on its own page. Full contract: [Dataset Formats](https://whitecircle.com/halo/docs/agent-docs/data/dataset-formats.md#preference-dposmpo).

## Parallelism

All four run EP, TP, ETP, EP+TP and PP (DPO and KTO precompute-only). SMPO alone declares CP support: it CP-aggregates its per-sequence log-prob sums, while DPO and KTO run TRL's CP-unaware loss path and the reward head needs the whole sequence to pool. Full matrix: [Trainer Compatibility](https://whitecircle.com/halo/docs/agent-docs/reference/trainer-architecture.md#trainer-compatibility).

## Launch

```bash
torchrun --nproc_per_node=8 scripts/training/preference/smpo.py \
    examples/preference/gptoss/smpo-gptoss-20b-tulu3-prefmix-ep.yaml \
    --expert_parallel_size=8
```

`halo launch smpo <config> --nproc 8` builds the same line; the other methods are `dpo`, `kto` and `reward/reward-model`.
