# Preference tuning

Preference tuning trains a model on judgments of which answer is better, not on gold answers. The model learns to make the preferred answer more likely than the rejected one.

## How it works

Usually a row holds a prompt and two answers to it: one chosen, one rejected. The judgment comes from a person or from another model. The loss looks at how likely the model finds each answer and shifts probability toward the chosen one.

With methods such as DPO (Direct Preference Optimization), nothing is generated during training: no sampling loop, no separate reward model.

## When to use it

Use it when comparing two answers is easier than writing the ideal one: tone, helpfulness, refusals, style. It usually follows supervised fine-tuning, which teaches the task first.

It is the short route to what [RLHF](/glossary/rlhf) (reinforcement learning from human feedback) does with a separate reward model. To get a reusable scorer, train a [reward model](/glossary/reward-model) on the same pairs.

## In Halo

Halo has three methods, each a `halo launch` name: `smpo`, `dpo` and `kto`.

| Method | Columns | Pick it when |
| --- | --- | --- |
| [SMPO](/glossary/smpo) | `prompt`, `chosen`, `rejected` | You have pairs. It holds one model in memory. Start here. |
| [DPO](/glossary/dpo) | `prompt`, `chosen`, `rejected` | The leash to a frozen reference model is the point. |
| [KTO](/glossary/kto) (Kahneman-Tversky Optimization) | `prompt`, `completion`, `label` | You never collected pairs. |

The [preference guide](/halo/docs/training-methods/preference#which-one) has the configs.

