KTO (Kahneman-Tversky Optimization) is a preference-tuning method that needs no pairs. Each row is one answer labeled good or bad, and the model learns to make good answers more likely and bad ones less.
How it works
DPO (Direct Preference Optimization) needs two answers to the same prompt, one better than the other. KTO needs one answer and a label: desirable or undesirable. That is the shape of a thumbs-up or thumbs-down on a reply.
The loss borrows from prospect theory, Kahneman and Tversky's account of how people weigh gains and losses. It measures how much more likely the model makes an answer than a frozen reference model does, and compares that with a baseline estimated from the rest of the batch. Desirable answers are pushed above the baseline and undesirable ones below it. The KTO paper has the derivation.
When to use it
Use KTO when you never collected pairs, only single answers marked good or bad. With pairs, DPO and SMPO read them as they are.
In Halo
One row is a prompt, a completion and a boolean label:
JSON Lines
{"prompt": [{"role": "user", "content": "What is 2+2?"}], "completion": [{"role": "assistant", "content": "4"}], "label": true}Start a run with halo launch kto and a config such as examples/preference/qwen3_5/kto-qwen3.5-9b-kto-mix-14k.yaml. The default kto loss builds its baseline, a Kullback-Leibler (KL) term, from each row's neighbors in the batch, so batch size 1 raises. Set desirable_weight and undesirable_weight to the inverse label ratio when the labels are skewed. See the preference guide.
