Preference tuning

Preference tuning trains a model on judgments of which answer is better, not on gold answers. The model learns to make the preferred answer more likely than the rejected one.

How it works

Usually a row holds a prompt and two answers to it: one chosen, one rejected. The judgment comes from a person or from another model. The loss looks at how likely the model finds each answer and shifts probability toward the chosen one.

With methods such as DPO (Direct Preference Optimization), nothing is generated during training: no sampling loop, no separate reward model.

When to use it

Use it when comparing two answers is easier than writing the ideal one: tone, helpfulness, refusals, style. It usually follows supervised fine-tuning, which teaches the task first.

It is the short route to what RLHF (reinforcement learning from human feedback) does with a separate reward model. To get a reusable scorer, train a reward model on the same pairs.

In Halo

Halo has three methods, each a halo launch name: smpo, dpo and kto.

MethodColumnsPick it when
SMPOprompt, chosen, rejectedYou have pairs. It holds one model in memory. Start here.
DPOprompt, chosen, rejectedThe leash to a frozen reference model is the point.
KTO (Kahneman-Tversky Optimization)prompt, completion, labelYou never collected pairs.

The preference guide has the configs.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL