What is DPO?

DPO (Direct Preference Optimization) fine-tunes a model on pairs of a chosen and a rejected answer. It needs no reward model and no sampling, only a frozen copy of the starting model as a reference.

How it works

Each row has a prompt, a chosen answer and a rejected answer. Two models compute the log-probability of each answer: the model being trained and a frozen reference, usually its own starting weights. The loss pushes the trained model to raise the chosen answer's log-probability, relative to the reference, by more than it raises the rejected answer's.

A coefficient, beta, sets how far the model may move from the reference. A higher beta holds it closer.

DPO comes out of the RLHF (reinforcement learning from human feedback) objective. The DPO paper shows that the reward can be written in terms of the model's own probabilities, which turns the pipeline into a simple classification loss.

When to use it

Use DPO when you have preference pairs and want the result to stay close to a specific checkpoint. The cost is the reference model in memory. SMPO drops the reference, and KTO (Kahneman-Tversky Optimization) works without pairs.

In Halo

From examples/preference/qwen3_5/dpo-qwen3.5-9b-tulu3-prefmix.yaml:

YAML

beta: 0.1
loss_type: sigmoid
learning_rate: 5.0e-07
max_length: 4096

Start it with halo launch dpo and that path. With use_peft: true no second model is loaded: the reference is the base model with the adapter switched off. Expert and tensor parallelism refuse a second copy, so set precompute_ref_log_probs: true there. See the preference guide.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL