# What is DPO?

DPO (Direct Preference Optimization) fine-tunes a model on pairs of a chosen and a rejected answer. It needs no reward model and no sampling, only a frozen copy of the starting model as a reference.

## How it works

Each row has a prompt, a chosen answer and a rejected answer. Two models compute the log-probability of each answer: the model being trained and a frozen reference, usually its own starting weights. The loss pushes the trained model to raise the chosen answer's log-probability, relative to the reference, by more than it raises the rejected answer's.

A coefficient, beta, sets how far the model may move from the reference. A higher beta holds it closer.

DPO comes out of the [RLHF](/glossary/rlhf) (reinforcement learning from human feedback) objective. The [DPO paper](https://arxiv.org/abs/2305.18290) shows that the reward can be written in terms of the model's own probabilities, which turns the pipeline into a simple classification loss.

## When to use it

Use DPO when you have preference pairs and want the result to stay close to a specific checkpoint. The cost is the reference model in memory. [SMPO](/glossary/smpo) drops the reference, and [KTO](/glossary/kto) (Kahneman-Tversky Optimization) works without pairs.

## In Halo

From `examples/preference/qwen3_5/dpo-qwen3.5-9b-tulu3-prefmix.yaml`:

```yaml
beta: 0.1
loss_type: sigmoid
learning_rate: 5.0e-07
max_length: 4096
```

Start it with `halo launch dpo` and that path. With `use_peft: true` no second model is loaded: the reference is the base model with the adapter switched off. Expert and tensor parallelism refuse a second copy, so set `precompute_ref_log_probs: true` there. See the [preference guide](/halo/docs/training-methods/preference#dpo).

