# What is SMPO?

SMPO is a preference-tuning method in Halo that learns from pairs of a chosen and a rejected answer without a reference model. Its loss is a margin between the two answers plus cross-entropy anchors on both.

## How it works

Each row has a prompt, a chosen answer and a rejected answer, the same columns [DPO](/glossary/dpo) (Direct Preference Optimization) reads. One forward pass covers both sides of the pair. The loss has two parts:

- **A margin term.** It compares the mean log-probabilities of the two answers against a margin that ramps up over the run. Once a pair is far enough apart, the term is exactly zero.
- **Cross-entropy anchors.** They keep training the chosen and rejected text directly, which keeps generation quality from drifting.

SMPO holds no reference model, so one model sits in memory instead of two.

## When to use it

Start with SMPO for [preference tuning](/glossary/preference-tuning) unless you have a reason not to. Pick DPO when a leash to a specific checkpoint is the point, and [KTO](/glossary/kto) (Kahneman-Tversky Optimization) when you never collected pairs.

## In Halo

The shipped recipe sets:

```yaml
beta: 1.0
target_margin: 0.4
initial_margin: 0.2
chosen_sft_ratio: 0.75
loss_type: smooth_lower_bound
```

`target_margin` is the log-probability gap a pair should reach, and `initial_margin` is where the ramp starts. `chosen_sft_ratio` splits the anchor between the chosen and rejected sides. Run it with:

```bash
halo launch smpo examples/preference/qwen3_5/smpo-qwen3.5-9b-tulu3-prefmix.yaml -n 8
```

SMPO is the only one of the three preference methods that also takes [context parallelism](/glossary/context-parallelism). See the [preference guide](/halo/docs/training-methods/preference#smpo).

