What is SMPO?

SMPO is a preference-tuning method in Halo that learns from pairs of a chosen and a rejected answer without a reference model. Its loss is a margin between the two answers plus cross-entropy anchors on both.

How it works

Each row has a prompt, a chosen answer and a rejected answer, the same columns DPO (Direct Preference Optimization) reads. One forward pass covers both sides of the pair. The loss has two parts:

  • A margin term. It compares the mean log-probabilities of the two answers against a margin that ramps up over the run. Once a pair is far enough apart, the term is exactly zero.
  • Cross-entropy anchors. They keep training the chosen and rejected text directly, which keeps generation quality from drifting.

SMPO holds no reference model, so one model sits in memory instead of two.

When to use it

Start with SMPO for preference tuning unless you have a reason not to. Pick DPO when a leash to a specific checkpoint is the point, and KTO (Kahneman-Tversky Optimization) when you never collected pairs.

In Halo

The shipped recipe sets:

YAML

beta: 1.0
target_margin: 0.4
initial_margin: 0.2
chosen_sft_ratio: 0.75
loss_type: smooth_lower_bound

target_margin is the log-probability gap a pair should reach, and initial_margin is where the ramp starts. chosen_sft_ratio splits the anchor between the chosen and rejected sides. Run it with:

Bash

halo launch smpo examples/preference/qwen3_5/smpo-qwen3.5-9b-tulu3-prefmix.yaml -n 8

SMPO is the only one of the three preference methods that also takes context parallelism. See the preference guide.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL