SMPO is a preference-tuning method in Halo that learns from pairs of a chosen and a rejected answer without a reference model. Its loss is a margin between the two answers plus cross-entropy anchors on both.
How it works
Each row has a prompt, a chosen answer and a rejected answer, the same columns DPO (Direct Preference Optimization) reads. One forward pass covers both sides of the pair. The loss has two parts:
- A margin term. It compares the mean log-probabilities of the two answers against a margin that ramps up over the run. Once a pair is far enough apart, the term is exactly zero.
- Cross-entropy anchors. They keep training the chosen and rejected text directly, which keeps generation quality from drifting.
SMPO holds no reference model, so one model sits in memory instead of two.
When to use it
Start with SMPO for preference tuning unless you have a reason not to. Pick DPO when a leash to a specific checkpoint is the point, and KTO (Kahneman-Tversky Optimization) when you never collected pairs.
In Halo
The shipped recipe sets:
YAML
beta: 1.0
target_margin: 0.4
initial_margin: 0.2
chosen_sft_ratio: 0.75
loss_type: smooth_lower_boundtarget_margin is the log-probability gap a pair should reach, and initial_margin is where the ramp starts. chosen_sft_ratio splits the anchor between the chosen and rejected sides. Run it with:
Bash
halo launch smpo examples/preference/qwen3_5/smpo-qwen3.5-9b-tulu3-prefmix.yaml -n 8SMPO is the only one of the three preference methods that also takes context parallelism. See the preference guide.
