# Direct Preference Optimization (DPO)

DPO fits the policy to pairwise preferences against a frozen reference model, with no reward model. Use it on `prompt` / `chosen` / `rejected` rows when a reference fits the budget; for the same data without one use [SMPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/preference/smpo.md), for unpaired thumbs-up/down rows [KTO](https://whitecircle.com/halo/docs/agent-docs/training-methods/preference/kto.md).

Trainer `DistributedDPOTrainer`, script `scripts/training/preference/dpo.py` (text or VLM). EP, TP, ETP and [PP](#pipeline-parallelism) apply; CP does not — TRL's loss path is not CP-aware ([matrix](https://whitecircle.com/halo/docs/agent-docs/reference/trainer-architecture.md#trainer-compatibility)).

## Configuration

```yaml
model_name_or_path: Qwen/Qwen3.5-9B
dataset: allenai/llama-3.1-tulu-3-8b-preference-mixture
test_size: 0.005
beta: 0.1
loss_type: sigmoid                 # normalizes to ["sigmoid"]
max_length: 4096
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 5.0e-07             # the shipped recipes' value
gradient_checkpointing: true       # bf16 and Liger are toolkit defaults
use_peft: false                    # true makes the adapter-free base the reference
output_dir: checkpoints/dpo-qwen3.5-9b
```

`beta` scales the implicit-reward gap inside the loss (`0` flattens the objective), `max_length` truncates prompt + completion keeping the start, and `generation_max_prompt_length` caps prompts of the eval generation split only. Types and defaults: [DPOConfig](https://whitecircle.com/halo/docs/agent-docs/reference/configuration-reference.md#trl-trainer-configs) and [DPOScriptArguments](https://whitecircle.com/halo/docs/agent-docs/reference/configuration-reference.md#other-script-arguments).

`loss_type` takes TRL's losses ([values](https://whitecircle.com/halo/docs/agent-docs/reference/configuration-reference.md#trl-trainer-configs)), and a list combines several under `loss_weights` (unset: `1.0` each), so `[sigmoid, sft]` adds an SFT anchor to the preference term.

No training-side prompt cap exists, so an over-long prompt eats its own completion: filter those rows in the dataset ([sequence length](https://whitecircle.com/halo/docs/agent-docs/reference/configuration-reference.md#sequence-length-caps-vs-generation-budgets)).

## Log-prob precision

The trainer sums sequence log-probs in fp32. TRL's own `selective_log_softmax` keeps the per-token log-probs of bf16 logits in bf16, and sums them in bf16. At `|logp|` in [8192, 16384) that grid is 64 nats, so a long completion's margin carries tens of nats of rounding.

`TRLPreferenceMixin` (`halo/trainers/preference/trl_preference.py`) swaps that function out while TRL's loss and reference pass run. The replacement is a chunked fp32 log-softmax (`halo/kernels/logprobs.py`) that never builds a full fp32 `[B, T, V]` plane. TRL sums its output in fp32, so these are fp32:

- every loss term built on the sequence log-probs, and the `ld_alpha` split;
- the log-prob term of the WPO weights;
- the `logps/*` and `rewards/*` metrics.

Terms TRL takes from the logits directly keep TRL's precision: the `sft` loss type's cross-entropy, WPO's normalizer, and the `entropy`, `logits/*` and `mean_token_accuracy` metrics. Reference columns the dataset supplies keep the precision they were computed in.

Every run takes the fp32 path. The trainer turns TRL's `use_liger_kernel` off at construction: the flag selects TRL's chunked log-prob path, which refuses the router aux loss and skips the family's head transform. Liger kernels applied at load are unaffected.

## Reference model

Three shapes, decided by `load_reference_model_for_preference` (`halo/model_loading/parallel/frozen_models.py`):

- **PEFT** (`use_peft: true`) — no second model: the script passes a LoRA config, so the reference is the base with the adapter disabled. A trainer built by hand around an already-wrapped `PeftModel` gets a frozen `ref` adapter copy instead.
- **EP / TP with `precompute_ref_log_probs: true`** — no reference is loaded either. Log-probs come from the untrained policy before step 1, and a resume restores them from the checkpoint ([below](#resuming-a-precompute-run)).
- **A frozen copy** — every other shape, precompute on plain data parallelism included. It mirrors the policy load (same revision, attention validator, sink policy) and stays resident for the run.

The frozen copy is never parallelized. Under EP or TP that makes it a whole dense replica on every rank, experts included, run through the model's own MoE forward. Its log-probs match the policy's up to kernel numerics, so it loads, with a warning about its memory: precompute scores the same reference once and is both exact and cheaper, and PEFT avoids it too. TP rejects PEFT, leaving precompute or the frozen copy. Expert-only native EP LoRA needs precompute: with no `PeftModel` nothing switches the adapters off. A mixed attention + expert adapter keeps the implicit reference, since `disable_adapter()` drops the expert adapters too.

A reference model whose attention sinks differ from the policy's is refused in every mode, single GPU included. The script's frozen copy takes the policy's `reset_sinks`, so it passes with live sinks (`reset_sinks: false`) too. A reference TRL builds itself neutralizes nothing, and GPT-OSS's default attention (eager) applies its pretrained sinks, so it is refused beside neutralized ones (`reset_sinks: true`, the default).

### Resuming a precompute run

TRL's constructor only records each split, and the sweep runs at the end of the trainer's `__init__`, after the parallel-mode wrap. It then computes at the training forward's dtype: unwrapped, fp32 masters would reach DeepEP's combine as fp32. It scores the policy when no separate reference exists, and every rank attaches the gathered columns in memory, so per-node storage needs no shared cache file. A [Path-B resume](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#resuming-training) builds the policy from the checkpoint before that, so a sweep there would score the trained weights and zero every log-ratio. Every checkpoint of a sweeping run therefore carries the columns in `reference_logps.pt`: one entry per split (`train`, then each eval dataset by name) with its row count, a digest of each token-id column the reference read, and the settings that shape the values (`max_length` and the [log-prob precision](#log-prob-precision), plus `truncation_mode` and `ld_alpha` for DPO). The script hands the trainer the resume checkpoint, and the shared lifecycle attaches the saved columns instead of sweeping (`halo/trainers/reference_logps.py`).

- When the policy was built from the checkpoint's weights and no separate reference exists, a split with no saved entry raises, and so does one whose row count, token ids, settings or columns (a KL column the run's KTO loss needs) differ from the run's. Any other resume (an adapter or merge-on-save checkpoint builds the policy from the base; a frozen reference copy) sweeps untrained weights for such a split and proceeds.
- A split whose settings record another log-prob precision, or none, is such a mismatch: its values carry that precision's rounding. The refusal names the one-step regeneration below.
- Splits the resumed run does not precompute (an eval dataset switched off) ride unchanged into its checkpoints, for a later resume that uses them again.
- To give a checkpoint without the file one, run the same config for one step from the base into a scratch directory outside the run's `output_dir`, whose rotation could otherwise delete the checkpoint (`--output_dir=<scratch> --max_steps=1 --save_strategy=steps --save_steps=1 --save_only_model=true --resume_from_checkpoint=null`; the refusal names a scratch path), and copy its `checkpoint-1/reference_logps.pt` into the checkpoint, on every node when checkpoints are node-local; the resume validates it like its own. A checkpoint whose save stopped before the file can take the previous checkpoint's copy, which holds the same values.
- Columns the dataset already carries are read from it and not persisted.

## Launch

```bash
# Plain FSDP2 data parallel
torchrun --nproc_per_node=8 scripts/training/preference/dpo.py \
    examples/preference/qwen3_5/dpo-qwen3.5-9b-tulu3-prefmix.yaml

# MoE, expert parallel (that recipe is LoRA, so the adapters are the reference)
torchrun --nproc_per_node=8 scripts/training/preference/dpo.py \
    examples/preference/gptoss/dpo-gptoss-20b-tulu3-prefmix-ep.yaml --expert_parallel_size=8
```

`halo launch dpo <config> --nproc 8` builds the same line; any field is overridable (`--beta=0.05`). `accelerate launch` serves plain DP only — EP/CP/TP reject it.

## Pipeline parallelism

Under `pipeline_parallel_size > 1` DPO is precompute-only and text-only: no stage holds the whole model, so set `precompute_ref_log_probs: true` with `ref_chosen_logps` / `ref_rejected_logps` already in the train and every eval split, and pass no `ref_model`. The last stage re-implements `sigmoid`, `hinge` and `ipo`, alone or combined; every other `loss_type` is refused (`aot`, `aot_unpaired` and `sft` couple the whole batch), as are an `f_divergence_type` other than `reverse_kl`, `use_weighting`, `ld_alpha` and `compute_metrics`. The refusals every PP trainer shares, PEFT included: [Rejected combinations](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md#rejected-combinations).

## Vision-language

`dpo.py` trains a VLM checkpoint on image+text pairs. The model class follows the checkpoint, the data path follows the **dataset**: text-only pairs on a natively multimodal model take the text pipeline, hub-shape normalization included.

An `images`/`image` column puts TRL in vision mode with `DataCollatorForVisionPreference`, except on a family declaring `refuses_trl_image_rows` (Nemotron-H), refused before the load. Those rows must already be contract-shaped — prompt a message list, chosen/rejected continuation-only — since TRL normalizes no hub shapes. `tools_field` is refused: the vision render passes no `tools=`. A prefix-LM checkpoint (PaliGemma) is refused on image rows before the model load ([Modality support](https://whitecircle.com/halo/docs/agent-docs/reference/trainer-architecture.md#modality-support)); train it on images with SFT or distillation.

Set `images_field: <column>` when the dataset stores images under another name — it is renamed to `images` before the dispatch, the spelling TRL probes for. A name the splits do not carry raises at dataset preparation, after the policy and the reference have loaded.

TRL rejects `precompute_ref_log_probs` on vision datasets, so under EP the vision reference is standard PEFT adapters or a frozen copy, and under TP, where PEFT is rejected, a frozen copy.

## Testing a setup

```bash
torchrun --nproc_per_node=2 scripts/training/preference/dpo.py <config> \
    --max_steps=5 --save_strategy=no --report_to=none
```

Covering tests:

- `pytest tests/cpu/trainers -m cpu`
- `tests/gpu/trainers/preference/test_dpo.py`, `test_dpo_vlm.py` and `test_pref_ep_expert_lora_reference.py`
- pipeline parallelism on precomputed reference log-probs: `tests/gpu/parallelism/pp/test_pp_preference_precompute.py --trainer dpo`, and its save, exact resume and export: `test_pp_trainer_checkpoints.py --trainer dpo`
- precompute resume, over `tests/common/preference_precompute_e2e.py`: `tests/gpu/parallelism/ep/test_ep_preference_precompute_resume.py` (Qwen3-MoE and the dense model in the core tier; every other EP family, with TP and EP+TP where its attention has a TP plan, and EP+ETP on four ranks for every EP family in the full tier)

## What to watch

| Signal | Reading |
|---|---|
| `rewards/accuracies` | Share of pairs scoring chosen above rejected (chance is 0.5) |
| `rewards/margins` | `beta ×` the log-ratio gap; rises as pairs separate |
| `logps/chosen`, `logps/rejected` | Both falling together is log-prob collapse: lower the LR, raise `beta`, or add `sft` |

Failure signatures:

- A reference-model raise on a live-sinks policy — `use_peft: true`, or precompute under EP/TP. `beta: 0` does not help: it changes the loss, not whether a reference loads.
- A host OOM while the frozen reference loads, or a CUDA OOM when the trainer places it, under EP or TP — the dense replica does not fit beside the sharded policy; set `precompute_ref_log_probs: true` (or `use_peft: true` under EP; TP rejects PEFT).
- "Cannot resume precompute_ref_log_probs" — the checkpoint's `reference_logps.pt` is missing or lacks that split. Recover it with the one-step run [above](#resuming-a-precompute-run), or put the reference columns (computed on the base model) in the dataset.
- "does not belong to this '<split>' dataset" on resume — the resumed data or reference settings differ from the saving run's (dataset, split, chat template, tokenizer, `max_length`, `truncation_mode`, `ld_alpha`). Resume with the saving run's data and settings, or regenerate the file with the one-step run [above](#resuming-a-precompute-run).
- "Regenerate the '<split>' reference log-probs for this run" on resume — the saved split was summed at another [log-prob precision](#log-prob-precision), which no setting selects. Run the one-step regeneration the message names.
