Direct Preference Optimization (DPO)
DPO fits the policy to pairwise preferences against a frozen reference model, with no reward model. Use it on prompt / chosen / rejected rows when a reference fits the budget; for the same data without one use SMPO, for unpaired thumbs-up/down rows KTO.
Trainer DistributedDPOTrainer, script scripts/training/preference/dpo.py (text or VLM). EP, TP, ETP and PP apply; CP does not — TRL's loss path is not CP-aware (matrix).
Configuration
YAML
model_name_or_path: Qwen/Qwen3.5-9B
dataset: allenai/llama-3.1-tulu-3-8b-preference-mixture
test_size: 0.005
beta: 0.1
loss_type: sigmoid # normalizes to ["sigmoid"]
max_length: 4096
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 5.0e-07 # the shipped recipes' value
gradient_checkpointing: true # bf16 and Liger are toolkit defaults
use_peft: false # true makes the adapter-free base the reference
output_dir: checkpoints/dpo-qwen3.5-9bbeta scales the implicit-reward gap inside the loss (0 flattens the objective), max_length truncates prompt + completion keeping the start, and generation_max_prompt_length caps prompts of the eval generation split only. Types and defaults: DPOConfig and DPOScriptArguments.
loss_type takes TRL's losses (values), and a list combines several under loss_weights (unset: 1.0 each), so [sigmoid, sft] adds an SFT anchor to the preference term.
No training-side prompt cap exists, so an over-long prompt eats its own completion: filter those rows in the dataset (sequence length).
Log-prob precision
The trainer sums sequence log-probs in fp32. TRL's own selective_log_softmax keeps the per-token log-probs of bf16 logits in bf16, and sums them in bf16. At |logp| in [8192, 16384) that grid is 64 nats, so a long completion's margin carries tens of nats of rounding.
TRLPreferenceMixin (halo/trainers/preference/trl_preference.py) swaps that function out while TRL's loss and reference pass run. The replacement is a chunked fp32 log-softmax (halo/kernels/logprobs.py) that never builds a full fp32 [B, T, V] plane. TRL sums its output in fp32, so these are fp32:
- every loss term built on the sequence log-probs, and the
ld_alphasplit; - the log-prob term of the WPO weights;
- the
logps/*andrewards/*metrics.
Terms TRL takes from the logits directly keep TRL's precision: the sft loss type's cross-entropy, WPO's normalizer, and the entropy, logits/* and mean_token_accuracy metrics. Reference columns the dataset supplies keep the precision they were computed in.
Every run takes the fp32 path. The trainer turns TRL's use_liger_kernel off at construction: the flag selects TRL's chunked log-prob path, which refuses the router aux loss and skips the family's head transform. Liger kernels applied at load are unaffected.
Reference model
Three shapes, decided by load_reference_model_for_preference (halo/model_loading/parallel/frozen_models.py):
- PEFT (
use_peft: true) — no second model: the script passes a LoRA config, so the reference is the base with the adapter disabled. A trainer built by hand around an already-wrappedPeftModelgets a frozenrefadapter copy instead. - EP / TP with
precompute_ref_log_probs: true— no reference is loaded either. Log-probs come from the untrained policy before step 1, and a resume restores them from the checkpoint (below). - A frozen copy — every other shape, precompute on plain data parallelism included. It mirrors the policy load (same revision, attention validator, sink policy) and stays resident for the run.
The frozen copy is never parallelized. Under EP or TP that makes it a whole dense replica on every rank, experts included, run through the model's own MoE forward. Its log-probs match the policy's up to kernel numerics, so it loads, with a warning about its memory: precompute scores the same reference once and is both exact and cheaper, and PEFT avoids it too. TP rejects PEFT, leaving precompute or the frozen copy. Expert-only native EP LoRA needs precompute: with no PeftModel nothing switches the adapters off. A mixed attention + expert adapter keeps the implicit reference, since disable_adapter() drops the expert adapters too.
A reference model whose attention sinks differ from the policy's is refused in every mode, single GPU included. The script's frozen copy takes the policy's reset_sinks, so it passes with live sinks (reset_sinks: false) too. A reference TRL builds itself neutralizes nothing, and GPT-OSS's default attention (eager) applies its pretrained sinks, so it is refused beside neutralized ones (reset_sinks: true, the default).
Resuming a precompute run
TRL's constructor only records each split, and the sweep runs at the end of the trainer's __init__, after the parallel-mode wrap. It then computes at the training forward's dtype: unwrapped, fp32 masters would reach DeepEP's combine as fp32. It scores the policy when no separate reference exists, and every rank attaches the gathered columns in memory, so per-node storage needs no shared cache file. A Path-B resume builds the policy from the checkpoint before that, so a sweep there would score the trained weights and zero every log-ratio. Every checkpoint of a sweeping run therefore carries the columns in reference_logps.pt: one entry per split (train, then each eval dataset by name) with its row count, a digest of each token-id column the reference read, and the settings that shape the values (max_length and the log-prob precision, plus truncation_mode and ld_alpha for DPO). The script hands the trainer the resume checkpoint, and the shared lifecycle attaches the saved columns instead of sweeping (halo/trainers/reference_logps.py).
- When the policy was built from the checkpoint's weights and no separate reference exists, a split with no saved entry raises, and so does one whose row count, token ids, settings or columns (a KL column the run's KTO loss needs) differ from the run's. Any other resume (an adapter or merge-on-save checkpoint builds the policy from the base; a frozen reference copy) sweeps untrained weights for such a split and proceeds.
- A split whose settings record another log-prob precision, or none, is such a mismatch: its values carry that precision's rounding. The refusal names the one-step regeneration below.
- Splits the resumed run does not precompute (an eval dataset switched off) ride unchanged into its checkpoints, for a later resume that uses them again.
- To give a checkpoint without the file one, run the same config for one step from the base into a scratch directory outside the run's
output_dir, whose rotation could otherwise delete the checkpoint (--output_dir=<scratch> --max_steps=1 --save_strategy=steps --save_steps=1 --save_only_model=true --resume_from_checkpoint=null; the refusal names a scratch path), and copy itscheckpoint-1/reference_logps.ptinto the checkpoint, on every node when checkpoints are node-local; the resume validates it like its own. A checkpoint whose save stopped before the file can take the previous checkpoint's copy, which holds the same values. - Columns the dataset already carries are read from it and not persisted.
Launch
Bash
# Plain FSDP2 data parallel
torchrun --nproc_per_node=8 scripts/training/preference/dpo.py \
examples/preference/qwen3_5/dpo-qwen3.5-9b-tulu3-prefmix.yaml
# MoE, expert parallel (that recipe is LoRA, so the adapters are the reference)
torchrun --nproc_per_node=8 scripts/training/preference/dpo.py \
examples/preference/gptoss/dpo-gptoss-20b-tulu3-prefmix-ep.yaml --expert_parallel_size=8halo launch dpo <config> --nproc 8 builds the same line; any field is overridable (--beta=0.05). accelerate launch serves plain DP only — EP/CP/TP reject it.
Pipeline parallelism
Under pipeline_parallel_size > 1 DPO is precompute-only and text-only: no stage holds the whole model, so set precompute_ref_log_probs: true with ref_chosen_logps / ref_rejected_logps already in the train and every eval split, and pass no ref_model. The last stage re-implements sigmoid, hinge and ipo, alone or combined; every other loss_type is refused (aot, aot_unpaired and sft couple the whole batch), as are an f_divergence_type other than reverse_kl, use_weighting, ld_alpha and compute_metrics. The refusals every PP trainer shares, PEFT included: Rejected combinations.
Vision-language
dpo.py trains a VLM checkpoint on image+text pairs. The model class follows the checkpoint, the data path follows the dataset: text-only pairs on a natively multimodal model take the text pipeline, hub-shape normalization included.
An images/image column puts TRL in vision mode with DataCollatorForVisionPreference, except on a family declaring refuses_trl_image_rows (Nemotron-H), refused before the load. Those rows must already be contract-shaped — prompt a message list, chosen/rejected continuation-only — since TRL normalizes no hub shapes. tools_field is refused: the vision render passes no tools=. A prefix-LM checkpoint (PaliGemma) is refused on image rows before the model load (Modality support); train it on images with SFT or distillation.
Set images_field: <column> when the dataset stores images under another name — it is renamed to images before the dispatch, the spelling TRL probes for. A name the splits do not carry raises at dataset preparation, after the policy and the reference have loaded.
TRL rejects precompute_ref_log_probs on vision datasets, so under EP the vision reference is standard PEFT adapters or a frozen copy, and under TP, where PEFT is rejected, a frozen copy.
Testing a setup
Bash
torchrun --nproc_per_node=2 scripts/training/preference/dpo.py <config> \
--max_steps=5 --save_strategy=no --report_to=noneCovering tests:
pytest tests/cpu/trainers -m cputests/gpu/trainers/preference/test_dpo.py,test_dpo_vlm.pyandtest_pref_ep_expert_lora_reference.py- pipeline parallelism on precomputed reference log-probs:
tests/gpu/parallelism/pp/test_pp_preference_precompute.py --trainer dpo, and its save, exact resume and export:test_pp_trainer_checkpoints.py --trainer dpo - precompute resume, over
tests/common/preference_precompute_e2e.py:tests/gpu/parallelism/ep/test_ep_preference_precompute_resume.py(Qwen3-MoE and the dense model in the core tier; every other EP family, with TP and EP+TP where its attention has a TP plan, and EP+ETP on four ranks for every EP family in the full tier)
What to watch
Failure signatures:
- A reference-model raise on a live-sinks policy —
use_peft: true, or precompute under EP/TP.beta: 0does not help: it changes the loss, not whether a reference loads. - A host OOM while the frozen reference loads, or a CUDA OOM when the trainer places it, under EP or TP — the dense replica does not fit beside the sharded policy; set
precompute_ref_log_probs: true(oruse_peft: trueunder EP; TP rejects PEFT). - "Cannot resume precompute_ref_log_probs" — the checkpoint's
reference_logps.ptis missing or lacks that split. Recover it with the one-step run above, or put the reference columns (computed on the base model) in the dataset. - "does not belong to this '<split>' dataset" on resume — the resumed data or reference settings differ from the saving run's (dataset, split, chat template, tokenizer,
max_length,truncation_mode,ld_alpha). Resume with the saving run's data and settings, or regenerate the file with the one-step run above. - "Regenerate the '<split>' reference log-probs for this run" on resume — the saved split was summed at another log-prob precision, which no setting selects. Run the one-step regeneration the message names.
Source on GitHubUpdated 18 November 2026