KTO

Kahneman-Tversky Optimization trains on unpaired binary feedback: each row is one completion labeled desirable or not. Use it for thumbs-up/down labels rather than preference pairs; for chosen/rejected pairs use DPO or SMPO, for several scored completions per prompt Offline GRPO.

Trainer DistributedKTOTrainer, script scripts/training/preference/kto.py (text or VLM). EP, TP, ETP and PP apply; CP does not — TRL's loss path is not CP-aware. TRL's use_liger_kernel is turned off at construction: it selects TRL's chunked log-prob path, which refuses the router aux loss. Liger kernels still apply at the model level.

Dataset

JSON Lines

{"prompt": [{"role": "user", "content": "What is 2+2?"}], "completion": [{"role": "assistant", "content": "4"}], "label": true}

prompt / completion may be plain strings or message lists — TRL templates them itself. completion_field / label_field rename those columns, and a name the dataset lacks raises before training. tools_field and log_decoded_samples are refused: TRL templates the raw columns without tools=, and the rows reach it untokenized.

Configuration

YAML

model_name_or_path: Qwen/Qwen3.5-9B
dataset: trl-lib/kto-mix-14k
beta: 0.1
desirable_weight: 1.0
undesirable_weight: 1.0
loss_type: kto

per_device_train_batch_size: 2     # > 1 is required by the `kto` loss
gradient_accumulation_steps: 4
learning_rate: 5.0e-07
max_length: 4096
gradient_checkpointing: true
output_dir: checkpoints/kto-qwen3.5-9b

beta bounds the deviation from the reference, desirable_weight / undesirable_weight re-weight the two classes when the label counts are skewed, loss_type: apo_zero_unpaired carries no KL term, and max_length truncates prompt + completion keeping the start. Types and defaults: KTOConfig.

The default kto loss builds each row's KL completion from its neighbors in a fixed-order batch, so TRL raises at per_device_train_batch_size: 1 and at any train_sampling_strategy other than the default sequential. Only apo_zero_unpaired is exempt. KTO has no prompt cap, so an over-long prompt eats its own completion.

The reference model follows DPO's rules — PEFT, precompute under EP/TP, or a frozen copy, which under EP/TP is a whole dense replica per rank (warned) (DPO — Reference model) — and so does a precompute resume (DPO — Resuming a precompute run). The policy, reference and KL sequence log-probs the trainer computes are summed in fp32; reference columns a dataset supplies keep their own precision (DPO — Log-prob precision). Under the default kto loss the saved digest covers the KL completions, which are built per per_device_train_batch_size, so a resume that changes it raises when the policy was built from the checkpoint and no separate reference exists. Any other resume (an adapter or merge-on-save checkpoint, a frozen reference copy) sweeps instead.

Under pipeline parallelism KTO runs only loss_type: apo_zero_unpaired (the default kto needs a world-global KL baseline no microbatch can compute), text-only, with precompute_ref_log_probs: true and a ref_logps column already in the train and every eval split; a live ref_model and compute_metrics are refused (Rejected combinations).

Launch

Bash

torchrun --nproc_per_node=8 scripts/training/preference/kto.py \
    examples/preference/qwen3_5/kto-qwen3.5-9b-kto-mix-14k.yaml

halo launch kto <config> --nproc 8 builds the same line. That recipe is dense, so it runs plain FSDP2 data parallel; add --expert_parallel_size=8 on an MoE checkpoint. Like the other padded-batch scripts, kto.py gives an unset attn_implementation the padded-batch default under reset_sinks: true: the loader's own pick on Blackwell (SDPA for a remote-code checkpoint, FA2 for a softcapped policy), SDPA elsewhere (Flash Attention).

Vision-language

Add an images/image column — or name one with images_field — to train a VLM checkpoint on {prompt, completion, label, images} rows, except on a family declaring refuses_trl_image_rows (Nemotron-H), refused before the load. Text-only data on a multimodal checkpoint stays on the text path. A prefix-LM checkpoint (PaliGemma) is refused on image rows before the model load (Modality support); train it on images with SFT or distillation.

A dataset that embeds {"type": "image"} parts in its messages while shipping no image column is refused: TRL would template them as text, expanding each part into pixel-less placeholder tokens. Move the images into an images column, or drop the parts.

Vision KTO is unpaired-only — a chosen/rejected column alongside images raises, so unpair the dataset first. precompute_ref_log_probs is rejected on vision rows, so under EP the reference is PEFT adapters or a frozen copy, and under TP, where PEFT is rejected, a frozen copy.

Testing a setup

Bash

torchrun --nproc_per_node=2 scripts/training/preference/kto.py <config> \
    --max_steps=5 --save_strategy=no --report_to=none

Covering tests: pytest tests/cpu/trainers tests/cpu/config -m cpu, plus tests/gpu/trainers/preference/test_kto.py, test_kto_fsdp_multi_gpu.py and, for pipeline parallelism on precomputed reference log-probs, tests/gpu/parallelism/pp/test_pp_preference_precompute.py --trainer kto and test_pp_trainer_checkpoints.py --trainer kto (save, exact resume, export).

What to watch

SignalReading
rewards/chosen, rewards/rejectedDesirable rows should rise, undesirable fall
rewards/marginsTheir difference; the separation KTO is buying
klWorld-gathered mean, clamped at ≥ 0; constant 0.0 under apo_zero_unpaired. A runaway climb means beta is too low for the LR

Failure signatures:

  • "Actual (not effective) batch size must be > 1" — raise per_device_train_batch_size and halve gradient_accumulation_steps.
  • One class dominating the gradient — set desirable_weight / undesirable_weight to the inverse label ratio.

Source on GitHubUpdated 18 November 2026