Choosing a method
Pick a method by the data you have and what you want the model to do. Every
method uses the same YAML format and parallelism stack, so switching is a config
change. The last column is the name you pass to halo launch
(Quickstart).
Similar methods compared
- DPO, SMPO or KTO. All three train on preference data. DPO scores against a frozen reference model, a second copy in memory unless you train LoRA or precompute the reference log-probs. SMPO needs no reference: its bounded loss stops pushing a pair once it is well separated, and an SFT term protects generation quality. KTO takes unpaired thumbs-up / thumbs-down labels and keeps a reference under DPO's rules.
- Preference training or reward modeling. Same pairs, different output.
DPO, SMPO and KTO change the policy.
reward/reward-modeltrains a scorer you use later, for rejection sampling or as a GRPO reward. - Offline, online or async GRPO. Offline GRPO trains on pre-scored
completions and generates nothing. Online GRPO (RLVR) generates single-turn
completions and scores them with verifiable rewards. Async GRPO with
environments adds multi-turn tool use and overlaps rollouts with training.
Each scores against a reference only while its KL term is on (
kl_betaoffline,betaonline and async). - Teacher or self-distillation.
teacher-distilllearns from a separate, larger model. Self-distillation uses the same model, given the answer as a hint, as its own teacher: offline (self-distill) or on-policy (grpo/online --use_sdpg=true).
Which trainer takes which parallelism mode is in the Supported matrix.
Pretraining
Pretraining runs sft on raw text that you tokenize offline first
(halo run prepare-dataset --mode text). To train from random weights, set
init_from_scratch: true.
From-scratch runs support data parallelism only (one GPU, DDP or FSDP2) and reject EP, CP, TP, ETP and PP. To use those, create the random-init checkpoint in a separate step and train from it. See Pretraining ↗.
Extra infrastructure
Online GRPO and online SDPG generate with a separate vLLM server, async GRPO with environments with vLLM or SGLang (Rollout servers). Every other method trains without a rollout server or Ray.
Async GRPO also runs Ray rollout actors. With two or more servers, or one that
opts in, it overlaps rollouts with training through a prefetch queue. It can
generate with SGLang instead (rollout_backend: sglang), with the limits
listed in the Supported matrix.
Every hyperparameter, per method, is in the training methods reference ↗.
Source on GitHubUpdated 18 November 2026