Choosing a method

Pick a method by the data you have and what you want the model to do. Every method uses the same YAML format and parallelism stack, so switching is a config change. The last column is the name you pass to halo launch (Quickstart).

MethodUse whenDatahalo launch
SFTTeach format, behavior or a domain from example conversationsconversations (prompt)sft
SFT (VLM)Same, with images. sft takes the VLM path when a multimodal checkpoint meets image dataimage conversationssft
ClassificationPredict a sequence-level label (safety, topic, toxicity)conversation (prompt) or raw text, + labelclassification
Reward modelingTrain a Bradley-Terry scorer to use laterchosen / rejected pairsreward/reward-model
DPOAlign on preference pairs, held close to a reference model by a KL termchosen, rejected (+ optional prompt)dpo
SMPOAlign on preference pairs without a reference modelchosen, rejected (+ optional prompt)smpo
KTOYour feedback is unpaired thumbs-up / thumbs-down, not pairsprompt, completion, labelkto
Offline GRPOYou already have scored completions; no live generationprompt, completions, rewardsgrpo/offline
Online GRPO (RLVR)The model generates and earns rule-based, verifiable rewards (math, format)prompt, answergrpo/online
Async GRPO with EnvironmentsMulti-turn, tool-using, agentic trajectoriesprompt, plus answer where the environment grades onegrpo/environmental
Teacher distillationCompress a separate, larger teacher into a smaller studentconversationsteacher-distill
Self-distillationImprove the model with a privileged answer hint, no second modelconversations + answerself-distill
Online SDPGOn-policy self-distillationprompt, answergrpo/online --use_sdpg=true
EmbeddingTune for retrieval, similarity or clusteringpairs, triplets or scored pairsembedding

Similar methods compared

  • DPO, SMPO or KTO. All three train on preference data. DPO scores against a frozen reference model, a second copy in memory unless you train LoRA or precompute the reference log-probs. SMPO needs no reference: its bounded loss stops pushing a pair once it is well separated, and an SFT term protects generation quality. KTO takes unpaired thumbs-up / thumbs-down labels and keeps a reference under DPO's rules.
  • Preference training or reward modeling. Same pairs, different output. DPO, SMPO and KTO change the policy. reward/reward-model trains a scorer you use later, for rejection sampling or as a GRPO reward.
  • Offline, online or async GRPO. Offline GRPO trains on pre-scored completions and generates nothing. Online GRPO (RLVR) generates single-turn completions and scores them with verifiable rewards. Async GRPO with environments adds multi-turn tool use and overlaps rollouts with training. Each scores against a reference only while its KL term is on (kl_beta offline, beta online and async).
  • Teacher or self-distillation. teacher-distill learns from a separate, larger model. Self-distillation uses the same model, given the answer as a hint, as its own teacher: offline (self-distill) or on-policy (grpo/online --use_sdpg=true).

Which trainer takes which parallelism mode is in the Supported matrix.

Pretraining

Pretraining runs sft on raw text that you tokenize offline first (halo run prepare-dataset --mode text). To train from random weights, set init_from_scratch: true.

From-scratch runs support data parallelism only (one GPU, DDP or FSDP2) and reject EP, CP, TP, ETP and PP. To use those, create the random-init checkpoint in a separate step and train from it. See Pretraining ↗.

Extra infrastructure

Online GRPO and online SDPG generate with a separate vLLM server, async GRPO with environments with vLLM or SGLang (Rollout servers). Every other method trains without a rollout server or Ray.

Async GRPO also runs Ray rollout actors. With two or more servers, or one that opts in, it overlaps rollouts with training through a prefetch queue. It can generate with SGLang instead (rollout_backend: sglang), with the limits listed in the Supported matrix.

Every hyperparameter, per method, is in the training methods reference ↗.

Source on GitHubUpdated 18 November 2026