Training Methods

Every method is YAML-driven, runs through the same distributed parallelism system, and ships example configs under examples/.

Methods at a glance

MethodTrainerDataset columnsWhen to use
SFTDistributedSFTTrainerprompt (message list)Instruction tuning, text or vision-language; also pre-training, same trainer and script, and a release's MTP head (mtp_loss_weight)
DPODistributedDPOTrainerprompt, chosen, rejectedPaired preference data, against a reference model; 15 TRL loss_type values, combinable
KTODistributedKTOTrainerprompt, completion, labelUnpaired binary feedback — one completion, thumbs up or down
SMPOSMPOTrainerprompt, chosen, rejectedPreference tuning with no reference model to hold in memory
Offline GRPOOfflineGRPOTrainerprompt, completions, rewardsCompletions already generated and scored
Online GRPO (RLVR)DistributedOnlineGRPOTrainerprompt, answerSingle-turn verifiable rewards (math, format) against a live vLLM server
Async GRPO with EnvironmentsDistributedAsyncEnvironmentalGRPOTrainerprompt (+ answer where the environment grades against one)Multi-turn RL with tools, sandboxes or a judge, on vLLM or SGLang
Reward modelingDistributedRewardTrainerchosen, rejectedA Bradley-Terry scorer to rank completions
ClassificationClassificationTrainerprompt or text_field, labelSingle- and multi-label sequence classification
DistillationDistributedTeacherDistillationTrainer, DistributedSelfDistillationTrainer, DistributedSDPGTrainerSFT conversations (conversation_field), or prompt, answer (online SDPG)Compress a teacher, or self-distill from a privileged hint (offline or online SDPG)
EmbeddingEmbeddingTrainertext pairs or triplets, optional label / scoreRetrieval and similarity models, 10 SBERT losses and Matryoshka

Family overviews: Preference · GRPO.

Column types and per-script defaults: Dataset Formats. Scripts and their flags: Scripts Reference. Undecided: Choosing a method.

Vision-language support

SFT, DPO, KTO, SMPO, both off-policy distillation scripts and reward modeling (on families with a sequence-classification head) take images, PaliGemma only through SFT and distillation; classification, embedding and the GRPO family are text-only. One script serves both modalities — the model class follows the checkpoint and the data path follows the run, so a text-only dataset on a multimodal checkpoint trains through the text pipeline.

Per-method rules: Modality support. text_only_model: true loads a multimodal checkpoint through its text-only sibling (field reference).

Parallelism support

No trainer refuses EP, TP or ETP (Trainer Compatibility). Context parallelism is declare-to-enable per trainer (SUPPORTS_CP): SFT, SMPO and offline GRPO full fine-tuning support it; online and environment GRPO do not.

Pipeline parallelism is declare-to-enable the same way (PP_REFUSAL = None): SFT, SMPO, DPO and KTO (precompute-only), offline GRPO, reward and classification support it (Pipeline Parallelism).

Full trainer × mode matrix: Trainer Compatibility.

Experiment tracking

Every script reports to the backend named by report_to, logging loss, learning rate and gradient norm automatically. The toolkit callbacks add throughput, MoE expert load, eval-time generations and parameter stats (Training Callbacks).

YAML

report_to: wandb        # wandb | clearml | tensorboard | none
run_name: my-sft-run    # optional; names the run in the tracker

Source on GitHubUpdated 18 November 2026