How to choose a fine-tuning method

To choose a fine-tuning method, start from the data you have. Example answers call for supervised fine-tuning, pairs of better and worse answers for preference tuning, and a way to score answers for reinforcement fine-tuning.

Steps

  1. List the data you have. The method follows from the data and the goal, not the other way round.
  2. Match the data to a method. The table below has the mapping.
  3. Narrow the choice. With preference pairs, DPO (Direct Preference Optimization) keeps a frozen reference model in memory and SMPO does not. With scores, use offline GRPO (Group Relative Policy Optimization) if the completions are already scored, online GRPO for single-turn answers a rule can check, and async GRPO for multi-turn tool use.
  4. Decide what you want out. The same pairs can change the model itself, or train a reward model: a scorer to use later.
  5. Start simple. Most projects start with supervised fine-tuning and add preference tuning or reinforcement learning after it.

Which to pick

You haveUse
Example conversationsSupervised fine-tuning
Chosen and rejected answersPreference tuning with DPO or SMPO
Single answers marked good or badKTO (Kahneman-Tversky Optimization)
A way to score answersReinforcement fine-tuning with GRPO
A larger teacher modelKnowledge distillation
Text with labelsClassification
Text pairs, triplets or scored pairsEmbedding fine-tuning

With Halo

Every method uses the same YAML config format and the same parallelism, so switching is a config change, not a rewrite. The method is the name after halo launch: sft, dpo, smpo, kto, rewards, offline-grpo, rlvr, environmental-grpo, teacher-distill, self-distill, classification or embedding. Online and async GRPO also need a separate rollout server. Choosing a method has the full table.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL