To choose a fine-tuning method, start from the data you have. Example answers call for supervised fine-tuning, pairs of better and worse answers for preference tuning, and a way to score answers for reinforcement fine-tuning.
Steps
- List the data you have. The method follows from the data and the goal, not the other way round.
- Match the data to a method. The table below has the mapping.
- Narrow the choice. With preference pairs, DPO (Direct Preference Optimization) keeps a frozen reference model in memory and SMPO does not. With scores, use offline GRPO (Group Relative Policy Optimization) if the completions are already scored, online GRPO for single-turn answers a rule can check, and async GRPO for multi-turn tool use.
- Decide what you want out. The same pairs can change the model itself, or train a reward model: a scorer to use later.
- Start simple. Most projects start with supervised fine-tuning and add preference tuning or reinforcement learning after it.
Which to pick
With Halo
Every method uses the same YAML config format and the same parallelism, so switching is a config change, not a rewrite. The method is the name after halo launch: sft, dpo, smpo, kto, rewards, offline-grpo, rlvr, environmental-grpo, teacher-distill, self-distill, classification or embedding. Online and async GRPO also need a separate rollout server. Choosing a method has the full table.
