Training Methods
Each method is one script under scripts/training/ and one name you pass to halo launch. Every
method reads a YAML config, takes a HuggingFace checkpoint and a dataset, and writes a HuggingFace
checkpoint. If you are still deciding, Choosing a Method maps the data you
have to the trainer that reads it.
What every method shares
- Config. A config is a flat YAML file of trainer fields, any of which can be overridden on the command line after the config path (Writing a config).
- Data. Datasets come from the Hub, a local path or
s3://through the samedataset:field, and a list mixes them (Datasets). The columns each method reads differ. - Parallelism. Expert, tensor and expert-tensor parallelism work on every trainer; which ones also take context or pipeline parallelism is in the Supported matrix.
- Checkpoints. They load with
from_pretrainedand upload to the Hub without their training state (Uploading). Some MoE families need a conversion before vLLM or SGLang can serve them. A LoRA run writes an adapter you merge with one command; expert LoRA under EP folds into the weights at save time instead (merge_expert_lora_on_save: true). See Checkpoints & Export.
Every hyperparameter and refusal per method is in the training-methods reference ↗.
Source on GitHubUpdated 18 November 2026