Halo
Halo is an LLM training toolkit built on HuggingFace TRL, Transformers, and Accelerate. It adds Expert, Context, Tensor, Expert-Tensor, and Pipeline parallelism (EP / CP / TP / ETP / PP) for MoE and dense models, plus pre-training and post-training methods: SFT, preference optimization (SMPO, DPO, KTO), the GRPO family (offline, online/RLVR, async with environments), distillation (teacher, self, online SDPG), reward modeling, classification, and embedding. SFT also trains the multi-token-prediction head a release ships (MTP heads, mtp_loss_weight).
Trainers subclass TRL/Transformers trainers, so any Hub model trains with the same TrainingArguments, Accelerate FSDP configs, and PEFT/LoRA — no checkpoint conversion, except a one-time BF16 dequantize of the FP8 releases (DeepSeek-V4, GLM-5 Next, Mistral 4; Models).
Start with the guides below, or jump to Training Methods, Parallelism, Models, Data, Optimization, Infrastructure, Contributing, or the Reference.
- Installation — Pull or build the Blackwell/Hopper image (the supported path, everything compiled in) and start a container.
- Quickstart — Launch an example config, watch it, use the result.
- Choosing a method — Match the data you have to a trainer.
- Configuration Reference — The YAML parser, CLI overrides, every field and environment variable.
GPU Training Theory covers the roofline and training-step bottlenecks behind every optimization lever — read it before tuning. For the design rationale and how the toolkit relates to TRL, Accelerate, Unsloth, Axolotl, MS-SWIFT, veRL, and Megatron-LM, see Why this framework.
Source on GitHubUpdated 18 November 2026