Halo

Halo is an LLM training toolkit built on HuggingFace TRL, Transformers, and Accelerate. It adds Expert, Context, Tensor, Expert-Tensor, and Pipeline parallelism (EP / CP / TP / ETP / PP) for MoE and dense models, plus pre-training and post-training methods: SFT, preference optimization (SMPO, DPO, KTO), the GRPO family (offline, online/RLVR, async with environments), distillation (teacher, self, online SDPG), reward modeling, classification, and embedding. SFT also trains the multi-token-prediction head a release ships (MTP heads, mtp_loss_weight).

Trainers subclass TRL/Transformers trainers, so any Hub model trains with the same TrainingArguments, Accelerate FSDP configs, and PEFT/LoRA — no checkpoint conversion, except a one-time BF16 dequantize of the FP8 releases (DeepSeek-V4, GLM-5 Next, Mistral 4; Models).

Start with the guides below, or jump to Training Methods, Parallelism, Models, Data, Optimization, Infrastructure, Contributing, or the Reference.

  • Installation — Pull or build the Blackwell/Hopper image (the supported path, everything compiled in) and start a container.
  • Quickstart — Launch an example config, watch it, use the result.
  • Choosing a method — Match the data you have to a trainer.
  • Configuration Reference — The YAML parser, CLI overrides, every field and environment variable.

GPU Training Theory covers the roofline and training-step bottlenecks behind every optimization lever — read it before tuning. For the design rationale and how the toolkit relates to TRL, Accelerate, Unsloth, Axolotl, MS-SWIFT, veRL, and Megatron-LM, see Why this framework.

Source on GitHubUpdated 18 November 2026