# Halo

Halo is an LLM training toolkit built on HuggingFace TRL, Transformers, and Accelerate. It adds Expert, Context, Tensor, Expert-Tensor, and Pipeline parallelism (EP / CP / TP / ETP / PP) for MoE and dense models, plus pre-training and post-training methods: SFT, preference optimization (SMPO, DPO, KTO), the GRPO family (offline, online/RLVR, async with environments), distillation (teacher, self, online SDPG), reward modeling, classification, and embedding. SFT also trains the multi-token-prediction head a release ships ([MTP heads](https://whitecircle.com/halo/docs/agent-docs/optimization/mtp-heads.md), `mtp_loss_weight`).

Trainers subclass TRL/Transformers trainers, so any Hub model trains with the same `TrainingArguments`, Accelerate FSDP configs, and PEFT/LoRA — no checkpoint conversion, except a one-time BF16 dequantize of the FP8 releases (DeepSeek-V4, GLM-5 Next, Mistral 4; [Models](https://whitecircle.com/halo/docs/agent-docs/models.md)).

Start with the guides below, or jump to [Training Methods](https://whitecircle.com/halo/docs/agent-docs/training-methods.md), [Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism.md), [Models](https://whitecircle.com/halo/docs/agent-docs/models.md), [Data](https://whitecircle.com/halo/docs/agent-docs/data.md), [Optimization](https://whitecircle.com/halo/docs/agent-docs/optimization.md), [Infrastructure](https://whitecircle.com/halo/docs/agent-docs/infrastructure.md), [Contributing](https://whitecircle.com/halo/docs/agent-docs/contributing.md), or the [Reference](https://whitecircle.com/halo/docs/agent-docs/reference.md).

- **[Installation](https://whitecircle.com/halo/docs/installation.md)** — Pull or build the Blackwell/Hopper image (the supported path, everything compiled in) and start a container.
- **[Quickstart](https://whitecircle.com/halo/docs/quickstart.md)** — Launch an example config, watch it, use the result.
- **[Choosing a method](https://whitecircle.com/halo/docs/choosing-a-method.md)** — Match the data you have to a trainer.
- **[Configuration Reference](https://whitecircle.com/halo/docs/agent-docs/reference/configuration-reference.md)** — The YAML parser, CLI overrides, every field and environment variable.

[GPU Training Theory](https://whitecircle.com/halo/docs/agent-docs/reference/gpu-training-theory.md) covers the roofline and training-step bottlenecks behind every optimization lever — read it before tuning. For the design rationale and how the toolkit relates to TRL, Accelerate, Unsloth, Axolotl, MS-SWIFT, veRL, and Megatron-LM, see [Why this framework](https://whitecircle.com/halo/docs/agent-docs/reference/why-this-framework.md).
