Megatron-LM is NVIDIA's framework for training very large transformers, with models defined in its own code. Halo is a post-training framework that wraps Hugging Face models as they are, so no checkpoint is converted in or out.
The difference
Megatron-LM defines models in its own code and checkpoint format. A Hugging Face checkpoint is converted into that format to train, and converted back to use the result.
Halo applies distributed behavior as wrappers around the Hugging Face model, not as a fork of it. A Hugging Face model goes in and a Hugging Face model comes out, loadable with from_pretrained.
In Halo, a family that trains under plain data parallelism needs no code. Laguna's wrapper reuses GLM-4's and is 39 lines. GPT-OSS is the outlier at about 360.
Measured
In the Halo tech report, on gpt-oss-20b with 4,096-token sequences at batch 1 on B300 GPUs, we measured Halo at 20,786 tokens per second per GPU and Megatron-LM at 8,645.
Which to pick
- Megatron-LM to pre-train a very large model on a big cluster, when you need pipeline parallelism, or when your team already works in its model code.
- Halo to post-train a Hugging Face model, above all a mixture-of-experts one, and get a Hugging Face model back. It also covers preference tuning and reinforcement learning with rollouts.
The model integration cost page counts what each Halo wrapper takes.
