Halo vs Megatron

Megatron-LM is NVIDIA's framework for training very large transformers, with models defined in its own code. Halo is a post-training framework that wraps Hugging Face models as they are, so no checkpoint is converted in or out.

The difference

Megatron-LM defines models in its own code and checkpoint format. A Hugging Face checkpoint is converted into that format to train, and converted back to use the result.

Halo applies distributed behavior as wrappers around the Hugging Face model, not as a fork of it. A Hugging Face model goes in and a Hugging Face model comes out, loadable with from_pretrained.

HaloMegatron-LM
Model codeThe Hugging Face model, wrappedMegatron's own
CheckpointsHugging Face in and outConverted in and out (Megatron Bridge)
A new model familyOne wrapper file for expert parallelism, most under 140 linesModel, config and conversion code that connects the family to Megatron's stack
Pipeline parallelismNot yet availableYes

In Halo, a family that trains under plain data parallelism needs no code. Laguna's wrapper reuses GLM-4's and is 39 lines. GPT-OSS is the outlier at about 360.

Measured

In the Halo tech report, on gpt-oss-20b with 4,096-token sequences at batch 1 on B300 GPUs, we measured Halo at 20,786 tokens per second per GPU and Megatron-LM at 8,645.

Which to pick

  • Megatron-LM to pre-train a very large model on a big cluster, when you need pipeline parallelism, or when your team already works in its model code.
  • Halo to post-train a Hugging Face model, above all a mixture-of-experts one, and get a Hugging Face model back. It also covers preference tuning and reinforcement learning with rollouts.

The model integration cost page counts what each Halo wrapper takes.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL