Megatron-LM is NVIDIA's open-source framework for training very large transformer models across many GPUs. It splits a model with tensor and pipeline parallelism, and it defines models in its own code.
What it is
Megatron Core is a library of GPU-optimized building blocks: transformer layers and parallelism. Megatron-LM is the reference example built on it, with ready-made training scripts.
Megatron is known for two ways of splitting a model that is too large for one GPU:
- Tensor parallelism. The weight matrices of each layer are split across GPUs.
- Pipeline parallelism. The layers are split into stages, and each stage runs on its own GPUs.
Both combine with data parallelism. Megatron Core also supports context and expert parallelism.
When it fits
Megatron fits pre-training and other very large runs on big clusters, by teams that can work in its model code. Models are defined in Megatron's own code and checkpoint format. NVIDIA's Megatron Bridge converts checkpoints between that format and Hugging Face, in both directions.
Megatron and Halo
Halo takes the other route. It wraps the Hugging Face model instead of reimplementing it, so a Hugging Face model goes in and comes out with no conversion. Expert parallelism for a new model family is one wrapper file, and most stay under 140 lines.
In the Halo tech report, on gpt-oss-20b with 4,096-token sequences at batch 1 on B300 GPUs, we measured Halo at 20,786 tokens per second per GPU and Megatron-LM at 8,645. See Halo vs Megatron.
