Distributed training

Distributed training runs one training job on several GPUs or machines. The batch, the model's weights or a long sequence can each be split across them, and every split trades memory or time for communication.

What can be split

What it costs

Every split adds traffic between GPUs: gradients, activations or tokens. Inside one machine the GPUs are linked directly. Across machines the traffic crosses the cluster network, which is slower. Sharding buys capacity and costs throughput, so take the least of it that fits.

In Halo

Plain data parallelism needs no flag: -n 8 gives FSDP2 (Fully Sharded Data Parallel) sharding across 8 GPUs. The other modes are flags:

ModeFlag
Expert parallelism (EP)--expert_parallel_size=N
Context parallelism (CP)--context_parallel_size=N
Tensor parallelism (TP)--tensor_parallel_size=N
Expert-tensor parallelism (ETP)--expert_tensor_parallel_size=N

Halo validates the layout at startup and rejects an invalid shape, with an explanation, before touching the GPUs. See Parallelism.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL