Distributed training runs one training job on several GPUs or machines. The batch, the model's weights or a long sequence can each be split across them, and every split trades memory or time for communication.
What can be split
- The batch. Data parallelism gives each GPU different rows and averages the gradients. It adds throughput.
- The weights. Tensor parallelism slices each layer's matrices. In a mixture-of-experts (MoE) model, expert parallelism gives each GPU a share of the experts, and expert-tensor parallelism slices the experts themselves. These make a large model fit.
- The sequence. Context parallelism gives each GPU a chunk of one long sequence. It makes a long context fit.
What it costs
Every split adds traffic between GPUs: gradients, activations or tokens. Inside one machine the GPUs are linked directly. Across machines the traffic crosses the cluster network, which is slower. Sharding buys capacity and costs throughput, so take the least of it that fits.
In Halo
Plain data parallelism needs no flag: -n 8 gives FSDP2 (Fully Sharded Data Parallel) sharding across 8 GPUs. The other modes are flags:
Halo validates the layout at startup and rejects an invalid shape, with an explanation, before touching the GPUs. See Parallelism.
