Tensor parallelism

Tensor parallelism splits each layer's weight matrices across GPUs, so every GPU holds a slice of every layer. A model too large for one GPU can then train, at the price of communication inside each layer.

How it works

A layer's weight matrix is cut into slices, one per GPU. Each GPU multiplies its slice by the input, and the partial results are concatenated or summed, depending on how the matrix was cut. In a transformer the attention projections and the feed-forward matrices are sliced this way, so each GPU holds a share of the attention heads and of the feed-forward units. Megatron-LM made this layout common.

The GPUs of a tensor-parallel group work on the same rows. They are parts of one model, not replicas, so the group counts once toward data parallelism.

When to use it

Use it for a dense model whose weights do not fit on the GPUs you have. The GPUs exchange activations inside every layer, so it needs the fastest links and is usually kept inside one machine. For a mixture-of-experts model, expert parallelism comes first, since the experts hold most of the weights.

In Halo

Set --tensor_parallel_size=N. The rules that apply:

  • It cannot leave the NVLink domain: the node, on a typical 8-GPU host.
  • It does not combine with context parallelism or expert-tensor parallelism.
  • LoRA (low-rank adaptation) does not combine with it, and QLoRA is rejected.
  • Family support varies. A dense model without a tp_plan raises at load.

See the parallelism rules.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL