Expert-tensor parallelism slices each expert of a mixture-of-experts (MoE) model across GPUs, so every GPU holds a part of every expert. It applies tensor parallelism to the expert weights alone.
How it works
An expert in a mixture-of-experts layer is a small feed-forward network (FFN): a few weight matrices. Expert parallelism (EP) keeps each expert whole and gives each GPU some of them. Expert-tensor parallelism (ETP) cuts each expert's matrices into slices, the way tensor parallelism cuts a dense layer, and gives each GPU one slice of every expert.
Only the experts are sliced. Attention and the other dense weights are not.
When to use it
Use it when the expert weights are the memory problem and expert parallelism alone does not solve it, for example when the experts are so large that even a share of them is too much for one GPU.
The two combine: EP divides the experts among GPUs, and ETP slices each one.
In Halo
Set --expert_tensor_parallel_size=N.
- ETP runs alone or with EP. It does not combine with tensor parallelism or context parallelism.
- For a 4-way expert split on 8 GPUs,
ep4 + etp2is the shape Halo accepts. Plain EP=4 on 8 GPUs is rejected. - LoRA (low-rank adaptation) adapters on the expert projections are rejected once
expert_tensor_parallel_size > 1. - The supported matrix marks ETP as experimental.
The performance page gives one example of what it buys: a 119B MoE trains on four GPUs at ep2 + etp2. See Parallelism.
