Expert parallelism splits the experts of a mixture-of-experts (MoE) model across GPUs. Each GPU holds a share of the experts, and every token travels to the GPU that owns its expert and back.
How it works
In a mixture-of-experts layer a router picks a few experts for each token. Under expert parallelism (EP) each GPU holds only a slice of the experts. After routing:
- Each token is dispatched to the GPU that holds its expert.
- The expert runs there.
- The result is combined back on the GPU the token came from.
Tokens move, batches do not. Each GPU still works on its own rows, so EP stacks with data parallelism.
Why it matters
The experts hold most of an MoE model's weights, so splitting them is the direct way to make a large MoE model fit. The price is the all-to-all traffic of the dispatch, which costs throughput.
In Halo
Set expert_parallel_size in the config, or pass --expert_parallel_size=N. This recipe pins it in the config:
Bash
halo launch sft examples/sft/qwen3_5/qwen3.5-35b-a3b-ultrachat-ep.yaml -n 8GPUs split into dispatch groups of ep_size. GPUs that hold the same expert slice are data-parallel replicas, and their expert gradients are averaged after the backward pass.
- On one node, EP must fill the NVLink domain or be
ep_size=2. EP=4 on 8 GPUs is rejected;ep4 + etp2, with expert-tensor parallelism, is accepted. - EP can span nodes with
--ep_scope=globalon an RDMA (remote direct memory access) fabric. - Pick the lowest EP that fits, not the largest your GPUs allow.
See Parallelism.
