# Expert parallelism

Expert parallelism splits the experts of a mixture-of-experts (MoE) model across GPUs. Each GPU holds a share of the experts, and every token travels to the GPU that owns its expert and back.

## How it works

In a [mixture-of-experts](/glossary/mixture-of-experts) layer a router picks a few experts for each token. Under expert parallelism (EP) each GPU holds only a slice of the experts. After routing:

1. Each token is dispatched to the GPU that holds its expert.
2. The expert runs there.
3. The result is combined back on the GPU the token came from.

Tokens move, batches do not. Each GPU still works on its own rows, so EP stacks with [data parallelism](/glossary/data-parallelism).

## Why it matters

The experts hold most of an MoE model's weights, so splitting them is the direct way to make a large MoE model fit. The price is the all-to-all traffic of the dispatch, which costs throughput.

## In Halo

Set `expert_parallel_size` in the config, or pass `--expert_parallel_size=N`. This recipe pins it in the config:

```bash
halo launch sft examples/sft/qwen3_5/qwen3.5-35b-a3b-ultrachat-ep.yaml -n 8
```

GPUs split into dispatch groups of `ep_size`. GPUs that hold the same expert slice are data-parallel replicas, and their expert gradients are averaged after the backward pass.

- On one node, EP must fill the NVLink domain or be `ep_size=2`. EP=4 on 8 GPUs is rejected; `ep4 + etp2`, with [expert-tensor parallelism](/glossary/expert-tensor-parallelism), is accepted.
- EP can span nodes with `--ep_scope=global` on an RDMA (remote direct memory access) fabric.
- Pick the lowest EP that fits, not the largest your GPUs allow.

See [Parallelism](/halo/docs/parallelism).

