Expert parallelism

Expert parallelism splits the experts of a mixture-of-experts (MoE) model across GPUs. Each GPU holds a share of the experts, and every token travels to the GPU that owns its expert and back.

How it works

In a mixture-of-experts layer a router picks a few experts for each token. Under expert parallelism (EP) each GPU holds only a slice of the experts. After routing:

  1. Each token is dispatched to the GPU that holds its expert.
  2. The expert runs there.
  3. The result is combined back on the GPU the token came from.

Tokens move, batches do not. Each GPU still works on its own rows, so EP stacks with data parallelism.

Why it matters

The experts hold most of an MoE model's weights, so splitting them is the direct way to make a large MoE model fit. The price is the all-to-all traffic of the dispatch, which costs throughput.

In Halo

Set expert_parallel_size in the config, or pass --expert_parallel_size=N. This recipe pins it in the config:

Bash

halo launch sft examples/sft/qwen3_5/qwen3.5-35b-a3b-ultrachat-ep.yaml -n 8

GPUs split into dispatch groups of ep_size. GPUs that hold the same expert slice are data-parallel replicas, and their expert gradients are averaged after the backward pass.

  • On one node, EP must fill the NVLink domain or be ep_size=2. EP=4 on 8 GPUs is rejected; ep4 + etp2, with expert-tensor parallelism, is accepted.
  • EP can span nodes with --ep_scope=global on an RDMA (remote direct memory access) fabric.
  • Pick the lowest EP that fits, not the largest your GPUs allow.

See Parallelism.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL