Expert Tensor Parallelism (ETP)
expert_tp_size shards expert FFN weights only (gate_proj, up_proj, down_proj) across
NVLink-local groups, cutting per-rank expert memory without touching attention. Two shapes:
- Pure ETP (
ep_size=1, expert_tp_size>1): each expert's FFN is shardedexpert_tp_size-way; experts are replicated, not distributed. - EP+ETP (
ep_size>1ANDexpert_tp_size>1, experimental): experts distributedep_size-way and each expert's FFN shardedexpert_tp_size-way (ep_group_size = ep_size · expert_tp_size). The EP group splits intoexpert_tp_sizeDeepEP dispatch groups coupled by a strided expert-TP all-reduce.
Pure ETP also runs inside each pipeline stage (PP + ETP); EP+ETP does not compose with PP.
Use ETP when expert weights don't fit per rank after EP distribution and attention fits on one GPU. Per-rank expert memory drops ~`expert_tp_size`×; ranks in the same ETP group share a batch, reducing DP.
expert_tp_size excludes attention TP and CP
(why).
Forward path
Text
tokens, routing weights → SumGradAcrossGroup (identity fwd, all-reduce bwd)
→ dispatch (sub-EP via DeepEP)
→ expert_compute (TP-sharded weights → partial output)
→ combine (sub-EP via DeepEP)
→ SumAcrossGroup (all-reduce SUM fwd, identity bwd) → outputExpert compute is one grouped GEMM over the local shards on every family. On gpt-oss-20b against its per-expert loop (8× B300, s4096, 1.2.0, three rounds, peaks equal): ep4 × etp2 +68.6%, pure etp8 3.17×.
The expert-TP all-reduce sits at the layer boundary (token space), outside the DeepEP
dispatch→combine span (EPMoELayerBase._dispatch_compute_combine). This is exact because the
combine is linear in the expert outputs, so summing per-shard partials in token space equals summing
them in recv space (combine(Σ) = Σ combine).
It must stay outside: an all-reduce between dispatch and combine couples the dispatch groups inside DeepEP's intranode combine barrier, and under FSDP2 multi-stream drift they form a circular wait and the barrier times out.
The Megatron-LM scatter-gather pattern, a matched autograd pair from halo/distributed/grad_reduce.py, both
shared with TP — the scatter half SumGradAcrossGroup, the gather half SumAcrossGroup:
SumGradAcrossGroup— identity forward;all_reduce(SUM)backward overexpert_tp_group. It is applied to both the layer input tokens and the routing weights.The gate multiply happens on this rank's partial expert output, so
d(loss)/d(weights)here is a partial whose true value is the sum over the group. Without the second application the router, and every upstream tensor reached through it, trains on a gradient scaled by1/expert_tp_size, silently.SumAcrossGroup—all_reduce(SUM)forward on the combined output to sum the partial expert outputs; identity backward, correct becauseSumGradAcrossGroupowns the input-gradient reduction and each shard's weight gradient is independently correct.
Gradient divisors (the trainer's expert sweep, _run_deferred_grad_sweep in halo/trainers/parallelism/grad_sync.py, and build_dp_average_grad_hook in halo/distributed/expert_parallel/moe_layer/grad_sync.py):
Under PP every world_size here is the stage's rank count.
Quick start
Bash
# Pure ETP on 8 GPUs: experts replicated, expert FFN sharded 8-way (ep_group_size=8).
# CLI flags override any expert_parallel_size set in the YAML.
torchrun --nproc_per_node=8 scripts/training/sft.py \
examples/sft/gptoss/gptoss-20b-multinode-ep.yaml \
--expert_parallel_size=1 \
--expert_tensor_parallel_size=8 \
--use_grouped_gemm=true
# 2-GPU correctness test (expert_tp_size=2, ep_size=1)
torchrun --nproc_per_node=2 \
tests/gpu/parallelism/ep/test_ep_modes_vs_reference.py --model gpt_oss_20b --mode etpValidation rules
Enforced in halo/distributed/parallelism_rules.py before model loading:
expert_tp_sizemust divide the NVLink domain (=gpus_per_nodeon a standard node, the rack on NVL72).ep_group_sizemust divide the NVLink domain (ep_scope=node). Underep_scope=globalit must tile the world as equal contiguous per-domain blocks — divisibility alone is not sufficient (Multi-Node).Cross-node EP (
ep_scope=global) + ETP is supported as one ETP group per NVLink domain only:expert_tp_size == EP members per domainandep_size == domains spanned. A finer split that would straddle a domain boundary is rejected — the ETP all-reduce must stay on NVLink.Multi-domain multi-group EP+ETP is rejected — expert-TP keeps the deferred cross-replica DP path (
is_deferred_dp) off, so FSDP2's DP-wide reduce-scatter would race the narrower DeepEP combine across domains.This rule, not the one above, is what refuses
ep2+etp4andep4+etp2on 2×8 (both leaveep_group_size=8under a 16-rank world), at either scope;ep2+etp8is the working shape.On a single NVLink domain the EP rule keys on
ep_group_size:ep4+etp2on 8 GPUs fills the domain and passes, whileep_size > 2withep_group_sizeshort of the domain is refused like bareep4(ep3+etp2on 12; single-domain rule).ep2+etp2on 8 passes: its dispatch groups are 2 ranks wide.make benchmeasuresep4+etp2; the GPU test matrix runsep2+etp4(the 8-GPU Mistral4 and Cohere2 MoE matrices,tests/gpu/manifest.py): the same 8-way split through 2-rank dispatch groups, at DP 2 instead of 4.
ETP reduces data parallelism — within a stage it counts toward the max of TP, CP and ETP
(Stacking the dimensions). On 8 GPUs
(ep_group_size = 8 throughout): EP-only ep8 → DP 8; ep4+etp2 → DP 4; ep2+etp4 → DP 2; pure
ETP etp8 → DP 1.
Process groups (EP+ETP combo)
EPConfig._build_expert_tp_groups builds one of two layouts by EP scope, both keeping the ETP
all-reduce on NVLink.
Node-local EP (ep_scope=node) — the whole EP group is in one domain, so the DeepEP dispatch
takes the contiguous part: dispatch_ep_rank = ep_rank % ep_size, expert_tp_rank = ep_rank // ep_size. Dispatch groups are contiguous ep_size-rank chunks (DeepEP requires contiguous P2P
buffers); ETP groups are strided, one rank per chunk. On 8 GPUs with ep_size=4, expert_tp_size=2:
Text
EP group: [0,1,2,3,4,5,6,7]
Sub-EP (dispatch): [0,1,2,3] [4,5,6,7] ← DeepEP, size = ep_size = 4
ETP groups: [0,4] [1,5] [2,6] [3,7] ← NCCL all-reduce of FFN shardsCross-node EP (ep_scope=global) — the layout mirrors: each domain's contiguous block is one
ETP group, and the dispatch group strides across domains over the internode Gin path.
expert_tp_rank = ep_rank % expert_tp_size, dispatch_ep_rank = ep_rank // expert_tp_size. On 16
GPUs (2×8) with ep_size=2, expert_tp_size=8:
Text
EP group: [0..15] (domain0 [0-7], domain1 [8-15])
ETP groups: [0-7] [8-15] ← node-local NCCL all-reduce
Sub-EP (dispatch): [0,8] [1,9] ... [7,15] ← cross-node DeepEP, size = ep_size = 2This is the memory-lean cross-node MoE shape: per-rank expert memory matches a much larger pure-EP
group. On two 8-GPU B300 nodes over RDMA (before 1.2.0), gpt-oss-20b at ep_size=2, expert_tp_size=8
holds ~18 GB per rank, the footprint of EP=16. Its eight ETP partners share one batch, so it buys expert
memory, not throughput. Validate convergence on your own config before a long run.
Weight sharding
ETP shards the intermediate dimension M of expert FFN weights; the recipe is keyed on the model's
expert layout (halo/distributed/expert_parallel/moe_layer/compute.py + families/).
GptOss (fused interleaved GLU) —
gate_up_proj [E, H, 2M]is[g0,u0,g1,u1,…]. De-interleave (gate = [...,::2],up = [...,1::2]), shard each along dim 2 to[E, H, M/tp]; sharddown_proj [E, M, H]dim 1.Fused-GLU contiguous halves (every family whose block stores a fused
gate_up_proj, through the base_init_fused_glu_params) —gate_up_proj [E, H, 2M]is[gate(M) | up(M)].Slicing the fused tensor along dim 2 would mismatch the gate/up pairing, so ETP splits the halves first and stores separate
gate_proj/up_proj/down_projparams.Qwen3 / Bailing (separate GLU projections) — the base
_store_separate_glu_paramsshardsMand stores matmul-conventiongate_proj/up_proj [E, H, M/tp],down_proj [E, M/tp, H]for Grouped GEMM.Nemotron-H (no gate, on a latent) —
up_proj [E, L, M/tp],down_proj [E, M/tp, L], with the expert-TP sums inside the latent projections.
Checkpoints
Save formats, resume paths, and merge scripts: Checkpoints. The
ETP-unique step is that a gathered save all-gathers each expert attribute's TP shards within
expert_tp_group (EPGatherMixin.EXPERT_TP_SHARD_DIMS) before the dispatch-EP gather, giving the full
runtime tensors; the family's ExpertLayout then writes them as it would without ETP:
- GptOss re-interleaves.
- Contiguous-halves families concatenate gate and up along dim 2 and transpose back from matmul convention.
- GLM4/GLM-5 Next/Laguna/LFM2/Qwen3-Next (
FUSED_SPLIT_PER_EXPERT) split that to per-expert names. - Qwen3/Bailing (
SEPARATE_GLU) write per-expertexperts.{i}.{gate,up,down}_proj.weight. - Nemotron-H (
SEPARATE_MLP) writes per-expertexperts.{i}.{up,down}_proj.weight.
The result is a standard HuggingFace checkpoint; ETP sharding is re-applied automatically at model
patching when expert_tp_size > 1. Per-rank sharded EP save is rejected — see below.
Limitations
Trainers. No trainer refuses ETP or EP. Matrix: Trainer Compatibility.
Models. Every EP-capable MoE family
(roster). ETP shards expert FFN
weights only: the router stays replicated in every family, whether it sits outside the EP layer
and is FSDP-managed (Gemma 4) or inside it and is synced by the EP router hook (everyone else,
including Zaya). A dense model raises at load: pure ETP has ep_group_size > 1, which
reject_ep_without_family refuses where no EP layer wraps.
expert_tp_size must divide each expert's intermediate size, else the top units of every expert
would be dropped silently. reject_unsupported_model checks it off
config.json at the top of the model load, through resolve_expert_ffn_width.
That helper is the one home for "what does ETP actually shard", so the config-time gate and the layer that does the split cannot disagree on which config spelling carries the per-expert width.
Axis combinations. Pure ETP, EP+ETP, and pure ETP inside PP stages (allowlist).
Knobs. ETP inherits every EP knob rule (has_ep_group is
ep_group_size > 1, which pure ETP satisfies). On top of those:
The same-token invariant ETP depends on — partners must hold identical batches, since
SumAcrossGroup sums element-wise in token space — is maintained by the rank layout, the
DP-rank-sharded dataloader, and the ETP-leader rollout broadcast. It is never asserted at
runtime: a custom data path that bypasses those would sum unrelated tokens with no error.
Troubleshooting
- "Node-local EP group size (N) cannot exceed the NVLink domain (M)" —
ep_size * expert_tp_sizemust fit one domain underep_scope=node. Reduce either, or useep_scope=global. - "Cross-node EP+ETP supports one ETP group per NVLink domain only" — set
expert_tensor_parallel_sizeto the EP members per domain andexpert_parallel_sizeto the domain count (both quoted in the message), or useep_scope=node. - "Sharded EP save (save_sharded_ep=True) is not supported with expert_tensor_parallel_size=N" — use the default gathered save.
- OOM — raise
expert_tp_size, or switch to EP+TP if attention is also a bottleneck (the two are mutually exclusive).
Source on GitHubUpdated 18 November 2026