Parallelism

On top of data parallelism, four strategies address different bottlenecks: model size (TP), sequence length (CP), expert distribution (EP), or depth across NVLink domains (PP). They combine within the supported set below. Each mode is a communication pattern with a price: GPU Training Theory §9.

Multi-GPU DP modes run on FSDP2 for gradient sync (pure TP at DP=1 and single-GPU skip it); DistributedTrainerMixin selects the sharding strategy from the active mode. See the Data Parallelism Guide for launchers and Accelerate configs.

Parallelism modes

  • Expert parallelism (EP) — distributes MoE expert layers across GPUs via DeepEP all-to-all (the router stays replicated on every rank). Each GPU holds a subset of experts and routes tokens to the owning GPU. For MoE models; see Expert Parallelism Guide.
  • Tensor parallelism (TP) — shards dense attention and MLP weights across GPUs via DTensor with all-reduce. For large dense models (Llama 70B+, Mistral Large) that exceed single-GPU memory: plain FSDP2 and DDP load the whole model onto every GPU first (why), and TP's HF tp_plan load, which shards as it reads, is the route past that inside one NVLink domain (PP's stage loader is the one across domains); see Tensor Parallelism Guide.
  • Context parallelism (CP) — splits long sequences across GPUs via Ulysses-style attention with all-to-all. For long-context training (32K+ tokens); see Context Parallelism Guide.
  • Pipeline parallelism (PP) — splits decoder layers into stages, one contiguous rank block each, and is the only axis meant to cross NVLink domains. Off by default (why). See Pipeline Parallelism.

Which MoE families each axis wraps: Supported Models → Compatibility matrix.

Supported combinations

All modes use FSDP2 (fully_shard, per unit of decoder layers). EP expert modules sit in ignored_params; their routers sync via EP backward hooks, and their experts (with one EP group) or everything (with more) via the deferred post-backward sweep, except at ep_group_size == 1, where fsdp_shard_ep1_experts (default on) has FSDP2 shard them; CP wraps the attention path for sequence splitting and lets FSDP2 sync the rest.

ModeData Parallel SizeNotes
No parallelismworld_sizeStandard multi-GPU
EP onlyworld_sizeEP orthogonal to DP; full DP preserved
TP onlyworld_size / tp_sizeDTensor mesh reduces effective DP
CP onlyworld_size / cp_sizeUlysses attention reduces effective DP
EP+CPworld_size / cp_sizeOnly CP reduces DP
EP+TPworld_size / tp_sizeAttention sharded (node-local TP), experts distributed; TP adds no expert sharding
ETP (pure, ep_size=1)world_size / expert_tp_sizeExpert FFN sharded expert_tp_size-way; experts replicated. MoE only — a dense model raises. See ETP guide
EP+ETPworld_size / expert_tp_sizeExperts distributed ep_size-way and each expert's FFN sharded expert_tp_size-way (ep_group_size = ep_size × expert_tp_size). Experimental. The expert-TP reduce runs in token space, outside the DeepEP dispatch→combine span
PPworld_size / pp_sizeStages are contiguous rank blocks of whole NVLink domains; every other axis runs unchanged inside one stage. See Pipeline Parallelism
PP+EPworld_size / pp_sizeEP inside each stage; the cross-replica expert average is deferred to a post-backward sweep
PP+ETPworld_size / (pp_size × expert_tp_size)Pure ETP (ep_size=1) inside each stage. The deferred sweep divides the expert leg by stage_world_size / expert_tp_size — ETP partners share a batch and are not DP replicas
Anything elseNot supportedRejected at config time with the mechanism named — see below

The supported set is an allowlist (SUPPORTED_AXIS_SETS in halo/distributed/parallelism_rules.py), checked first by reject_unsupported_axes, before any rank math, so a combination nobody considered is refused rather than run unvalidated. Adding one means adding the equivalence gate that proves its gradients against an unsplit reference. Rejections quote the flags you typed and the mechanism from AXIS_SET_MECHANISMS that would break, for example:

  • TP+CP — TP shards attention over a DTensor (dp, tp) mesh while CP's Ulysses groups partition the same ranks a second, incompatible way.
  • TP+ETP, with or without EP — attention TP and expert TP would shard the same ranks along two different axes. Use EP+TP or pure ETP.
  • ETP+CP — expert-TP partners hold shards of one expert and must see the same tokens (SumAcrossGroup sums their outputs element-wise in token space), but CP hands each rank a different sequence chunk, so that sum would add unrelated tokens.
  • PP with anything but EP or pure ETP — the full PP refusal list: Pipeline Parallelism → Rejected combinations.

Topology rules

Inside an allowed set, reject_invalid_composition checks the layout against the NVLink domain (nvlink_domain_size: the OS node, or the rack on NVL72), also before the model loads:

  • Locality. tp_size, cp_size and expert_tp_size divide the domain (_reject_tp, _reject_cp_locality, _reject_expert_tp). A node-scope ep_group_size divides the domain; a global one must tile the stage's ranks as equal contiguous per-domain blocks (_reject_ep_group, Multi-Node). world_size is a multiple of both gpus_per_node and the domain (ParallelismConfig._resolve_nvlink_domain).
  • One domain, one dispatch group. Within a single domain pure EP runs at ep_size == 2 or with ep_group_size filling the domain; ep4 on 8 GPUs is refused (_reject_single_domain_multigroup_ep, mechanism and evidence).
  • EP+TP. ep_size is a multiple of tp_size. TP leaves ep_group_size at ep_size, so the single-domain rule applies unchanged: ep2+tp2 and ep8+tp2 on 8 pass, ep4+tp2 is refused. Across domains the EP group must be a single global one (ep_size == world_size, ep_scope: global): multi-group EP there defers its replica average by sharding over the EP group, while EP+TP shards over the (dp, tp) mesh (_reject_tp).
  • EP+CP. Node-local EP with ep_group_size == nvlink_domain_size; global-scope EP is refused (_reject_ep_cp). cp_size only has to divide the domain, so ep8+cp2, ep8+cp4 and ep8+cp8 all run on one 8-GPU node.
  • EP+ETP. The ETP groups stay inside one domain. EP may cross domains as a single EP group with exactly one ETP group per domain; multi-group EP+ETP across domains is refused, since expert TP keeps the deferred replica average off (_reject_expert_tp, ETP validation rules).
  • HSDP composes with pure DP and CP only (_reject_hsdp); the FSDP shaping knobs carry per-axis rules (_reject_fsdp_settings, Data Parallelism → Limitations).
  • Model shape. reject_unsupported_model reads config.json before any weight: num_experts divisible by ep_size, expert_tp_size dividing the expert FFN width, head counts divisible by tp_size, and the declared per-rank token budget under DeepEP's cross-domain dispatch ceiling (DeepEP backend). Then reject_unsupported_structure checks a meta-device shell of the model, still before any weight: the Ulysses refusals and CP head counts under CP (Context Parallelism), and the zero-shard check under selective TP (Tensor Parallelism).

Communication and data flow

Two ways to group ranks, used where each fits:

  • DeviceMesh (DTensor) for the data-parallel and tensor axes, where sharding is static and the collective follows from the tensor layout: fully_shard reduce-scatters FSDP params, and TP's linears sum their activations over the tp group.

    The meshes are built in one place, halo/distributed/mesh.py: a 1D (dp,) mesh, a 2D HSDP (dp_replicate, dp_shard) mesh (replicate across NVLink domains, shard within one), and the (tp,) / (dp, tp) tensor mesh. The trainer reads the resulting groups through its accessors (tp_group, tp_local_rank, dp_group).

  • Hand-built process groups (build_groups in halo/runtime/collectives.py) for EP, CP and PP, whose communication DTensor cannot express: EP routing is data-dependent (the router picks each token's experts at runtime), EP and CP run custom all-to-all kernels (DeepEP dispatch/combine, Ulysses sequence↔head swap) rather than a layout-inferred redistribution, and PP sends point-to-point along a pipeline chain (halo/distributed/pipeline_parallel/groups.py). On an NCCL default group bound to its device (the init_distributed launch) every group of one kind comes from one split_group, a single communicator split; a dist.new_group per group would split the world communicator once per group, each split blocking every rank (136 world-wide splits for ep8 on 1,024 ranks, 520 on 4,096).

    DeepEP also needs contiguous rank blocks for its intra-node kernel, which a mesh's row-major order cannot produce. EPConfig (halo/distributed/expert_parallel/config.py) builds the EP, expert-replica, dispatch (sub-EP) and expert-TP groups; CPConfig (halo/distributed/context_parallel/config.py) builds the CP groups. All of them, and ParallelismConfig, take their membership from the two layouts in halo/distributed/group_layout.py: NodeLocalLayout and the cross-node CrossNodeLayout.

What crosses the wire:

ModeCollectiveOver
FSDP2all-gather params (forward), reduce-scatter grads (backward)(dp,) mesh
HSDPall-gather/reduce-scatter within a domain, one all-reduce across replicas per backward(dp_replicate, dp_shard) mesh
TPall-reduce / all-gather per layer(tp,) axis
CPUlysses all-to-all (sequence ↔ heads) around attentionCP group
EPDeepEP all-to-all dispatch → local expert GEMM → all-to-all combineEP dispatch group
EP+ETPexpert-TP all-reduce in token space, outside the dispatch→combine spanexpert-TP group
PPboundary activation forward, its gradient back, point to pointpipeline chain (one rank per stage)

Backward mirrors the table: every activation all-to-all / all-reduce has an autograd transpose. By default FSDP2 reduce-scatters after every accumulation micro-step's backward (fsdp_defer_grad_sync holds it to the last); the EP gradient sync runs on the last micro-step: the router hooks during its backward (once per recompute under reentrant checkpointing, an average that repeats harmlessly), the deferred sweep after it.

EP gradients sync outside FSDP2: the replicated router all-reduces over the world group (the stage under PP), and expert grads all-reduce over the expert-replica group only when a run holds more than one EP group. fp32_grad_reduce runs that reduction in fp32 over bf16 storage.

The global batch is split along the data-parallel axis only: each of data_parallel_size ranks takes a disjoint slice, sharded by data_parallel_rank in halo/trainers/loop/dataloaders.py. Ranks inside a TP, CP or ETP group, and the stages of one pipeline chain, see the same batch; they shard weights, the sequence or the layers, not the data.

EP is orthogonal: every rank still draws its own batch, and tokens are all-to-all'd to the ranks owning their routed experts, then returned in place after combine. CP splits the sequence: each rank holds seq_len / cp_size tokens of its batch, and the Ulysses all-to-all reconstructs full sequences inside attention.

Stacking the dimensions

data_parallel_size counts distinct batches. PP carves the world first (stage_world_size = world_size / pp_size), then TP, CP or ETP divide each stage:

Text

data_parallel_size = world_size / pp_size / max(tp_size, cp_size, expert_tp_size)

EP does not reduce it, and the divisor is a max, not a product: no two of TP, CP and ETP may exceed 1 at once. FSDP2's shard group is chosen per mode: the whole stage (the world at pp_size 1) for plain DP, CP, EP and ETP (CP and ETP ranks shard parameters even though they share a batch); the dp dimension of the (dp, tp) mesh under TP and EP+TP; and the EP group under multi-group EP across NVLink domains, where the cross-replica average is deferred to the post-backward sweep.

HSDP is not a separate layer on top of FSDP2; it is how the plain-DP or CP shard group is meshed. The default is 1D full-shard. Under --use_hsdp it is a 2D (dp_replicate, dp_shard) mesh that shards within one NVLink domain (dp_shard_size = nvlink_domain_size) and replicates across domains (dp_replicate_size = num_nvlink_domains), so on a single domain it is a no-op (HSDP).

Node-local EP+CP reaches cross-domain depth without HSDP: EP dispatch/combine and CP Ulysses all-to-all stay node-local on NVLink, FSDP shards the non-expert params within the EP group, and the cross-replica DP average is deferred to one post-backward sweep over RDMA per step. EP+TP and EP+ETP reach the same depth by sharding attention (TP) or the expert FFN (ETP) across the NVLink domain instead of replicating the DP dimension across domains.

torch.compile

torch.compile does not pay on EP MoE, and PP refuses it; keep torch_compile off and Liger on (torch.compile).

Per-axis limitation surface

Each axis page states its own knob rejections, every one traced to the code that raises: EP · ETP · TP · CP · PP · DP. No trainer refuses EP, ETP or TP; CP and PP are declared per class (SUPPORTS_CP / PP_REFUSAL): Trainer Compatibility. Which family runs which axis: Compatibility matrix.

Recommendations

ModelShort Seq (<32K)Long Seq (>32K)Max Memory Efficiency
DenseStandard or TPCP where the family has a wrapper, else TPTP
MoE with CP and TPEP or EP+TPEP+CPEP+TP or pure ETP
MoE with TP, no CPEP or EP+TPEP at max single-GPU seqEP+TP or pure ETP
MoE with CP, no TPEPEP+CPpure ETP
MoE with neitherEPEP at max single-GPU seqpure ETP
VLM without a TP plan (Qwen3-VL)StandardStandard, CP on text-only batchesZeRO-3 FSDP2 (fsdp_reshard_after_forward: true)

A model too large for one NVLink domain adds PP across domains where its family is PP-registered, a MoE with EP (or experimental pure ETP) inside each stage (When PP is worth it).

A family's CP, TP and PP verdicts, and its qualifiers (Zaya trains without gradient checkpointing in every mode): Compatibility matrix.

Additional guides

When pipeline parallelism is worth it

On one 8-GPU machine, run as two 4-GPU agents, PP runs 8.5% below dp8 on Qwen3-8B and 9.2% above ep8 on gpt-oss-20b, where halving the dispatch group pays for the bubble. Across domains the trade is which traffic crosses RDMA: global-scope EP puts four latency-bound all-to-alls there per MoE layer per microbatch (dispatch and combine each repeat in backward), while PP keeps EP node-local and sends 2 × (pp_size − 1) point-to-point activations per microbatch. On an NVL72 rack the whole EP group rides NVLink and global EP wins outright.

Measured on two 8-GPU B300 machines, DP, with any experts on one machine, beats PP on a fast link: PP runs 4.0% below dp16 and 15.9% below ep8 with DP. PP leads layouts that cross the link with attention or experts (1.63× over dp8 × tp2, 3.82× over ep16), a slow link (13.8× over dp16 forced onto plain TCP sockets), and a model too large for one machine's expert group (GLM-5.3-Flash, 1.61× over ep16, where ep8 with DP runs out of memory). The rows: Measured throughput.

PP is not a drop-in: what it refuses. Layouts without it: Large-Scale Scenarios and Capabilities & Limitations at Scale.

Source on GitHubUpdated 18 November 2026