Parallelism
On top of data parallelism, four strategies address different bottlenecks: model size (TP), sequence length (CP), expert distribution (EP), or depth across NVLink domains (PP). They combine within the supported set below. Each mode is a communication pattern with a price: GPU Training Theory §9.
Multi-GPU DP modes run on FSDP2 for gradient sync (pure TP at DP=1 and single-GPU skip it);
DistributedTrainerMixin selects the sharding strategy from the active mode. See the
Data Parallelism Guide for launchers and Accelerate configs.
Parallelism modes
- Expert parallelism (EP) — distributes MoE expert layers across GPUs via DeepEP all-to-all (the router stays replicated on every rank). Each GPU holds a subset of experts and routes tokens to the owning GPU. For MoE models; see Expert Parallelism Guide.
- Tensor parallelism (TP) — shards dense attention and MLP weights across GPUs via DTensor with all-reduce. For large dense models (Llama 70B+, Mistral Large) that exceed single-GPU memory: plain FSDP2 and DDP load the whole model onto every GPU first (why), and TP's HF
tp_planload, which shards as it reads, is the route past that inside one NVLink domain (PP's stage loader is the one across domains); see Tensor Parallelism Guide. - Context parallelism (CP) — splits long sequences across GPUs via Ulysses-style attention with all-to-all. For long-context training (32K+ tokens); see Context Parallelism Guide.
- Pipeline parallelism (PP) — splits decoder layers into stages, one contiguous rank block each, and is the only axis meant to cross NVLink domains. Off by default (why). See Pipeline Parallelism.
Which MoE families each axis wraps: Supported Models → Compatibility matrix.
Supported combinations
All modes use FSDP2 (fully_shard, per unit of decoder layers). EP expert modules sit in ignored_params; their routers sync via EP backward hooks, and their experts (with one EP group) or everything (with more) via the deferred post-backward sweep, except at ep_group_size == 1, where fsdp_shard_ep1_experts (default on) has FSDP2 shard them; CP wraps the attention path for sequence splitting and lets FSDP2 sync the rest.
The supported set is an allowlist (SUPPORTED_AXIS_SETS in
halo/distributed/parallelism_rules.py), checked first by reject_unsupported_axes, before any rank
math, so a combination nobody considered is refused rather than run unvalidated. Adding one means
adding the equivalence gate that proves its gradients against an unsplit reference. Rejections quote
the flags you typed and the mechanism from AXIS_SET_MECHANISMS that would break, for example:
- TP+CP — TP shards attention over a DTensor
(dp, tp)mesh while CP's Ulysses groups partition the same ranks a second, incompatible way. - TP+ETP, with or without EP — attention TP and expert TP would shard the same ranks along two different axes. Use EP+TP or pure ETP.
- ETP+CP — expert-TP partners hold shards of one expert and must see the same tokens (
SumAcrossGroupsums their outputs element-wise in token space), but CP hands each rank a different sequence chunk, so that sum would add unrelated tokens. - PP with anything but EP or pure ETP — the full PP refusal list: Pipeline Parallelism → Rejected combinations.
Topology rules
Inside an allowed set, reject_invalid_composition checks the layout against the NVLink domain
(nvlink_domain_size: the OS node, or the rack on NVL72), also before the model loads:
- Locality.
tp_size,cp_sizeandexpert_tp_sizedivide the domain (_reject_tp,_reject_cp_locality,_reject_expert_tp). A node-scopeep_group_sizedivides the domain; a global one must tile the stage's ranks as equal contiguous per-domain blocks (_reject_ep_group, Multi-Node).world_sizeis a multiple of bothgpus_per_nodeand the domain (ParallelismConfig._resolve_nvlink_domain). - One domain, one dispatch group. Within a single domain pure EP runs at
ep_size == 2or withep_group_sizefilling the domain;ep4on 8 GPUs is refused (_reject_single_domain_multigroup_ep, mechanism and evidence). - EP+TP.
ep_sizeis a multiple oftp_size. TP leavesep_group_sizeatep_size, so the single-domain rule applies unchanged:ep2+tp2andep8+tp2on 8 pass,ep4+tp2is refused. Across domains the EP group must be a single global one (ep_size == world_size,ep_scope: global): multi-group EP there defers its replica average by sharding over the EP group, while EP+TP shards over the(dp, tp)mesh (_reject_tp). - EP+CP. Node-local EP with
ep_group_size == nvlink_domain_size; global-scope EP is refused (_reject_ep_cp).cp_sizeonly has to divide the domain, soep8+cp2,ep8+cp4andep8+cp8all run on one 8-GPU node. - EP+ETP. The ETP groups stay inside one domain. EP may cross domains as a single EP group with
exactly one ETP group per domain; multi-group EP+ETP across domains is refused, since expert TP keeps
the deferred replica average off (
_reject_expert_tp, ETP validation rules). - HSDP composes with pure DP and CP only (
_reject_hsdp); the FSDP shaping knobs carry per-axis rules (_reject_fsdp_settings, Data Parallelism → Limitations). - Model shape.
reject_unsupported_modelreadsconfig.jsonbefore any weight:num_expertsdivisible byep_size,expert_tp_sizedividing the expert FFN width, head counts divisible bytp_size, and the declared per-rank token budget under DeepEP's cross-domain dispatch ceiling (DeepEP backend). Thenreject_unsupported_structurechecks a meta-device shell of the model, still before any weight: the Ulysses refusals and CP head counts under CP (Context Parallelism), and the zero-shard check under selective TP (Tensor Parallelism).
Communication and data flow
Two ways to group ranks, used where each fits:
DeviceMesh(DTensor) for the data-parallel and tensor axes, where sharding is static and the collective follows from the tensor layout:fully_shardreduce-scatters FSDP params, and TP's linears sum their activations over thetpgroup.The meshes are built in one place,
halo/distributed/mesh.py: a 1D(dp,)mesh, a 2D HSDP(dp_replicate, dp_shard)mesh (replicate across NVLink domains, shard within one), and the(tp,)/(dp, tp)tensor mesh. The trainer reads the resulting groups through its accessors (tp_group,tp_local_rank,dp_group).Hand-built process groups (
build_groupsinhalo/runtime/collectives.py) for EP, CP and PP, whose communication DTensor cannot express: EP routing is data-dependent (the router picks each token's experts at runtime), EP and CP run custom all-to-all kernels (DeepEP dispatch/combine, Ulysses sequence↔head swap) rather than a layout-inferred redistribution, and PP sends point-to-point along a pipeline chain (halo/distributed/pipeline_parallel/groups.py). On an NCCL default group bound to its device (theinit_distributedlaunch) every group of one kind comes from onesplit_group, a single communicator split; adist.new_groupper group would split the world communicator once per group, each split blocking every rank (136 world-wide splits forep8on 1,024 ranks, 520 on 4,096).DeepEP also needs contiguous rank blocks for its intra-node kernel, which a mesh's row-major order cannot produce.
EPConfig(halo/distributed/expert_parallel/config.py) builds the EP, expert-replica, dispatch (sub-EP) and expert-TP groups;CPConfig(halo/distributed/context_parallel/config.py) builds the CP groups. All of them, andParallelismConfig, take their membership from the two layouts inhalo/distributed/group_layout.py:NodeLocalLayoutand the cross-nodeCrossNodeLayout.
What crosses the wire:
Backward mirrors the table: every activation all-to-all / all-reduce has an autograd transpose. By
default FSDP2 reduce-scatters after every accumulation micro-step's backward
(fsdp_defer_grad_sync holds
it to the last); the EP gradient sync runs on the last micro-step: the router hooks during its backward (once per
recompute under reentrant checkpointing, an average that repeats harmlessly), the deferred sweep after it.
EP gradients sync outside FSDP2: the replicated router all-reduces over the world group (the stage
under PP), and expert grads all-reduce over the expert-replica group only when a run holds more than
one EP group.
fp32_grad_reduce runs that reduction in fp32 over bf16 storage.
The global batch is split along the data-parallel axis only: each of data_parallel_size ranks takes
a disjoint slice, sharded by data_parallel_rank in halo/trainers/loop/dataloaders.py. Ranks
inside a TP, CP or ETP group, and the stages of one pipeline chain, see the same batch; they
shard weights, the sequence or the layers, not the data.
EP is orthogonal: every rank still draws its own batch, and tokens are all-to-all'd to the ranks
owning their routed experts, then returned in place after combine. CP splits the sequence: each
rank holds seq_len / cp_size tokens of its batch, and the Ulysses all-to-all reconstructs full
sequences inside attention.
Stacking the dimensions
data_parallel_size counts distinct batches. PP carves the world first (stage_world_size = world_size / pp_size), then TP, CP or ETP divide each stage:
Text
data_parallel_size = world_size / pp_size / max(tp_size, cp_size, expert_tp_size)EP does not reduce it, and the divisor is a max, not a product: no two of TP, CP and ETP may exceed
1 at once. FSDP2's shard group is chosen per mode: the whole stage (the world at pp_size 1) for
plain DP, CP, EP and ETP (CP and ETP ranks shard parameters even though they share a batch); the
dp dimension of the (dp, tp) mesh under TP and EP+TP; and the EP group under multi-group EP
across NVLink domains, where the cross-replica average is deferred to the post-backward sweep.
HSDP is not a separate layer on top of FSDP2; it is how the plain-DP or CP shard group is meshed.
The default is 1D full-shard. Under --use_hsdp it is a 2D (dp_replicate, dp_shard) mesh that
shards within one NVLink domain (dp_shard_size = nvlink_domain_size) and replicates across domains
(dp_replicate_size = num_nvlink_domains), so on a single domain it is a no-op
(HSDP).
Node-local EP+CP reaches cross-domain depth without HSDP: EP dispatch/combine and CP Ulysses all-to-all stay node-local on NVLink, FSDP shards the non-expert params within the EP group, and the cross-replica DP average is deferred to one post-backward sweep over RDMA per step. EP+TP and EP+ETP reach the same depth by sharding attention (TP) or the expert FFN (ETP) across the NVLink domain instead of replicating the DP dimension across domains.
torch.compile
torch.compile does not pay on EP MoE, and PP refuses it; keep torch_compile off and Liger on (torch.compile).
Per-axis limitation surface
Each axis page states its own knob rejections, every one traced to the code that raises:
EP · ETP ·
TP · CP ·
PP · DP. No trainer
refuses EP, ETP or TP; CP and PP are declared per class (SUPPORTS_CP / PP_REFUSAL):
Trainer Compatibility. Which family
runs which axis: Compatibility matrix.
Recommendations
A model too large for one NVLink domain adds PP across domains where its family is PP-registered, a MoE with EP (or experimental pure ETP) inside each stage (When PP is worth it).
A family's CP, TP and PP verdicts, and its qualifiers (Zaya trains without gradient checkpointing in every mode): Compatibility matrix.
Additional guides
- Data Parallelism (FSDP) — FSDP strategies, Accelerate configs, mixin internals
- Expert Tensor Parallelism — shard expert FFN weights for extra memory reduction
- Large-Scale Scenarios — pick a layout by NVLink-domain size; start here for any multi-node job
- Multi-Node Parallelism — cross-node EP/CP/TP, EP scopes, RDMA setup
- Multi-Node Launch Recipes — runnable torchrun/SLURM launches, EP checkpoint save/load
- Distributed Data Loading — data loading across parallelism modes
- Checkpoints and Resume — distributed checkpoint formats (gathered vs sharded) and resume
When pipeline parallelism is worth it
On one 8-GPU machine, run as two 4-GPU agents, PP runs 8.5% below dp8 on Qwen3-8B and 9.2% above ep8 on gpt-oss-20b, where halving the dispatch group pays for the bubble. Across domains the trade is which traffic crosses RDMA: global-scope EP puts four latency-bound all-to-alls there per MoE layer per microbatch (dispatch and combine each repeat in backward), while PP keeps EP node-local and sends 2 × (pp_size − 1) point-to-point activations per microbatch. On an NVL72 rack the whole EP group rides NVLink and global EP wins outright.
Measured on two 8-GPU B300 machines, DP, with any experts on one machine, beats PP on a fast link: PP runs 4.0% below dp16 and 15.9% below ep8 with DP. PP leads layouts that cross the link with attention or experts (1.63× over dp8 × tp2, 3.82× over ep16), a slow link (13.8× over dp16 forced onto plain TCP sockets), and a model too large for one machine's expert group (GLM-5.3-Flash, 1.61× over ep16, where ep8 with DP runs out of memory). The rows: Measured throughput.
PP is not a drop-in: what it refuses. Layouts without it: Large-Scale Scenarios and Capabilities & Limitations at Scale.
Source on GitHubUpdated 18 November 2026