Expert Parallelism (EP)
EP distributes MoE experts across GPUs to cut per-GPU expert memory while keeping full data
parallelism. EP is orthogonal to DP: unlike CP, TP and ETP it never reduces data_parallel_size.
Every GPU loads different data and holds a different expert slice; tokens reach the right experts via
DeepEP all-to-all.
Use it when expert layers exceed single-GPU memory. Why the tokens-per-expert count sets the speed of every expert GEMM: GPU Training Theory §2.
EP requires DeepEP — there is no NCCL fallback.
Non-EP gradients sync via FSDP2 (fully_shard, EP modules in ignored_params); router gradients
sync via the EP layer's own backward hooks and expert gradients via the trainer's post-backward sweep,
which takes the routers too when there is more than one EP group
(Gradient synchronization). Expert Tensor
Parallelism further shards each expert's FFN inside the EP group.
Every family's EP layer subclasses EPMoELayerBase (halo/distributed/expert_parallel/moe_layer/layer.py),
which owns construction and the dispatch → expert compute → combine forward; expert compute, routing,
gradient-sync hooks, native expert LoRA, the balancing bias and the collective half of the checkpoint
export are one mixin each in moe_layer/. The export's layout half is model-side: each family's
MoEFamilySpec.expert_layout, applied by to_hub_expert_tensors (halo/models/moe/expert_layout.py) in
the gather and in merge_ep_shards.py alike. Callers reach the layers through named_ep_layers(model).

Supported models
Source of truth: MOE_LAYER_MAP in halo/distributed/expert_parallel/families/roster.py, built from its
explicit import of every family class.
Bailing / Ling need
trust_remote_code=Trueand have no aux router loss; under EP they balance through the gate's nativeexpert_bias(moe_balancing: autoresolves tobias_update), so the gate trains unfrozen (Bailing → Router balancing).LoRA targets attention (PEFT) and the experts (native grouped adapters — list expert names in
lora_target_modules). Attention adapters are replicated across EP ranks; expert adapters are rank-local and gathered on save.Expert LoRA is rejected with
expert_tp_size > 1. The grouped adapters honorr/alpha/dropout/use_rslora, withra multiple of 8 — the grouped GEMM's stride contract; knobs with no grouped implementation (use_dora,lora_target_parameters) are rejected rather than applied to the attention half alone (PEFT).
Per-family EP restrictions
A family declares seven capabilities, six on its MoEFamilySpec and one on its EP layer, all on by
default; a family turning one off is the whole restriction. Every family absent from the table inherits the defaults and runs the full
EP surface. The table is pinned against the specs and classes by
tests/cpu/parallelism/test_docs_limitation_tables.py.
A second class of weight-sync restriction sits outside that table, keyed on the model type and
the engine. Each client declares the model types its pinned release cannot take an online update for
(UNSERVABLE_MODEL_TYPES in halo/weight_sync/clients/), and reject_unsyncable_model
refuses that model+backend pair at trainer construction, quoting the loader fact.
The SGLang client's list is the longer of the two: Laguna and Step-3.7 sync on vLLM only. Full roster with each reason: Rollout Servers.
Attention-side support is a separate question owned by each axis: which families TP can shard is
TP_SHARDABLE_ATTENTION_CLASSES (TP), and which CP can
split is the Ulysses wrapper registry
(CP). Router balancing per family:
Callbacks.
EP grouping
ep_size must divide num_experts exactly; DeepEP dispatch assumes a uniform expert→rank
division. reject_unsupported_model raises off config.json at the top of
load_distributed_model, before the process groups and the meta shell.
EPConfig.finalize_expert_assignment re-checks it once the EP groups exist.
ep_group_size must divide the NVLink domain (node-local). Under ep_scope="global" the bound is
the world (the stage under PP), and divisibility alone is not enough: the group must also tile it as
equal contiguous per-domain blocks (Multi-Node).
On a single 8-GPU node every pure-EP job is DP=8; what changes is how many DeepEP dispatch groups form.

Example: 32 experts, world=4, ep_size=2 → 2 EP groups of 2 GPUs, 16 experts each, DP=4.
Node-local EP assigns consecutive ranks within a domain; cross-node EP (ep_scope="global") uses
the column-block layout — see Multi-Node.
ep_scope="auto" (the CLI default) picks node-local when ep_group_size <= nvlink_domain_size,
else cross-node. "node" forces NVLink-only groups; "global" spans domains over RDMA.
Single-domain multi-group EP races and hangs
Inside one NVLink domain (not one OS node; on NVL72 the domain is the rack), ep_size=4 on
8 GPUs (two 4-rank groups) makes the groups' DeepEP combine barriers
race FSDP2's DP-wide NCCL collectives.
Both backends fail, with different symptoms, measured on 8×B300 against a clean ep8 control. The
legacy buffer (DeepEP V1) deadlocks deterministically on ~step 2 (DeepEP timeout check failed →
cudaErrorLaunchFailure); the elastic default (V2) faults with CUDA error: Invalid access of peer GPU memory over nvlink. Gradient checkpointing on or off does not change this, and
CUDA_DEVICE_MAX_CONNECTIONS=1 (baked into the images) does not cover it.
The predicate is ParallelismConfig.is_racy_single_domain_multigroup_ep: one NVLink domain,
ep_size > 2 and ep_group_size < nvlink_domain_size. It keys on ep_group_size, not ep_size.
parallelism_rules._reject_single_domain_multigroup_ep rejects the shape at config
construction, before the model loads, hand-built configs included. Deferring the cross-replica
average does not help here: on one domain FSDP2 still shards the non-expert params over every rank
(is_deferred_dp engages only across domains), so its reduce-scatter spans the whole domain while
each combine spans a subset.
Safe single-node pure-EP shapes — one group, or 2-rank groups:
ep_size == GPUs in the job→ single group.ep4on 4 GPUs runs clean (the online-GRPO shape where training GPUs join EP and a separate vLLM server holds the rest);ep4on 8 races.ep_size == gpus_per_node→ one group across the node (ep8on 8).ep_size == 2→ many 2-rank groups, clean.
The only 4-way expert split left on 8 GPUs is EP+ETP (ep4 + etp2), which the gate accepts
because ep_group_size then fills the domain, although its DeepEP dispatch groups stay 4 ranks wide;
make bench measures it and the GPU test matrix runs ep2 + etp4
(ETP validation rules). Attention TP does not
help at all: it leaves ep_group_size untouched, so ep4 + tp2 is rejected exactly like bare
ep4.
Multi-node multi-group EP is supported
Across nodes, multiple EP groups (num_ep_groups > 1) are data-parallel replicas — node-local
EP across domains, or a cross-node EP group replicated within a larger cluster. Here the toolkit
avoids the race instead of blocking: every backward-time collective stays inside the EP group and
the cross-replica DP average is deferred to one post-backward sweep. Node-local ep8×2 and
cross-node ep8 (two replicas) match single-group ep16 on two 8-GPU machines
(Verified topologies).
That sweep covers every shape holding more than one EP group, the single-node ones (ep2,
ep2+tp2) included. Mechanism:
Multi-Node → deferred cross-replica sync.
Quick start
Bash
torchrun --nproc_per_node=8 scripts/training/sft.py \
examples/sft/gptoss/gptoss-20b-multinode-ep.yaml \
--expert_parallel_size=8 --use_grouped_gemm=trueThe recipe pins the per-expert loop for its own ep16; at ep8 grouped GEMM is faster (Grouped GEMM).
EP patching and gradient sync are automatic. Programmatically, pass
parallelism_config=ParallelismConfig(ep_size=8); trainer.save_model() gathers expert slices, and
trainer.cleanup_ep() destroys DeepEP buffers before exit.
Configuration
The EP flags (expert_parallel_size, ep_scope, nvlink_domain_size, use_grouped_gemm,
ep_buffer_backend, the fp32_* masters, fp32_grad_reduce) and their defaults:
Configuration reference → ParallelismConfig;
save_sharded_ep:
checkpoint flags.
Training args: bf16=True, and gradient_checkpointing=True where activations do not fit: GC off is the
largest throughput lever (Maximizing achieved TFLOPS).
With GC the trainer calls enable_scoped_gradient_checkpointing() and sets
args.gradient_checkpointing=False so HF Trainer does not re-enable it.
fp32_grad_reduce upcasts the router grad (always reduced) and the expert cross-replica grad
(only when num_ep_groups > 1) to fp32 for the collective and writes back bf16. bf16 reduction is
lossy and the error grows with rank count — worth enabling for many-rank / multi-node EP.
Precision control
fp32_router and fp32_experts are read only by the EP layers: a run that builds none (a dense model, a family with no EP layer, use_grouped_gemm: false at expert_parallel_size: 1) is refused at load.
Training checkpoints keep these masters in fp32 and exports write them bf16 (What gets saved).
Construction restores their checkpoint values before the EP layers adopt them, through one shared
selector (halo/model_loading/parallel/fp32_masters.py): Load precision.
fp32_non_ep_params: true unconditionally implies fp32_router: true: every family except Gemma 4
keeps its router inside the EP layer, which that upcast skips, so the implication keeps every
non-expert weight an FP32 master.
Gemma 4: fp32_non_ep_params under EP is refused at load, off the family spec's
supports_fp32_non_ep_params=False and before the model is built. Its router (Gemma4TextRouter)
lives in the parent decoder layer and its norms re-emit activations at weight dtype, so the upcast
would feed fp32 tokens into DeepEP's 2-byte transport.
fp32_router is a warned no-op there for the same reason: the router is outside the EP layer and
stays BF16. Only fp32_experts applies.
AdamWBF16 is auto-enabled with bf16=True (6 vs 12 bytes/param, stochastic rounding on the
weight write) and takes precedence over the dtype-grouped optimizer, so it — not
fp32_non_ep_params — decides the optimizer. It handles the mix internally: bf16 params take the
fused SR kernel, fp32 params standard in-place AdamW. See
BF16 Optimizer.
DeepEP backend
DeepEP owns installation, buffer sizing, transport and fabric tuning. What an EP run has to plan around:
One dispatch ceiling, refused at config time off the declared budget and re-checked at buffer sizing, rather than left to wedge in transit:
- Cross-node (Gin) dispatch caps at
HALO_DEEPEP_GIN_MAX_TOKENS_PER_RANK(default 8192,0disables), above which an EFA proxy-GIN dispatch wedges instead of erroring. Intra-node NVLink dispatch has no cap of its own, but the defaultelasticbackend deadlocks ep8 at ≥~64k tokens per rank; gpt-oss-20b ep8 trains at 65,536 onep_buffer_backend: legacyand at 49,152 on either (DeepEP → Transport backend).
The Gin cap is the binding limit on
per_device_train_batch_size× sequence length for anyep_scope=globalrun spanning more than one NVLink domain (AWS EFA). How the config-time budget is derived: DeepEP → Dispatch token ceiling.- Cross-node (Gin) dispatch caps at
The dispatched count is
per_device_train_batch_size × tokens-per-sequence, doubled for DPO, SMPO and reward, which forward chosen and rejected together. It does not scale withnum_generationsorgradient_accumulation_steps, and the buffer is per-rank, so raisingep_sizedoes not lower it. Bound the single-sequence length:per_device_train_batch_size = 1for long sequences, and cap SFTmax_lengthor the RL rollout budget.PYTORCH_CUDA_ALLOC_CONF=expandable_segments:Truecomposes with theElasticBufferbut costs throughput under EP, so set it only to fit (measured).Dtype contract (
halo/distributed/expert_parallel/deepep/dispatcher.py):topk_weightsmust be FP32 contiguous; token tensors keep their dtype across dispatch/combine. The wire zero-pads the feature dim to a multiple of 256 and slices it back symmetrically (GPT-OSS hidden 2880 → 3072), so gradients are exact.ep_buffer_backend: legacyselects the V1 CUDA-IPCdeep_ep.Buffer: numerically identical, intranode only. It is rejected at config time for a cross-node group, a dispatch width above 8 NVLink peers or abovegpus_per_node, or a rank count DeepEP ships no tunedConfigfor.
Gradient synchronization
Hybrid sync, with one EP group:
- Non-EP params — FSDP2
fully_shardreduce-scatter over the DP world; EP modules sit inignored_params. - Router and replicated params (
register_post_accumulate_grad_hook,halo/distributed/expert_parallel/moe_layer/grad_sync.py): computed on the local batch before all-to-all, thenall_reduce(SUM) / world_size. The hooks skip sync while accelerate'sGradientState.sync_gradientsis False (accumulation steps). Under reentrant checkpointing a hook fires once per nested backward, which the average absorbs, being linear in the gradient; a layer recomputed K times pays K all-reduces of its router and shared-expert gradients. - Expert params: the DeepEP backward already aggregates within the EP group, so their sync is a
divide by
world_size / expert_tp_size, with no collective. It runs once per optimizer step in the trainer's post-backward sweep (_run_deferred_grad_sweep, from the clip or, with clipping off, the optimizer's step pre-hook), never on a hook: a reentrant recompute re-enters the backward once per segment, and a divide repeated per firing would leave the earlier segments divided K times. The sweep divides every accumulated expert gradient, so a bank that idles in the sync microbatch after accumulating earlier still gets it; a bank with no accumulated grad keepsgrad=Noneand the optimizer skips it (EPConfig.runs_post_backward_sweep, off only without peers or underfsdp_shard_ep1_experts).
More than one EP group registers no hooks at all. The cross-replica all_reduce(SUM) those
topologies need cannot ride a post-accumulate hook: it fires only where a grad accumulated, so a
rank whose dispatch delivered no tokens for a layer would leave its replicas hanging in a collective
it never enters.
EPConfig.sweep_reduces_across_replicas routes every such shape, single-node included, to the
post-backward sweep. ep_group_size == 1 under
fsdp_shard_ep1_experts is the exception, since FSDP2 already owns those experts.
Expert-only training (native expert LoRA, or experts alone unfrozen) leaves nothing upstream of
the first MoE layer requiring grad. A rank whose experts received no token would then compute a
constant there, and its backward would skip the DeepEP collectives its peers enter: a barrier hang,
mispaired collectives, or no grad on its loss at all. _rank_uniform_dispatch_input makes such a
dispatch input a grad-requiring leaf in a grad-enabled training forward at ep_size > 1, so every
rank builds the same dispatch/combine graph. The leaf's gradient is discarded; the added work is one
dispatch-backward all-to-all and the expert compute's input-grad path, in the first such layer of
each forward.
Gradient clipping is custom because experts are distributed
(_compute_global_grad_norm, halo/trainers/parallelism/grad_clipping.py): local expert grad-norm² per rank → TP shard
norms batch-all_reduce(SUM)ed via ._local_tensor → expert norms all_reduce(SUM)ed over the
expert-TP group, then the dispatch group → sqrt(expert² + non_expert² + tp_shard²). Replicas
across EP groups need no reduce: the clip runs the deferred sweep first, which leaves them identical.
The striped checkpoint save relies on it: its writers take layers from different EP groups, so a
change to the sweep's cadence that leaves the groups unequal at a save writes a mixed checkpoint.
Determinism
Under full_determinism: true the EP path (dispatch, expert compute, combine) repeats bit for bit,
expert weight gradients included, as long as each EP group sits inside one NVLink domain. The rest of
the model needs its own deterministic kernels: the conv backward of the gated-DeltaNet families (Qwen3.5/3.6 MoE, Qwen3-Next, Qwen3.8-Flash-Next)
takes its deterministic path under torch's deterministic-algorithms flag, which full_determinism turns on, unless
the launch exports CAUSAL_CONV1D_DETERMINISTIC=0.
DeepEP's default dispatch claims each receive slot with an atomic, so the order an expert's tokens
arrive in changes from step to step, and with it the summation order of the expert weight gradient
(grouped GEMM and per-expert loop alike). Attention, router, norm and embedding gradients stay
bit-identical while every expert weight drifts in its last bits, and two runs diverge within a few
steps. Under torch's deterministic-algorithms mode, which HF's Trainer turns on for
full_determinism, the dispatcher builds the ElasticBuffer with DeepEP's deterministic=True
(halo/distributed/expert_parallel/deepep/backends.py): a prologue kernel places each received token by
source rank and token index. The legacy V1 buffer places tokens by prefix sums already and needs
nothing. The rest of the EP path is deterministic as is: grouped GEMM, the fused GLU, the atomic-free
permute, the combine, the per-expert loop (use_grouped_gemm: false), whose every index_add_ call
writes each row at most once, and ep_size == 1, which has no dispatch.
DeepEP asserts at every dispatch and combine, on both buffers, that deterministic mode does not run
beside torch.utils.deterministic.fill_uninitialized_memory, which defaults to on whether or not the
mode is. The dispatcher switches the fill off at the first dispatch under the mode, with one warning;
that lasts for the rest of the process and covers every op, not only EP.
Cost on gpt-oss-20b ep2 at 4,096 tokens per rank (2×B300, 30 steps, Halo 1.1.0 or earlier): the deterministic dispatch alone
costs about 2% of throughput (14.9k vs 15.1k tokens/s/GPU), and the whole full_determinism mode about
11% (13.4k), most of it torch's deterministic kernels. Runs without the mode keep the default dispatch.
Not supported across NVLink domains. DeepEP's cross-domain (hybrid RDMA) dispatch has no
deterministic mode. full_determinism on an EP group spanning domains is refused while the entry
script builds its config, before the load, and again at trainer construction for a hand-built config
(parallelism_rules.reject_cross_domain_determinism). Keep EP node-local (ep_scope: node) with DP
across domains; on a multi-node NVLink fabric set NVLINK_DOMAIN_SIZE to the fabric's size.
tests/gpu/parallelism/ep/test_ep_deterministic_expert_grads.py replays a backward under
full_determinism and requires the loss and every gradient bit-identical: GPT-OSS on every two-rank
layout (ep2 at the model's top-k and at top-1, the per-expert loop, the legacy buffer, ep1, pure
ETP), and every other family on ep2.
Gradient-checkpoint dispatch replay
A checkpointed layer runs its body twice, and the second run must NOT touch DeepEP: a fresh dispatch
reuses the same ElasticBuffer and invalidates the handle the original forward's backward node
still holds, which corrupts every gradient in the model and raises nothing.
So DeepEPDispatcher.dispatch / combine cache detached results under their owner layer on the first
pass, and the recompute replays them through ReplayDispatchFunction / ReplayCombineFunction. Only the expert compute is recomputed, and
backward still calls buffer.combine() / buffer.dispatch() for the gradient comm.
That cache lives on a RecomputeScope (halo/models/gradient_checkpointing.py) created
per checkpoint invocation and entered by both passes, not on the layer: concurrent frames stay
separate, and a no_grad reference or teacher pass
cannot reach a scope it did not create. The recompute releases the cache as it exits: the reentrant checkpoint
keeps every scope until the loss is dropped, so a layer's receive buffers would otherwise stay alive through
every later layer's backward. The recompute signal is the scope's pass counter, not grad
mode, so replay works in both checkpoint modes — but the two modes are still not
interchangeable.
use_reentrant=True is required — the trainer forces it outside pipeline parallelism, whose stages
run non-reentrant (PP + EP). Non-reentrant triggers its recompute
lazily, when backward first needs a discarded tensor; nothing imposes a common order across ranks,
they desynchronize, and DeepEP's barrier times out — cudaErrorLaunchFailure, an unrecoverable
abort, not a raised error. Measured on 2 GPUs: reentrant is grad-exact (worst relative gradient
error 4.3e-05 vs no checkpointing, across all 155 parameters), non-reentrant aborts. A layer with no
gradient-carrying input takes one under it where it holds a trainable parameter, which is how a tower
block behind a frozen patch embedding trains (Data Parallelism).
Under routing replay
the recompute reads the frame's saved expert selection. Expert-load counters record only a scope's
grad-driven original pass, gated on the outer grad mode captured at invocation: the recompute
never double-counts moe/* metrics or bias-update balancing, and a no_grad forward through a
train-mode checkpointed model (offline GRPO's KL reference pass) records nothing either.
Scopes are installed automatically when gradient_checkpointing=True and EP is active, and they
follow every later re-enable: enable_scoped_gradient_checkpointing wraps the model's own
gradient_checkpointing_enable. Online GRPO depends on this — TRL wraps generation in
disable_gradient_checkpointing, whose exit restores a bare checkpoint from the args once per step,
and a layer that entered backward without a scope raises rather than silently corrupt the gradients.
Kernels
Expert compute uses Grouped GEMM
(torch.nn.functional.grouped_mm) on SM90+ by default; otherwise a per-expert loop accumulates with
index_add_. The win scales with local experts per rank (num_experts / ep_size): large with many,
smaller with few, and the loop is never faster in the measured grid. Keep the default; the grid is on the
Grouped GEMM guide.
The expert GLU combine (SiLU, tanh-GELU, and the clamped SwiGLUs of GptOss, DeepSeek-V4, GLM-5 Next
and Step-3.7) runs as a fused Triton kernel on every expert path (halo/kernels/fused_glu.py;
HALO_FUSED_GLU=0 forces the eager form). Liger is the default; torch.compile adds nothing
measurable on top of it on EP MoE (torch.compile).
Whichever kernel a family resolves, a layer with a real dispatch group (ep_size > 1) traces it on
its first forward, before that forward's dispatch (_warm_expert_kernels): one grad-enabled
pass with a backward and one under no_grad, outside inference mode, so a first forward under
torch.inference_mode() still warms the backward kernels a later training forward runs. Every grouped-GEMM path,
at any ep_size, also runs the routing sort, forward and backward, at the layer's top_k and local expert count
(warm_route_sort_kernels, halo/kernels/moe_route_sort.py), the permute's backward and the weighted unpermute's
forward and backward (warm_permute_kernels, halo/kernels/moe_permute.py). One token count covers every dispatch
size: the kernels take their row, block and slot counts with do_not_specialize, so Triton compiles no separate
binary per class of them (1, a multiple of 16, neither). The routing sort is the counting sort at every ep_size
on CUDA, sized by the receive counts or, at ep_size 1, by every slot: the torch construction's argsort picks its
kernels by the slot count. The weighted unpermute's backward takes its routing-weight-gradient switch the same way, so the one warm
compile also serves routing weights that need no gradient. A first launch also loads its kernel (CUDA lazy loading).
index_select takes a separate kernel for an index of up to 16 rows, so the routing sort's backward reads the
gate-weight gradient through gather, one kernel at every receive size. The token permute and GptOss's expert-bias
gather run index_select, so the permute warm-up also launches it with an index of one row and of seventeen, at the
run dtype and at fp32, the expert biases' dtype under fp32_experts (_warm_index_select). The bf16 grouped GEMM's
forward and input gradient take one of two CUTLASS tiles by the mean rows per local expert, which follow the routing,
small at 128 or fewer (Grouped GEMM), so the same pass runs the
layer's own expert FFN, forward and backward, at one row per local expert and at 129 (_warm_expert_gemm), with the
expert adapters on even when that first forward has them off. Its backward takes the gradients of the expert
parameters as well as the tokens', so it also runs the expert-bias gather's backward, which sits off the tokens'
path: the one-hot GEMM (Grouped GEMM), at those two
receive sizes (Limitations).
The inputs are zeros rather than a draw, and the backward runs under identity saved-tensor hooks. The expert FFN's LoRA dropout draws, so the GEMM warm-up restores the RNG after it, and its backward leaves no gradient. The warm-up sits inside the gradient-checkpointed block, whose recompute restores the RNG to region entry and whose non-reentrant form would otherwise recompute the whole block mid-forward.
Without the warm-up, a cold kernel compiles or loads between dispatch and combine, where every peer
of the group is already inside DeepEP's barrier, and that barrier's budget bounds rank skew, not
idle time. At ep_size == 1 the dispatcher is a no-op and there is no barrier, but the grouped path still warms
whole: its receive is the micro-batch's tokens, whose count picks the GEMM tile and the index_select kernel, so a
rank first loading one in a later step holds up the expert-TP reduce or FSDP's next collective. Only the separate
activation pass is left to a dispatch group, since the GEMM warm-up's expert FFN runs the activation.
Model loading
ep_lazy_loading (default true) builds the model shell with parameters on the meta device and
buffers computed for real (accelerate.init_empty_weights(include_buffers=False)), then streams
each rank's expert slice straight from safetensors (load_ep_model in halo/model_loading/parallel/ep.py,
with the expert-key plan in lazy_plan.py and the per-expert fuser in expert_fuser.py beside it, and
the shared materializer in halo/model_loading/lazy_safetensors/). The shell carries the run's dtype, not the
checkpoint config's. Float parameters stream at that dtype; a float buffer keeps its stored dtype unless the
family's fp32 pins name it (GLM-5 Next's e_score_correction_bias), which then loads fp32, as from_pretrained
loads it. Configured parameter masters are read and fused at FP32 instead
(Load precision).
A per-expert checkpoint (a per-expert hub, or unfuse_moe_experts.py output) is read local-only and
fused per rank. Under text_only_model every key of the multimodal checkpoint drops its
language_model. segment, a per-expert one included, which is how the fuser finds the text-only
class's fused parameter it feeds.
The eager fallback replays only selected masters after HF construction, using the same checkpoint
mapping and expert layout. FP32-stored masters need a second read; BF16 safetensors masters
promote the already-loaded values without rereading their payload. Under fp32_experts, FP32 replay
materializes the full expert bank before the EP slice: twice the expert bytes of BF16, plus
conversion temporaries. Every eager EP path (EP, EP+CP, EP+TP, ETP) stages this bank on each admitted
CPU rank and slices it before the move to the GPU, which holds Qwen3-30B-A3B's eager load at ep2 × tp2 to about
34 GiB per GPU against about 62 GiB with the whole bank on the GPU. max_concurrent_loading bounds admitted ranks per
node; the lazy path reads only each rank's expert slice and avoids full-bank staging.
Buffers must be real: a config-less rotary derives inv_freq from ctor args it never stores, which a
meta build loses irrecoverably. The from_pretrained(device_map="meta") route strands the
non-persistent ones on meta, so it builds a config-only twin and grafts them back.
An architecture that route cannot place is built from the config alone. A failure there raises
rather than falling back, since from_pretrained inside a meta context still streams the whole
checkpoint into host RAM.
State the checkpoint does not carry (score for reward / classification on top of a base LM
checkpoint) is random-initialized through the _init_weights of the nearest PreTrainedModel above it, exactly as
from_pretrained dispatches them, so a composite's tower draws its own parameters; the run's identical seeding makes
every rank draw the same values.
A module only partially absent raises rather than overwrite the tensors that did load, and a tied
shadow (lm_head.weight) is left to the post-load tie_weights(). A parameter or buffer that
reaches device placement still on meta raises: filling either would train or score uninitialized
memory that differs on every rank.
The disk side is guarded symmetrically, because a ranged read is silently satisfiable by a checkpoint whose shapes disagree with the config:
- every materialized tensor's shape is checked against the live target;
- a fused expert axis longer than the config's expert count raises before the read;
- a per-expert fusion must cover the rank's full contiguous expert range with both GLU halves;
- keys matching no model tensor are warned once;
- two disk keys claiming one tensor is refused (an error).
The per-rank reads are fenced through the same rank-consensus seam the save side uses, so a torn shard on one rank fails the whole world with the real disk error instead of stranding peers at the next collective.
Checkpointing
Gathered: experts are all_gathered (every rank must enter) but only the save rank keeps
the tensors: global rank 0 on a shared FS, each node's local rank 0 otherwise. The RL weight sync instead delivers each
layer to the forwarding rank alone (Rollout Servers).
retain reaches inside the expert gather, so a non-writing rank joins every collective and returns
{} without running the layout assembly (per-expert split, re-interleave, transpose + contiguous,
host copy) that follows it. The expert-axis gather itself receives into one preallocated buffer
rather than a shard list plus a cat. Both bound the transient on every rank that keeps nothing.
The save streams into a StreamingShardWriter one chunk at a time, the non-expert tensors first and
then one EP layer per chunk, and finalizes with close_as_hf_checkpoint(). Its host RAM holds the
current chunk, the part it is filling and one part per background write thread
(save_max_shard_size, default 5GB; HALO_CHECKPOINT_WRITE_THREADS, default 0), not the whole
checkpoint. A multi-node shared-filesystem save stripes the layers over each node's first rank
(Checkpoints), so each writer holds its own current layer and parts. By arithmetic, writing in line, that peak is about 12 GB on the writer at gpt-oss-120b
(ep8) and about 21 GB at Qwen3.5-397B-A17B (ep64); each write thread adds up to one part.
The plain FSDP2, CP and TP gathered saves stream through the same writer
(stream_gathered_checkpoint), one decoder layer at a time.
Sharded exists for write bandwidth, not for memory. N ranks write their own slice in parallel, so the pause at each save is bounded by one rank's shard instead of by the whole artifact funneled through a single writer. Host memory is not a reason to choose it: a sharded save holds more per node (every local rank buffers its own slice).
The cost is that nothing loads the result. merge_ep_shards.py must run before resume or serving;
it carries the optimizer shards and their meta, hard-linked when the output is on the input's
filesystem, so a resume from the merged directory is exact on the save's topology. A later training save
into either directory replaces the linked files rather than writing through them
(Checkpoints & Resume).
save_max_shard_size does not apply to these files; a per-rank shard is one file by design. The cap
bounds the gathered save; the merged artifact takes merge_ep_shards.py --max_shard_size.
Sharded writes model-{rank:05d}-of-{world_size:05d}.safetensors plus an index carrying
ep_size. reject_unmergeable_ep_sharded_save() rejects it at trainer construction (and again at
the save) whenever any of these holds:
ep_group_size != world_size,expert_tp_size > 1, or CP;- native expert LoRA or
merge_expert_lora_on_save; - a
model_typeno family spec claims, or a load that converted names the gathered save reverts through transformers' save-side revert (Step-3.7 Flash, a SigLIP tower, a vendor namespace; the merge streams key by key and cannot apply it); - a model holding external tables (Qwen3.8-Flash-Next's n-gram tables), which the merge has no model to read from;
- a non-shared multi-node filesystem;
- no EP layers at all (dense, or an MoE without EP layers): every save would be an ordinary gathered checkpoint while a planned merge waits for shards that never appear.
PP refuses save_sharded_ep on its own gate: the per-rank format carries no stage layer offset. The
merge raises on an incomplete shard set.
Bash
python scripts/after_training/merge_ep_shards.py --input_dir checkpoint --output_dir mergedOn resume, EP is a Path B mode: the gathered HF checkpoint cannot load into the EP-fused tree, so the trained weights load at model construction. The training scripts repoint the model source at the checkpoint, and the checkpoint loader raises when the live model was constructed from anything else, rather than silently continuing on stale weights.
The loader then restores adapters; optimizer, scheduler and balancing
biases resume from trainer state. A merge_expert_lora_on_save checkpoint is the exception to the
repoint: it resumes from the base plus its resume_adapter/
(Merge-on-save checkpoints). An unmerged
sharded save is refused at resume resolution and as a model source, and
load_best_model_at_end is refused under EP full fine-tune (export the best checkpoint instead). See
Checkpoints & Resume.
Non-EP params in EP+TP saves take three routes: DTensor attention params through
full_tensor(), hand-sliced non-DTensor params (GptOss sinks) all-gathered over the TP group, and
replicas from rank 0. All ranks must enter the collectives even though only one writes.
Limitations
Trainers. No trainer refuses EP or the ep_size == 1 grouped-GEMM EP layers.
The families whose spec declares a
weight_sync_refusal narrow this — online and async GRPO raise at construction for
each of them (Per-family EP restrictions). Matrix:
Trainer Compatibility.
Models. The wrapped families in the table above, plus the per-family
restrictions. A ep_group_size > 1 run on a dense model or a MoE with no family spec is refused at load,
before the process groups (reject_ep_without_family), and one that still patches zero MoE layers raises.
Axis combinations. EP composes with one of TP, CP or ETP; the allowlist and the topology rules on top of it: Supported combinations.
Kernel warm-up. GptOss's expert-bias gradient is one cuBLAS GEMM contracting over the receive, whose kernel cuBLAS picks by the receive's length, so the warm-up runs it at its own receive sizes only and a rank can still load another kernel for it mid-step the first time its receive falls in a new size class.
Replica start. With more than one EP group, the copies the groups hold of a parameter (the expert shards, and under deferred DP the non-expert shards too) are not broadcast at startup, as HSDP's replicas are (Data Parallelism → HSDP). Weights read from the checkpoint load identical everywhere; a value each rank draws from its own RNG (a LoRA adapter, native expert LoRA included, or an added embedding row) starts identical only because every rank is seeded alike, as the training scripts seed it. A Python-API run that seeds ranks differently trains those copies apart.
Knobs. Everything below raises unless the verdict says otherwise.
Troubleshooting
gpt-oss-20b peak memory per EP degree on 8× B300: EP-only table (it grows with sequence length and batch).
Adding a new model
Procedure: Adding a Model → Add EP support. The per-PR
EP correctness gate (gpt-oss-20b, EP=2 against the undistributed forward) is
torchrun --nproc_per_node=2 tests/gpu/parallelism/ep/test_ep_modes_vs_reference.py --model gpt_oss_20b --mode ep;
a new family's EP=2-vs-stock row in tests/gpu/parallelism/ep/test_ep_family_vs_reference.py comes from its
TINY_MOE_FAMILIES entry.
Source on GitHubUpdated 18 November 2026