Large-Scale Scenarios — Qwen3.5-397B-A17B
Per-rank memory arithmetic, reachable parallelism layouts and launch recipes for
Qwen/Qwen3.5-397B-A17B on 2, 3, 4 and 8 nodes of 8 × B300 (288 GB). Every constraint here
is enforced in code and named by its symbol; the memory columns are arithmetic from the shape below,
so treat them as a plan, not a measurement. Mechanism pages:
Multi-Node, Expert Parallelism,
Launch Recipes, Qwen3.5,
GPU Training Theory.
Two facts set every cell and pull against each other. nvlink_domain_size decides whether EP's
per-layer all-to-all crosses the slow fabric. The routed experts are 386.5 B of the 396.8 B total,
only ever sharded by X = ep_size × expert_tp_size, so they need X ≥ 16 at 8 B/param to fit at
all.
On an 8-GPU domain X ≥ 16 forces the EP group across the domain boundary.
Pipeline parallelism is the other way across it: whole-node stages keep EP
on NVLink, at costs this model feels (Pipeline-parallel cells).
First question: how wide is the NVLink domain?
nvlink_domain_size (auto = gpus_per_node; set NVLINK_DOMAIN_SIZE on NVL72) is the locality unit
every validator divides by, and it decides which axis is allowed onto RDMA. The dispatcher keys the
decision off topology alone: EPConfig.spans_domains = num_domains > 1 and not node_local
(halo/distributed/expert_parallel/config.py), on which the elastic backend sets EP_DISABLE_GIN=0, Gin on.
num_domains counts NVLink domains, not OS nodes, so a group inside one NVL72 rack stays on NVLink
even though it spans ~18 hosts.
What EP at ep_scope=global across domains puts on that fabric per microbatch, at this model's 60
MoE layers, is 240 all-to-alls over the EP group: forward dispatch + combine per MoE layer, and
their backward mirrors — arithmetic from the layer count and the autograd graph, not a benchmark.
Four per layer, not two, because DeepEPDispatchFunction.backward is a combine and
DeepEPCombineFunction.backward is a dispatch (halo/distributed/expert_parallel/deepep/dispatcher.py).
Gradient checkpointing does not multiply the count — recompute replays cached results rather than
re-issuing the collective, and that cache is a memory term of its own
(GC replay cache).
The cost is worse than the count suggests: cross-domain dispatch is latency-bound, so its fixed per-collective cost does not amortize as the message grows (DeepEP → AWS EFA).
The fork:
Domain = 8 (a standard 8 × B300 node). Where a model fits at
ep_group_size <= 8, node-local EP groups replicate across domains as DP replicas, their cross-replica average deferred to a post-backward sweep (Multi-Node); that puts no per-layer traffic on RDMA.At 397B that does not fit (386 GB of experts per rank), so the EP group must cross the domain boundary at global scope, or PP must split the layers across domains; when the fabric is latency-bound, narrow the dispatch group with EP+ETP (Recommended cell).
Domain = 72 (GB200/GB300 NVL72). A node-scope EP group rides NVLink at any width that divides the declared domain,
requires_rdmastays false, and wide EP becomes the attractive shape.On this model that means declaring a 64-wide domain, not 72 (Recommended cell); the fabric check and the bring-up advice are in Scale & Limits.
Model shape
Source: hf_hub_download("Qwen/Qwen3.5-397B-A17B", "config.json"), revision
8472618112abcbd45acbcdc58436aff4233c23f7. The checkpoint is a VLM —
architectures: ["Qwen3_5MoeForConditionalGeneration"], model_type: qwen3_5_moe — and every field
below is from text_config (qwen3_5_moe_text).
layer_types is ["linear_attention"]*3 + ["full_attention"] repeated 15× —
45 gated-DeltaNet layers + 15 full-attention layers, full_attention_interval: 4. The linear
layers carry linear_num_value_heads: 64, linear_num_key_heads: 16, head dims 128,
linear_conv_kernel_dim: 4.
Parameter counts derived from those fields (fused expert layout
gate_up_proj [512, 2048, 4096] + down_proj [512, 1024, 4096]):
P_r is the trap. EPQwen3_5MoELayer owns the router gate, the shared_expert and
shared_expert_gate; the whole layer lands in FSDP2's ignored set (_fsdp_excluded_params), TP
never visits it (apply_tp_to_attention_only), and EP does not shard it — replicated_named_params
grad-syncs it by hand. So 0.88 B stays as a full 8 B/param copy on every rank ≈ 7 GB that no axis
shrinks. Under fp32_router its 0.13 B of router weights are fp32 at 16 B/param, +1 GB.
Per-rank memory model
P_f = P_ne − P_r = 9.37 B is the FSDP-managed non-expert share, F the FSDP shard width for it.
F is not data_parallel_size. It defaults to every rank of the job (F = stage_world_size), so
ep4 + etp8 on 4 nodes still shards non-expert params 32-way even at DP = 4; ETP partners hold
identical replicas of them.
Two topologies narrow F. Under TP the (dp, tp) mesh drops it to DP. Under
EPConfig.is_deferred_dp (multi-group EP across domains, the × N replicas rows below) FSDP shards
over the EP group, so F = ep_group_size and the cross-replica average moves to the
post-backward sweep (Multi-Node).
8 B/param = 2 weights + 2 grads + 4 optimizer state; AdamWBF16 (halo/optimizers/adamw_bf16.py)
allocates exp_avg and exp_avg_sq with zeros_like(p), so both are bf16 and there is no master
copy and no error-feedback buffer. fp32_grad_reduce upcasts only inside the reduce — no storage.
The stock AdamW is refused under EP unless fp32_non_ep_params is set (why),
and fp32 expert masters (fp32_experts) would take the expert term to 16 B/param, where no cell below
survives.
The activation term is one bf16 layer input per layer (2·L·H·b·S) plus the recomputed layer's
expert intermediates (6·k·I_moe·b·S — gate_up, SiLU product, down input at top-k = 10). The
45 linear layers additionally hold an fp32 recurrent state of
64 × 128 × 128 × 4 B ≈ 4 MB per sequence per layer (≈190 MB at b=1).
The GC replay cache is what lets the recompute skip the all-to-all
(DeepEP → How the toolkit drives it): each
MoE layer keeps its first pass's received tokens ([R, H_pad] bf16), their top-k expert ids (int64)
and weights (fp32), and its combined output ([b·S, H_pad] bf16), and every layer's cache is alive
when the backward starts. DeepEP sends a token once per destination rank, so R = r·b·S with r the
distinct ranks a token's top-k experts land on: ep·(1 − C(E − E/ep, k) / C(E, k)) under uniform
routing, 7.7 / 8.8 / 9.4 at ep = 16 / 32 / 64. H_pad is the hidden rounded up to 256 on the
elastic backend and the hidden itself on legacy (both 4096 here). Under EP+ETP r counts the
ep_size-wide dispatch group; under PP each in-flight microbatch holds its own. At this model it is
8–9× the activation term, and it decides the 2-node cells.
ZeRO-3 would remove the 19 GB resident term but is rejected under EP and under TP+DP
(_reject_fsdp_settings) — every layout on this page is EP, so
fsdp_reshard_after_forward: false is mandatory, not a choice.
Budget. A B300 reports 288 GB (275,040 MiB), and reserved memory is what counts against it. Reserved runs above allocated: Nemotron-H's 32,000-token recipe allocates 227–235 GB and reserves 268–283 GB, 99.4% of the device on its top rank. These plans count allocated memory, so cells are ⚠ between 230 and 260 GB, ✗ above.
The DeepEP arena is a separate term, paid once for the model rather than once per layer. It is
receive-side and linear in the dispatch-group width: at this model's hidden=4096, top_k=10 an
ep64 group at 8192 tokens per rank costs 4.06 GiB on top of the columns below. Formula and
growth law: DeepEP → Buffer sizing. A build or grow takes
1.5x the capacity needed, so the arena can sit up to 1.5x above that figure.
It does not shape the choice of ep_size at any cell here; a wide dispatch group keeps its
expert-memory saving instead of spending it back. The one exception is HALO_EP_CAPACITY_DEDUP=0,
which gives every layer a private arena and multiplies that figure by the 60 MoE layers to
243.8 GiB, past the card.
When the arena does not fit, the failure is a CUDA OOM inside ElasticBuffer at the first MoE
forward, after the whole model has loaded, with nothing in the message naming EP capacity.
What loads at all
Only three loaders shard at load time: WeightAction.EXPERT_SHARD
(halo/model_loading/lazy_safetensors/, planned by WeightPlanner in
halo/model_loading/parallel/lazy_plan.py), the PP stage loader (its stage's layers, plus the EP
slice under PP+EP) and HF tp_plan="auto" for dense models.
Every other branch of _dispatch_model_loading materializes all 396.8 B on one GPU: 794 GB, never
starts. That kills plain FSDP2, pure TP (MoE TP has no shard-aware load), pure ETP (ep_size == 1
sets experts_per_rank = num_experts, so every rank reads every expert) and pure CP before any
memory argument. A dense model meets the same ceiling: plain FSDP2/DDP loads it whole onto every GPU,
and TP is the route past one GPU (Data Parallelism).
The EP lazy loader slices by ep_size, not X — ETP partners share dispatch_ep_rank and load
the same experts at full intermediate width. The intermediate dim is split later, at EP patching
(_init_fused_glu_params / _store_separate_glu_params,
halo/distributed/expert_parallel/moe_layer/compute.py), which also holds one layer's transposed full-width
copy alongside the new shard. So ep2 + etp8 has the same steady state as ep16 and a 407 GB
load peak.
Scenario matrix
X = ep_size × expert_tp_size. Steady = experts + replicated + FSDP non-expert + activations +
GC replay cache at b1. Load = the separate per-rank peak while weights stream in. All EP cells need
512 % ep_size == 0, rejected by reject_unsupported_model off config.json
at the top of the model load — after the checkpoint download, before the process groups, the meta
shell or any weight read.
Read the Layout column for the fabric too: on an 8-GPU domain every global row above one node
puts its dispatch and combine on RDMA, 240 times per microbatch; every node row keeps them on
NVLink.
One node cannot train this model, in any shape. Routed experts are only ever divided by X,
which on 8 GPUs is at most 8, so every legal single-node shape puts the same 386 GB of expert state
on each GPU against a B300's 288 GB.
Two nodes with ep16 --ep_scope=global is the floor, where full fine-tuning fits only with frozen
layers or at s4096; four is the first comfortable size. Freezing
the experts is the single-node escape: LoRA on EP experts, or unfreeze_layers_patterns, drops them
from 8 to 2 bytes/param. That is arithmetic, not a validated cell.
Steady 32k is not reachable on the global rows. Cross-node EP dispatches
per_device_train_batch_size × max_length tokens per rank in one MoE forward (under PP, one microbatch's rows × max_length).
reject_unsupported_model refuses a declared budget above
HALO_DEEPEP_GIN_MAX_TOKENS_PER_RANK (8192) before any weight is read, and the dispatcher re-checks at
buffer sizing; above it a proxy-GIN dispatch wedges in transit instead of erroring.
So every ep_scope=global row is capped at 8192 tokens/rank whatever its memory column says. The
node-scope rows have no such cap, though intra-node ep8 at ≥~64k tokens/rank needs ep_buffer_backend: legacy
(DeepEP → Transport backend).
TP is a net loss on this model. apply_tp_to_attention_only shards only q/k/v/o of the 15
full-attention layers: 1.57 B of 10.25 B. It leaves the 45 gated-DeltaNet layers untouched, and
leaves embeddings and lm_head replicated by design. _find_attention probes
self_attn/attention/attn, and the linear block is linear_attn, so the loop skips those layers
and warns which ones it left replicated.
What TP does do is divide data_parallel_size, which divides F, so it raises the non-expert
term.
CP is rejected outright. reject_unsupported_ulysses_model
(halo/distributed/context_parallel/model_rules.py) raises on Qwen3_5MoeGatedDeltaNet and, on a meta
shell, on any linear_attention entry in layer_types: causal_conv1d_fn needs a cross-rank halo
and chunk_gated_delta_rule is a recurrent scan. Long context comes from the activation headroom at
4–8 nodes, not from CP.
The 3-node case
24 GPUs is three NVLink domains, and nothing in the code requires a power of two — world_size % nvlink_domain_size == 0 passes and ParallelismConfig validates world 24 exactly as it does world
16. The model is what fails, in two independent ways.
Global-scope EP on 24 must be a multiple of 3. CrossNodeLayout (built by
_reject_ep_group) requires ep_group_size % num_domains == 0 with num_domains = 3, and
nvlink_domain_size % members_per_domain == 0, leaving exactly ep_size ∈ {3, 6, 12, 24}. 512 is
2⁹, so every one of them fails 512 % ep_size == 0.
reject_unsupported_model raises that off config.json, at the top of the model load and
before any process group. _reject_ep_group refuses the rest at global scope in order:
ep_size=16 first, on reject_cross_node_ep_group (24 % 16 ≠ 0), then ep_size ∈ {2, 4, 8} on
CrossNodeLayout, because they cannot tile 3 domains equally.
The single EP+ETP shape at world 24 is ep_size=3, expert_tp_size=8, which dies on the same
divisor.
Node-scope EP caps X at 8. _reject_ep_group requires nvlink_domain_size % ep_group_size == 0, so ep_size ∈ {2, 4, 8} — and 8·P_e/8 = 386 GB of experts per rank before anything else. The
same ceiling makes node-scope EP useless at 2, 4 and 8 nodes too.
No EP layout survives without PP; at 24 GPUs only pp3 × ep8 fits outside experimental PP+ETP. Each stage is one
node holding 24, 20 or 16 of the 60 layers, so ep8 node-local divides the 512 experts and stage 0
carries 155 GB of them (Pipeline-parallel cells). pp3 × etp8 passes the
config gate with the same expert share per rank, but PP+ETP is verified on tiny models only; pp3 × ep2
passes and does not fit. Without PP, use 2 nodes
(ep16 global) or 4 (ep32 global).
Pipeline-parallel cells
Qwen3_5MoeTextModel is PP-registered, so whole-node stages
with ep8 inside each keep every dispatch on NVLink; only the boundary activations cross RDMA. The
default split is head-weighted on the 4-layer layer_types period — [32, 28] at pp2,
[24, 20, 16] at pp3, [16, 16, 16, 12] at pp4 — so stage 0 is the largest, and its matrix row
changes in three ways:
- the experts and the replicated router and shared expert hold only the stage's layers:
8·P_e·L_stage/60/ep_size; - the FSDP term adds the unsharded stage gradient the schedule holds across the microbatch loop:
4·P_f,stage + 8·P_f,stage/F, withFthe stage's 8 ranks; - under 1F1B the first stage keeps up to
pp_sizemicrobatches' layer inputs and GC replay caches in flight.
What PP buys here: the 8192 tokens/rank cross-node dispatch ceiling does not apply (node-local ep8
still needs ep_buffer_backend: legacy past ~64k tokens per rank), and 3 nodes become usable. What it
costs, and why an EP layout stays the recommendation wherever one fits:
- No router balancing. The split severs the aux-loss path and Qwen3.5 has no checkpoint slot for a
bias, so the run takes
moe_balancing: none: 512 experts at top-10 train unbalanced (PP + EP). - The bubble.
9 × (pp_size − 1)microbatches hold it at 10%, andper_device_train_batch_sizemust be a multiple of the microbatch count (Configuration). - The PP refusals.
save_sharded_ep, PEFT and expert LoRA, and layer-indexedunfreeze_layers_patternsare refused, andfp32_grad_reducedoubles the held gradient (Rejected combinations).
Recommended cell
On an 8-GPU domain every EP-only layout puts the dispatch on RDMA — pick the widest EP group that loads, and narrow the dispatch group (EP+ETP) when the fabric is latency-bound. PP keeps it on NVLink at the costs above.
On NVL72, declare a 64-GPU domain, not 72. _reject_ep_group requires
nvlink_domain_size % ep_group_size == 0, and 72 % 64 = 8.
At NVLINK_DOMAIN_SIZE=72 the only EP widths that survive validation are ep2 and ep72:
ep4/ep8 fall to the racy-EP guard (scope-blind on a single domain), ep16/ep32/ep64 to the
divisibility rule. 512 % 72 ≠ 0 then kills ep72 at model load. A 512-expert model has no legal
EP shape on a full 72-wide rack.
The working recipe is NVLINK_DOMAIN_SIZE=64 over 16 of the rack's 18 compute trays, with
ep_size=64, ep_scope=node: one dispatch group per rack, 8 experts/rank, ~48 GB of expert state,
requires_rdma false. It costs 8 of 72 GPUs, 11% idle. The general form
— the domain is the tiling unit, so the legal ep_group_size values are its divisors — is on
Multi-Node.
Scaling out, world=512 over 8 racks at domain 64 accepts every node-scope ep_size dividing 64
(the racy guard is inert across multiple domains); ep64 gives 8 EP groups, one per rack, averaged by
the deferred cross-replica sweep. Like the rest of this page, that is rank math and arithmetic: a plan,
not a measurement.
Frozen layers are the lever at 2 nodes. unfreeze_layers_patterns leaves untouched parameters at
2 B/param — weights only, no gradient and no optimizer state. Unfreezing the top 20 of 60 layers plus
routers and lm_head takes the expert term from 8·P_e/X to 4·P_e/X — 193 → 97 GB at X=16.
When the fabric is the bottleneck, narrow the dispatch group. Cross-node EP over EFA is proxy-Gin latency-bound (DeepEP → AWS EFA).
ep64 gives each rank 56 cross-node dispatch partners; ep8 + expert_tp_size=8 gives the same X=64 with 7,
because _reject_expert_tp forces exactly one ETP group per NVLink domain
(expert_tp_size == members_per_domain, ep_size == num_domains). EP+ETP is an experimental axis —
its expert-TP reduce runs in token space outside the DeepEP dispatch→combine span.
On InfiniBand with IBGDA the latency wall does not exist; use plain ep64 — but size the microbatch against the arena
term above, which ep8 + etp8 does not pay (its dispatch group is 8 wide, so its arena is an eighth
of ep64's at the same tokens/rank).
Launch recipes
One torchrun per node. No 397B config ships — derive one from the 122B sibling per
YAML knobs.
Bash
torchrun --nnodes=$NNODES --node_rank=$NODE_RANK --nproc_per_node=8 \
--rdzv_backend=c10d --rdzv_endpoint=$MASTER_ADDR:29500 \
scripts/training/sft.py <your-397b-config>.yaml <layout flags>Environment
The fabric env (EFA vs InfiniBand, proxy GIN) is the standard one from Multi-Node → RDMA fabrics, the rest from Launch Recipes. What this model changes:
Bash
export DIST_NCCL_TIMEOUT_MINUTES=90 # default 30; a 397B gathered save takes minutes
export DIST_STORE_TIMEOUT_HOURS=8 # default 4; a single-rank 794 GB download can outlast it
# GB200/GB300 NVL72 only — 64, not 72: ep_group_size must DIVIDE the domain, and 512 % 72 != 0.
# Launch on 16 of the rack's 18 compute trays; the other 8 GPUs sit idle.
export NVLINK_DOMAIN_SIZE=64PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True composes with the ElasticBuffer and answers
allocator fragmentation under variable-shape packing. Under EP it costs throughput, so set it only
when the run does not fit without it
(measured).
YAML knobs
Base the config on examples/sft/qwen3_5/qwen3.5-122b-a10b-ep.yaml — the closest shipped shape.
Values that differ from it or are load-bearing at 397B:
Pre-shard the corpus with scripts/before_training/prepare_dataset.py --num_shards N,
N >= data_parallel_size — at 64 ranks the on-the-fly path materializes the dataset per rank.
Other 300B-class checkpoints
The same arithmetic (8 B/param on the trained experts, X = ep_size × expert_tp_size, the
non-expert weights gathered whole under ZeRO-2) for the three other checkpoints in the 200–320B
class, on 8 × B300 (Budget). Every figure is a plan, not
a measurement, except the three GLM-5.3-Flash rows marked measured (rates).
The ep16 rows all sit on cross-node EP and inherit its per_device_train_batch_size × max_length ≤ 8192 contract.
Pre-flight checklist
Before the launch
512 % ep_size == 0,expert_tp_size | moe_intermediate_size, TP head divisibility and the declared tokens/rank budget are checked offconfig.jsonbefore any weight is read (reject_unsupported_model, at the top ofload_distributed_model, after the checkpoint download); the Ulysses and PP structure refusals run on a meta-device shell, also before any weight is read.Scratch: a single checkpoint is ~800 GB with
save_only_model: true(as shipped), ~2.4 TB with optimizer shards (AdamWBF16 adds 4 B/param).Rotation deletes the previous checkpoint only after the new one completes, so
save_total_limit: 1still transiently holds both: budget ~1.6 TB free per save, ~4.8 TB with optimizer shards. The trainer checks the peak againstoutput_dir's volume at construction (Checkpoints). Confirm the volume withfindmnt/df -hbefore pointingHF_HOME,HF_DATASETS_CACHE,TMPDIRandHALO_DATA_ROOTat it.On the
save_sharded_epcells budget for the merge too:merge_ep_shards.pystreams (peak host RAM ≈ one merged MoE layer, not the ~800 GB artifact), and--delete_input_shardsfrees the per-rank shards only once the merged checkpoint is complete, so the volume briefly holds both.Host RAM at load: the EP lazy path reads safetensors straight to GPU and needs almost none. Every slice is a view over the mmapped shard, so anonymous RSS stays near the process baseline (about one tensor at a time) and
max_concurrent_loadingnever applies.What
freeshows is page cache the size of the checkpoint (~800 GB here; 398 / 628 / 582 GB for the Step-3.7 / GLM-5.3 / DeepSeek-V4 Flash artifacts of the 300B table), shared by the node's ranks and reclaimable. Each rank's file-backed RSS reaches its dense + own expert slices while its shard handles are open. Not a leak.The gate+up fan-in's 2× transient lives on the GPU (~1 GB per layer). If
ep_lazy_loadingis off or the checkpoint layout is lazy-incompatible, the fallback stages the full model in CPU RAM per concurrent loader; setmax_concurrent_loading: 1whenfree -gis tight.fp32_expertsdoubles the expert bank it stages (EP → Model loading).Host RAM at save and resume: a run that keeps optimizer state (
save_only_model: false) passes each rank's optimizer shard through host RAM: ~28 GB/rank atep64, ~102 GB/rank atep16. Atep16an 8-GPU node holds ~410 GB during a save (the defaultmax_concurrent_loadingadmits 4 ranks at a time) and ~820 GB during a resume, where every rank keeps its copy until the collective apply.Every recommended cell runs one EP group, so each rank's expert state is unique; a job wide enough to run several EP groups (they are DP replicas) writes that half once per replica group instead (Checkpoints).
A
save_sharded_epsave, with or without optimizer state, also holds each rank's whole expert slice in host RAM at once, unthrottled: ~48 GB/rank and ~386 GB per 8-GPU node atep16, halving at each doubling ofep_size. Rank 0 holds ~20 GB more for the replicated half.Hardware gates on every node:
scripts/profiling/nvlink_health.py(non-zero exit = a degraded lane that silently caps or hangs the EP all-to-all),nvidia-smi(8 GPUs, near-zero used memory, no ECC/Xid), and the fabric check for your interconnect (RDMA fabrics).Dataset shards:
num_shards >= data_parallel_size, which isstage_world_size / max(tp, cp, etp), not the GPU count.Dry-run with the smaller Qwen3.5 MoE sibling (
Qwen/Qwen3.5-35B-A3B) before the real weights load. One 8-GPU host cannot form a recommended cell, since each needs its own node count. Twotorchrunagents on that host (--nnodes=2 --nproc_per_node=4) trainpp2×ep4and pass the config gates forep8 --ep_scope=globalandep2+etp4. For the real cell's config verdict, build aParallelismConfigwith itsworld_sizeandgpus_per_nodeon CPU.
After step 1
Source on GitHubUpdated 18 November 2026