Large-Scale Scenarios — Qwen3.5-397B-A17B

Per-rank memory arithmetic, reachable parallelism layouts and launch recipes for Qwen/Qwen3.5-397B-A17B on 2, 3, 4 and 8 nodes of 8 × B300 (288 GB). Every constraint here is enforced in code and named by its symbol; the memory columns are arithmetic from the shape below, so treat them as a plan, not a measurement. Mechanism pages: Multi-Node, Expert Parallelism, Launch Recipes, Qwen3.5, GPU Training Theory.

Two facts set every cell and pull against each other. nvlink_domain_size decides whether EP's per-layer all-to-all crosses the slow fabric. The routed experts are 386.5 B of the 396.8 B total, only ever sharded by X = ep_size × expert_tp_size, so they need X ≥ 16 at 8 B/param to fit at all.

On an 8-GPU domain X ≥ 16 forces the EP group across the domain boundary. Pipeline parallelism is the other way across it: whole-node stages keep EP on NVLink, at costs this model feels (Pipeline-parallel cells).

nvlink_domain_size (auto = gpus_per_node; set NVLINK_DOMAIN_SIZE on NVL72) is the locality unit every validator divides by, and it decides which axis is allowed onto RDMA. The dispatcher keys the decision off topology alone: EPConfig.spans_domains = num_domains > 1 and not node_local (halo/distributed/expert_parallel/config.py), on which the elastic backend sets EP_DISABLE_GIN=0, Gin on.

num_domains counts NVLink domains, not OS nodes, so a group inside one NVL72 rack stays on NVLink even though it spans ~18 hosts.

What EP at ep_scope=global across domains puts on that fabric per microbatch, at this model's 60 MoE layers, is 240 all-to-alls over the EP group: forward dispatch + combine per MoE layer, and their backward mirrors — arithmetic from the layer count and the autograd graph, not a benchmark.

Four per layer, not two, because DeepEPDispatchFunction.backward is a combine and DeepEPCombineFunction.backward is a dispatch (halo/distributed/expert_parallel/deepep/dispatcher.py). Gradient checkpointing does not multiply the count — recompute replays cached results rather than re-issuing the collective, and that cache is a memory term of its own (GC replay cache).

The cost is worse than the count suggests: cross-domain dispatch is latency-bound, so its fixed per-collective cost does not amortize as the message grows (DeepEP → AWS EFA).

The fork:

  • Domain = 8 (a standard 8 × B300 node). Where a model fits at ep_group_size <= 8, node-local EP groups replicate across domains as DP replicas, their cross-replica average deferred to a post-backward sweep (Multi-Node); that puts no per-layer traffic on RDMA.

    At 397B that does not fit (386 GB of experts per rank), so the EP group must cross the domain boundary at global scope, or PP must split the layers across domains; when the fabric is latency-bound, narrow the dispatch group with EP+ETP (Recommended cell).

  • Domain = 72 (GB200/GB300 NVL72). A node-scope EP group rides NVLink at any width that divides the declared domain, requires_rdma stays false, and wide EP becomes the attractive shape.

    On this model that means declaring a 64-wide domain, not 72 (Recommended cell); the fabric check and the bring-up advice are in Scale & Limits.

Model shape

Source: hf_hub_download("Qwen/Qwen3.5-397B-A17B", "config.json"), revision 8472618112abcbd45acbcdc58436aff4233c23f7. The checkpoint is a VLM — architectures: ["Qwen3_5MoeForConditionalGeneration"], model_type: qwen3_5_moe — and every field below is from text_config (qwen3_5_moe_text).

FieldValueFieldValue
num_hidden_layers60 (all MoE, mlp_only_layers: [])num_attention_heads32
hidden_size4096num_key_value_heads2
num_experts512head_dim256
num_experts_per_tok10attn_output_gatetrue (q_proj emits 2 × NH·head_dim)
moe_intermediate_size1024vocab_size248320
shared_expert_intermediate_size1024 (one shared expert per layer)tie_word_embeddingsfalse (top level)
intermediate_sizeabsent — no dense FFNmax_position_embeddings262144

layer_types is ["linear_attention"]*3 + ["full_attention"] repeated 15× — 45 gated-DeltaNet layers + 15 full-attention layers, full_attention_interval: 4. The linear layers carry linear_num_value_heads: 64, linear_num_key_heads: 16, head dims 128, linear_conv_kernel_dim: 4.

Parameter counts derived from those fields (fused expert layout gate_up_proj [512, 2048, 4096] + down_proj [512, 1024, 4096]):

GroupParamsSharded by
Routed experts P_e386.55 BX = ep_size × expert_tp_size
Gated-DeltaNet (45)5.31 BFSDP2 only
Full attention (15)1.57 BFSDP2, or TP
embed_tokens + lm_head (untied)2.03 BFSDP2 only
Vision tower0.45 BFSDP2 only
Shared experts + routers P_r0.88 Bnothing — see below
Non-expert P_ne (all four rows above + norms)10.25 B
Total396.80 B
Active per token (top-10 + shared + attention + head)17.35 B

P_r is the trap. EPQwen3_5MoELayer owns the router gate, the shared_expert and shared_expert_gate; the whole layer lands in FSDP2's ignored set (_fsdp_excluded_params), TP never visits it (apply_tp_to_attention_only), and EP does not shard it — replicated_named_params grad-syncs it by hand. So 0.88 B stays as a full 8 B/param copy on every rank ≈ 7 GB that no axis shrinks. Under fp32_router its 0.13 B of router weights are fp32 at 16 B/param, +1 GB.

Per-rank memory model

P_f = P_ne − P_r = 9.37 B is the FSDP-managed non-expert share, F the FSDP shard width for it. F is not data_parallel_size. It defaults to every rank of the job (F = stage_world_size), so ep4 + etp8 on 4 nodes still shards non-expert params 32-way even at DP = 4; ETP partners hold identical replicas of them.

Two topologies narrow F. Under TP the (dp, tp) mesh drops it to DP. Under EPConfig.is_deferred_dp (multi-group EP across domains, the × N replicas rows below) FSDP shards over the EP group, so F = ep_group_size and the cross-replica average moves to the post-backward sweep (Multi-Node).

TermBytes per rank397B value
Routed experts8·P_e/X193 / 97 / 48 GB at X = 16 / 32 / 64
Replicated router + shared expert8·P_r, plus 8·P_router under fp32_router7 GB, always (8 GB with fp32_router)
FSDP non-expert, ZeRO-2 (default)2·P_f + 8·P_f/F19 GB resident + 5 / 2 / 1 GB at F = 16 / 32 / 64
Activations, GC on(2·L·H + 6·k·I_moe)·b·S4.5 GB at b1 s8192, 18.1 GB at b1 s32768
GC replay cache (EP)L_moe·b·S·(r·(2·H_pad + 12·k) + 2·H_pad)35 / 40 / 42 GB at b1 s8192, ep = 16 / 32 / 64; 4× at s32768
Load-time peak (separate)2·(P_ne + P_e/ep_size)see the Load column

8 B/param = 2 weights + 2 grads + 4 optimizer state; AdamWBF16 (halo/optimizers/adamw_bf16.py) allocates exp_avg and exp_avg_sq with zeros_like(p), so both are bf16 and there is no master copy and no error-feedback buffer. fp32_grad_reduce upcasts only inside the reduce — no storage. The stock AdamW is refused under EP unless fp32_non_ep_params is set (why), and fp32 expert masters (fp32_experts) would take the expert term to 16 B/param, where no cell below survives.

The activation term is one bf16 layer input per layer (2·L·H·b·S) plus the recomputed layer's expert intermediates (6·k·I_moe·b·S — gate_up, SiLU product, down input at top-k = 10). The 45 linear layers additionally hold an fp32 recurrent state of 64 × 128 × 128 × 4 B ≈ 4 MB per sequence per layer (≈190 MB at b=1).

The GC replay cache is what lets the recompute skip the all-to-all (DeepEP → How the toolkit drives it): each MoE layer keeps its first pass's received tokens ([R, H_pad] bf16), their top-k expert ids (int64) and weights (fp32), and its combined output ([b·S, H_pad] bf16), and every layer's cache is alive when the backward starts. DeepEP sends a token once per destination rank, so R = r·b·S with r the distinct ranks a token's top-k experts land on: ep·(1 − C(E − E/ep, k) / C(E, k)) under uniform routing, 7.7 / 8.8 / 9.4 at ep = 16 / 32 / 64. H_pad is the hidden rounded up to 256 on the elastic backend and the hidden itself on legacy (both 4096 here). Under EP+ETP r counts the ep_size-wide dispatch group; under PP each in-flight microbatch holds its own. At this model it is 8–9× the activation term, and it decides the 2-node cells.

ZeRO-3 would remove the 19 GB resident term but is rejected under EP and under TP+DP (_reject_fsdp_settings) — every layout on this page is EP, so fsdp_reshard_after_forward: false is mandatory, not a choice.

Budget. A B300 reports 288 GB (275,040 MiB), and reserved memory is what counts against it. Reserved runs above allocated: Nemotron-H's 32,000-token recipe allocates 227–235 GB and reserves 268–283 GB, 99.4% of the device on its top rank. These plans count allocated memory, so cells are ⚠ between 230 and 260 GB, ✗ above.

The DeepEP arena is a separate term, paid once for the model rather than once per layer. It is receive-side and linear in the dispatch-group width: at this model's hidden=4096, top_k=10 an ep64 group at 8192 tokens per rank costs 4.06 GiB on top of the columns below. Formula and growth law: DeepEP → Buffer sizing. A build or grow takes 1.5x the capacity needed, so the arena can sit up to 1.5x above that figure.

It does not shape the choice of ep_size at any cell here; a wide dispatch group keeps its expert-memory saving instead of spending it back. The one exception is HALO_EP_CAPACITY_DEDUP=0, which gives every layer a private arena and multiplies that figure by the 60 MoE layers to 243.8 GiB, past the card.

When the arena does not fit, the failure is a CUDA OOM inside ElasticBuffer at the first MoE forward, after the whole model has loaded, with nothing in the message naming EP capacity.

What loads at all

Only three loaders shard at load time: WeightAction.EXPERT_SHARD (halo/model_loading/lazy_safetensors/, planned by WeightPlanner in halo/model_loading/parallel/lazy_plan.py), the PP stage loader (its stage's layers, plus the EP slice under PP+EP) and HF tp_plan="auto" for dense models.

Every other branch of _dispatch_model_loading materializes all 396.8 B on one GPU: 794 GB, never starts. That kills plain FSDP2, pure TP (MoE TP has no shard-aware load), pure ETP (ep_size == 1 sets experts_per_rank = num_experts, so every rank reads every expert) and pure CP before any memory argument. A dense model meets the same ceiling: plain FSDP2/DDP loads it whole onto every GPU, and TP is the route past one GPU (Data Parallelism).

The EP lazy loader slices by ep_size, not X — ETP partners share dispatch_ep_rank and load the same experts at full intermediate width. The intermediate dim is split later, at EP patching (_init_fused_glu_params / _store_separate_glu_params, halo/distributed/expert_parallel/moe_layer/compute.py), which also holds one layer's transposed full-width copy alongside the new shard. So ep2 + etp8 has the same steady state as ep16 and a 407 GB load peak.

Scenario matrix

X = ep_size × expert_tp_size. Steady = experts + replicated + FSDP non-expert + activations + GC replay cache at b1. Load = the separate per-rank peak while weights stream in. All EP cells need 512 % ep_size == 0, rejected by reject_unsupported_model off config.json at the top of the model load — after the checkpoint download, before the process groups, the meta shell or any weight read.

Read the Layout column for the fabric too: on an 8-GPU domain every global row above one node puts its dispatch and combine on RDMA, 240 times per microbatch; every node row keeps them on NVLink.

NodesLayoutXDPExpertsNon-expSteady 8kSteady 32kLoadVerdict
1 (8)ep8 node8838635>420—117✗ experts alone are 1.5× HBM
1ep4+etp2 · ep2+etp484 · 238635>420—214 · 407✗ same expert share — X cannot exceed the 8 GPUs
2 (16)ep16 global16161933126338369✗ the replay cache takes full fine-tuning past the budget; frozen layers or s4096 (244, ⚠) fit
2ep16 global + tp81621935528840769✗ TP collapses F to 2
2ep2 + etp8 global16219331240291407✗ load peak
2ep8 node · ep4 node · ep2 node≤81638635–63>420—117–407✗ experts alone exceed HBM
2pp2 × ep8 node8 per stage82062726837366✗ stage 0 holds 32 of the 60 layers and two microbatches' replay caches (PP cells)
3 (24)ep3/6/12/24 global3–2424—————✗ 512 % ep_size ≠ 0 at load (The 3-node case)
3ep2/4/8 node≤82438635–63>420—117–407✗ experts alone exceed HBM
3ep3 + etp8 global24312929178240278✗ 512 % 3 ≠ 0, and the ep_size-governed load peak would not fit either
3ep2/4/8/16 global———————✗ rejected at config (_reject_ep_group)
3pp3 × ep8 node8 per stage81552221633451✓ at 8k, the one legal 3-node layout, at the PP costs; ✗ at 32k
4 (32)ep32 global3232972816930345✓
4ep4 + etp8 global3249728149221214⚠ load peak, experimental axis
4ep32 global + tp8324973918031445✓ but strictly worse than ep32
4ep16 global × 2 replicas16321933126338369✗ as ep16 at 2 nodes; both groups still span all 4 domains (4 contiguous ranks per domain — column-block layout), so dispatch stays on RDMA. The win is the narrower 16-rank dispatch group, not domain locality
4pp4 × ep8 node8 per stage81031615425836✓ EP on NVLink and no 8192 tokens/rank cap, at the PP costs; ⚠ at 32k
8 (64)ep64 global6464482712226333✓ memory to spare; tokens/rank still capped at 8192
8ep8 + etp8 global6484827108206117✓ EFA-latency pick, experimental axis
8ep32 global × 2 replicas3264972816930345✓ narrower EP group
8ep16 global × 4 replicas16641933126338369✗ as ep16 at 2 nodes
anyplain FSDP2 · pure TP · pure ETP · pure CP——————794✗ no shard-aware load
anyany EP + CP———————✗ reject_unsupported_ulysses_model

One node cannot train this model, in any shape. Routed experts are only ever divided by X, which on 8 GPUs is at most 8, so every legal single-node shape puts the same 386 GB of expert state on each GPU against a B300's 288 GB.

Two nodes with ep16 --ep_scope=global is the floor, where full fine-tuning fits only with frozen layers or at s4096; four is the first comfortable size. Freezing the experts is the single-node escape: LoRA on EP experts, or unfreeze_layers_patterns, drops them from 8 to 2 bytes/param. That is arithmetic, not a validated cell.

Steady 32k is not reachable on the global rows. Cross-node EP dispatches per_device_train_batch_size × max_length tokens per rank in one MoE forward (under PP, one microbatch's rows × max_length). reject_unsupported_model refuses a declared budget above HALO_DEEPEP_GIN_MAX_TOKENS_PER_RANK (8192) before any weight is read, and the dispatcher re-checks at buffer sizing; above it a proxy-GIN dispatch wedges in transit instead of erroring.

So every ep_scope=global row is capped at 8192 tokens/rank whatever its memory column says. The node-scope rows have no such cap, though intra-node ep8 at ≥~64k tokens/rank needs ep_buffer_backend: legacy (DeepEP → Transport backend).

TP is a net loss on this model. apply_tp_to_attention_only shards only q/k/v/o of the 15 full-attention layers: 1.57 B of 10.25 B. It leaves the 45 gated-DeltaNet layers untouched, and leaves embeddings and lm_head replicated by design. _find_attention probes self_attn/attention/attn, and the linear block is linear_attn, so the loop skips those layers and warns which ones it left replicated.

What TP does do is divide data_parallel_size, which divides F, so it raises the non-expert term.

CP is rejected outright. reject_unsupported_ulysses_model (halo/distributed/context_parallel/model_rules.py) raises on Qwen3_5MoeGatedDeltaNet and, on a meta shell, on any linear_attention entry in layer_types: causal_conv1d_fn needs a cross-rank halo and chunk_gated_delta_rule is a recurrent scan. Long context comes from the activation headroom at 4–8 nodes, not from CP.

The 3-node case

24 GPUs is three NVLink domains, and nothing in the code requires a power of two — world_size % nvlink_domain_size == 0 passes and ParallelismConfig validates world 24 exactly as it does world 16. The model is what fails, in two independent ways.

Global-scope EP on 24 must be a multiple of 3. CrossNodeLayout (built by _reject_ep_group) requires ep_group_size % num_domains == 0 with num_domains = 3, and nvlink_domain_size % members_per_domain == 0, leaving exactly ep_size ∈ {3, 6, 12, 24}. 512 is 2⁹, so every one of them fails 512 % ep_size == 0.

reject_unsupported_model raises that off config.json, at the top of the model load and before any process group. _reject_ep_group refuses the rest at global scope in order: ep_size=16 first, on reject_cross_node_ep_group (24 % 16 ≠ 0), then ep_size ∈ {2, 4, 8} on CrossNodeLayout, because they cannot tile 3 domains equally.

The single EP+ETP shape at world 24 is ep_size=3, expert_tp_size=8, which dies on the same divisor.

Node-scope EP caps X at 8. _reject_ep_group requires nvlink_domain_size % ep_group_size == 0, so ep_size ∈ {2, 4, 8} — and 8·P_e/8 = 386 GB of experts per rank before anything else. The same ceiling makes node-scope EP useless at 2, 4 and 8 nodes too.

No EP layout survives without PP; at 24 GPUs only pp3 × ep8 fits outside experimental PP+ETP. Each stage is one node holding 24, 20 or 16 of the 60 layers, so ep8 node-local divides the 512 experts and stage 0 carries 155 GB of them (Pipeline-parallel cells). pp3 × etp8 passes the config gate with the same expert share per rank, but PP+ETP is verified on tiny models only; pp3 × ep2 passes and does not fit. Without PP, use 2 nodes (ep16 global) or 4 (ep32 global).

Pipeline-parallel cells

Qwen3_5MoeTextModel is PP-registered, so whole-node stages with ep8 inside each keep every dispatch on NVLink; only the boundary activations cross RDMA. The default split is head-weighted on the 4-layer layer_types period — [32, 28] at pp2, [24, 20, 16] at pp3, [16, 16, 16, 12] at pp4 — so stage 0 is the largest, and its matrix row changes in three ways:

  • the experts and the replicated router and shared expert hold only the stage's layers: 8·P_e·L_stage/60/ep_size;
  • the FSDP term adds the unsharded stage gradient the schedule holds across the microbatch loop: 4·P_f,stage + 8·P_f,stage/F, with F the stage's 8 ranks;
  • under 1F1B the first stage keeps up to pp_size microbatches' layer inputs and GC replay caches in flight.

What PP buys here: the 8192 tokens/rank cross-node dispatch ceiling does not apply (node-local ep8 still needs ep_buffer_backend: legacy past ~64k tokens per rank), and 3 nodes become usable. What it costs, and why an EP layout stays the recommendation wherever one fits:

  • No router balancing. The split severs the aux-loss path and Qwen3.5 has no checkpoint slot for a bias, so the run takes moe_balancing: none: 512 experts at top-10 train unbalanced (PP + EP).
  • The bubble. 9 × (pp_size − 1) microbatches hold it at 10%, and per_device_train_batch_size must be a multiple of the microbatch count (Configuration).
  • The PP refusals. save_sharded_ep, PEFT and expert LoRA, and layer-indexed unfreeze_layers_patterns are refused, and fp32_grad_reduce doubles the held gradient (Rejected combinations).

On an 8-GPU domain every EP-only layout puts the dispatch on RDMA — pick the widest EP group that loads, and narrow the dispatch group (EP+ETP) when the fabric is latency-bound. PP keeps it on NVLink at the costs above.

NodesCellCrosses onSteady b1 s8192Why
2ep16 --ep_scope=global + unfreeze_layers_patternsRDMA all-to-all≈155 GB partial (263 GB full FT, ✗)the only layout that loads; full fine-tuning fits only at s4096, no headroom for 32k or b>1
3pp3 × ep8 node, moe_balancing: noneboundary activations only216 GBthe one layout that fits outside experimental PP+ETP (The 3-node case); trains unbalanced
4ep32 --ep_scope=globalRDMA all-to-all169 GBfirst comfortable point on memory — but the binding limit is the 8192 tokens/rank cross-node dispatch ceiling, not the 90 GB spare
8ep64 --ep_scope=globalRDMA all-to-all122 GBmemory headroom to spare, same 8192 tokens/rank ceiling on b × max_length; ep8 + etp8 if the fabric is EFA

On NVL72, declare a 64-GPU domain, not 72. _reject_ep_group requires nvlink_domain_size % ep_group_size == 0, and 72 % 64 = 8.

At NVLINK_DOMAIN_SIZE=72 the only EP widths that survive validation are ep2 and ep72: ep4/ep8 fall to the racy-EP guard (scope-blind on a single domain), ep16/ep32/ep64 to the divisibility rule. 512 % 72 ≠ 0 then kills ep72 at model load. A 512-expert model has no legal EP shape on a full 72-wide rack.

The working recipe is NVLINK_DOMAIN_SIZE=64 over 16 of the rack's 18 compute trays, with ep_size=64, ep_scope=node: one dispatch group per rack, 8 experts/rank, ~48 GB of expert state, requires_rdma false. It costs 8 of 72 GPUs, 11% idle. The general form — the domain is the tiling unit, so the legal ep_group_size values are its divisors — is on Multi-Node.

Scaling out, world=512 over 8 racks at domain 64 accepts every node-scope ep_size dividing 64 (the racy guard is inert across multiple domains); ep64 gives 8 EP groups, one per rack, averaged by the deferred cross-replica sweep. Like the rest of this page, that is rank math and arithmetic: a plan, not a measurement.

Frozen layers are the lever at 2 nodes. unfreeze_layers_patterns leaves untouched parameters at 2 B/param — weights only, no gradient and no optimizer state. Unfreezing the top 20 of 60 layers plus routers and lm_head takes the expert term from 8·P_e/X to 4·P_e/X — 193 → 97 GB at X=16.

When the fabric is the bottleneck, narrow the dispatch group. Cross-node EP over EFA is proxy-Gin latency-bound (DeepEP → AWS EFA).

ep64 gives each rank 56 cross-node dispatch partners; ep8 + expert_tp_size=8 gives the same X=64 with 7, because _reject_expert_tp forces exactly one ETP group per NVLink domain (expert_tp_size == members_per_domain, ep_size == num_domains). EP+ETP is an experimental axis — its expert-TP reduce runs in token space outside the DeepEP dispatch→combine span.

On InfiniBand with IBGDA the latency wall does not exist; use plain ep64 — but size the microbatch against the arena term above, which ep8 + etp8 does not pay (its dispatch group is 8 wide, so its arena is an eighth of ep64's at the same tokens/rank).

Launch recipes

One torchrun per node. No 397B config ships — derive one from the 122B sibling per YAML knobs.

Bash

torchrun --nnodes=$NNODES --node_rank=$NODE_RANK --nproc_per_node=8 \
  --rdzv_backend=c10d --rdzv_endpoint=$MASTER_ADDR:29500 \
  scripts/training/sft.py <your-397b-config>.yaml <layout flags>
NodesLayout flags
2--expert_parallel_size=16 --ep_scope=global
3--pipeline_parallel_size=3 --expert_parallel_size=8 --ep_scope=node, with the PP keys of Pipeline-parallel cells (moe_balancing: none, max_length, pipeline_microbatches)
4--expert_parallel_size=32 --ep_scope=global
8--expert_parallel_size=64 --ep_scope=global
8, EFA--expert_parallel_size=8 --expert_tensor_parallel_size=8 --ep_scope=global

Environment

The fabric env (EFA vs InfiniBand, proxy GIN) is the standard one from Multi-Node → RDMA fabrics, the rest from Launch Recipes. What this model changes:

Bash

export DIST_NCCL_TIMEOUT_MINUTES=90    # default 30; a 397B gathered save takes minutes
export DIST_STORE_TIMEOUT_HOURS=8      # default 4; a single-rank 794 GB download can outlast it
# GB200/GB300 NVL72 only — 64, not 72: ep_group_size must DIVIDE the domain, and 512 % 72 != 0.
# Launch on 16 of the rack's 18 compute trays; the other 8 GPUs sit idle.
export NVLINK_DOMAIN_SIZE=64

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True composes with the ElasticBuffer and answers allocator fragmentation under variable-shape packing. Under EP it costs throughput, so set it only when the run does not fit without it (measured).

YAML knobs

Base the config on examples/sft/qwen3_5/qwen3.5-122b-a10b-ep.yaml — the closest shipped shape. Values that differ from it or are load-bearing at 397B:

KnobValueWhy
expert_parallel_size / ep_scopeper the table above / globalnode-scope EP caps X at 8 and cannot hold P_e
attn_implementationflash_attention_2unset, these packed rows load FA2 on Blackwell too (unpacked rows take FA4, Qwen3.5 → Configs) and Hopper picks FA3; the pin keeps Hopper on FA2 as well. An explicit FA2 is kept as-is
packing / padding_freetrue / falsefixed-length rows, as the shipped recipe
fsdp_reshard_after_forwardfalseZeRO-3 is rejected under EP
gradient_checkpointingtruethe activation column assumes it
bf16 + optimas shippedAdamWBF16 auto-enables whenever bf16: true and optim is adamw_torch_fused/adamw_torch outside accelerate-managed DDP — both are already the defaults, so neither key has to be written to get the 8 B/param cost. optim: muon or flash_adamw opts out
fp32_experts / fp32_non_ep_paramsomitthe 35B configs set them; fp32 masters would take the expert term to 16 B/param and OOM every cell
text_only_modeltrueloads Qwen3_5MoeForCausalLM, which drops the vision tower and MTP tail from the build and the export (Qwen3.5 → EP layer). aux_loss trains on either class
moe_balancingaux_losstrains and exports by construction. Strict bias_update raises: Qwen3.5 has no checkpoint slot for a routing bias (Callbacks)
router_balancing_ratewell below 1e-3, only under bias_update_transient (the 122B recipe's mode)refused under an explicit aux_loss or none. The sign step lands on softmax probabilities, whose uniform scale is 1/512 here — the default γ is ~50% of it
fp32_routertrue+1 GB per rank (the 0.13 B router weights at 16 B/param, not in the matrix columns), and 512 experts at top-10 is where bf16 routing logits start to tie
fp32_grad_reducetruebf16 sums lose precision at 16–64 ranks; no storage cost outside PP, where it doubles the held stage gradient
use_grouped_gemm / use_liger_kerneltrue / true (the defaults)the fused [512, 2048, 4096] layout is what grouped_mm wants on SM100
liger_kernel_config.fused_linear_cross_entropytrueopt-in: this family defaults to Liger's plain cross-entropy. The fused loss never materializes [b, S, 248320] — 4 GB per copy at S=8192
max_concurrent_loadingleave unsetnode-local wave gate; unset adapts to the node (min(4, max(1, local_world_size // 2)) — 4 on an 8-GPU node, 2 on a 4-GPU tray), and any explicit value is used verbatim. The lazy EP path bypasses it, so at load it bounds only the EP fallback; a resume's optimizer-shard reads take it too
save_sharded_eptrue on the pure-EP cells (ep16/ep32/ep64)every rank writes its own shard instead of funneling 794 GB through one. Requires a single EP group spanning all ranks (ep_group_size == world_size — exactly these cells) and a shared output filesystem; rejected under ETP, so the ep8+etp8 cell takes the layer-streaming gathered save instead. Merge before serving (Checkpoints)
per_device_train_batch_size1the activation column is b=1; raise only at 8 nodes and only after step 1's peak is known
max_length8192 on every ep_scope=global celldoubling it doubles the activation column — but on cross-node EP the hard limit is the 8192 tokens/rank dispatch ceiling above, not memory. Raise the ceiling only after validating a larger dispatch end-to-end on your fabric (DeepEP → AWS EFA)
unfreeze_layers_patternsrequired at 2 nodessee Recommended cell
save_total_limit: 1 + save_only_model: trueas shippeda bf16 checkpoint is ~800 GB per save

Pre-shard the corpus with scripts/before_training/prepare_dataset.py --num_shards N, N >= data_parallel_size — at 64 ranks the on-the-fly path materializes the dataset per rank.

Other 300B-class checkpoints

The same arithmetic (8 B/param on the trained experts, X = ep_size × expert_tp_size, the non-expert weights gathered whole under ZeRO-2) for the three other checkpoints in the 200–320B class, on 8 × B300 (Budget). Every figure is a plan, not a measurement, except the three GLM-5.3-Flash rows marked measured (rates).

CheckpointRouted expertsCellExpert leg / rankStatic / rankWhat binds
GLM-5.3-Flash (321B; 42 MoE layers × 288 × M=2048, H=4096)304 B2 × 8, ep16 --ep_scope=global19.0 B → 152 GBMeasured, GC on: 68.8 GiB after load, 212.7–214.1 GiB peak, 1,504 tok/s/GPUthe 8192 tokens/rank cross-node dispatch ceiling. Needs the bf16 conversion (~650 GB)
GLM-5.3-Flash304 B2 × 8, pp2 × ep819.0 B → 152 GBMeasured, GC on: 57.1 / 54.0 GiB after load, 218.0–218.7 / 197.5–197.9 GiB peak, 2,423 tok/s/GPUpadded rows only, as packing is refused; 1.61× ep16
GLM-5.3-Flash304 B2 × 8, ep8 with DP38.0 B → 304 GBMeasured: out of memory in the first optimizer step, 243–250 GiB allocated of 267.7optimizer state on 40.2B trainable per rank: weights and gradients fit, AdamWBF16's moments do not
GLM-5.3-Flash on EFA304 B2 × 8, ep2 --expert_tensor_parallel_size=8 --ep_scope=globalsame X=16~200 GB by arithmeticone cross-node dispatch partner per rank instead of 8 (one ETP group per domain, EP+ETP); experimental axis
Step-3.7-Flash (198B; 42 MoE layers × 288 × M=1280, H=4096)190 B1 × 8, ep823.8 B → 190 GB~213 GB (+15 GB gathered non-expert, +6 GB shards)~75 GB left on the 288 GB device for activations and the allocator's reserve at max_length: 8192 with checkpointing — marginal, unmeasured
Step-3.7-Flash190 B2 × 8, ep16 --ep_scope=global11.9 B → 95 GB~115 GBthe 8192 tokens/rank dispatch ceiling
DeepSeek-V4-Flash (284B; 43 MoE layers × 256 × M=2048, H=4096, 3 of them hash-routed)277 B2 × 8, ep16 --ep_scope=global17.3 B → 138 GB~155 GBthe 8192 tokens/rank dispatch ceiling. Needs the bf16 conversion (~580 GB output)

The ep16 rows all sit on cross-node EP and inherit its per_device_train_batch_size × max_length ≤ 8192 contract.

Pre-flight checklist

Before the launch

  • 512 % ep_size == 0, expert_tp_size | moe_intermediate_size, TP head divisibility and the declared tokens/rank budget are checked off config.json before any weight is read (reject_unsupported_model, at the top of load_distributed_model, after the checkpoint download); the Ulysses and PP structure refusals run on a meta-device shell, also before any weight is read.

  • Scratch: a single checkpoint is ~800 GB with save_only_model: true (as shipped), ~2.4 TB with optimizer shards (AdamWBF16 adds 4 B/param).

    Rotation deletes the previous checkpoint only after the new one completes, so save_total_limit: 1 still transiently holds both: budget ~1.6 TB free per save, ~4.8 TB with optimizer shards. The trainer checks the peak against output_dir's volume at construction (Checkpoints). Confirm the volume with findmnt/df -h before pointing HF_HOME, HF_DATASETS_CACHE, TMPDIR and HALO_DATA_ROOT at it.

    On the save_sharded_ep cells budget for the merge too: merge_ep_shards.py streams (peak host RAM ≈ one merged MoE layer, not the ~800 GB artifact), and --delete_input_shards frees the per-rank shards only once the merged checkpoint is complete, so the volume briefly holds both.

  • Host RAM at load: the EP lazy path reads safetensors straight to GPU and needs almost none. Every slice is a view over the mmapped shard, so anonymous RSS stays near the process baseline (about one tensor at a time) and max_concurrent_loading never applies.

    What free shows is page cache the size of the checkpoint (~800 GB here; 398 / 628 / 582 GB for the Step-3.7 / GLM-5.3 / DeepSeek-V4 Flash artifacts of the 300B table), shared by the node's ranks and reclaimable. Each rank's file-backed RSS reaches its dense + own expert slices while its shard handles are open. Not a leak.

    The gate+up fan-in's 2× transient lives on the GPU (~1 GB per layer). If ep_lazy_loading is off or the checkpoint layout is lazy-incompatible, the fallback stages the full model in CPU RAM per concurrent loader; set max_concurrent_loading: 1 when free -g is tight. fp32_experts doubles the expert bank it stages (EP → Model loading).

  • Host RAM at save and resume: a run that keeps optimizer state (save_only_model: false) passes each rank's optimizer shard through host RAM: ~28 GB/rank at ep64, ~102 GB/rank at ep16. At ep16 an 8-GPU node holds ~410 GB during a save (the default max_concurrent_loading admits 4 ranks at a time) and ~820 GB during a resume, where every rank keeps its copy until the collective apply.

    Every recommended cell runs one EP group, so each rank's expert state is unique; a job wide enough to run several EP groups (they are DP replicas) writes that half once per replica group instead (Checkpoints).

    A save_sharded_ep save, with or without optimizer state, also holds each rank's whole expert slice in host RAM at once, unthrottled: ~48 GB/rank and ~386 GB per 8-GPU node at ep16, halving at each doubling of ep_size. Rank 0 holds ~20 GB more for the replicated half.

  • Hardware gates on every node: scripts/profiling/nvlink_health.py (non-zero exit = a degraded lane that silently caps or hangs the EP all-to-all), nvidia-smi (8 GPUs, near-zero used memory, no ECC/Xid), and the fabric check for your interconnect (RDMA fabrics).

  • Dataset shards: num_shards >= data_parallel_size, which is stage_world_size / max(tp, cp, etp), not the GPU count.

  • Dry-run with the smaller Qwen3.5 MoE sibling (Qwen/Qwen3.5-35B-A3B) before the real weights load. One 8-GPU host cannot form a recommended cell, since each needs its own node count. Two torchrun agents on that host (--nnodes=2 --nproc_per_node=4) train pp2 × ep4 and pass the config gates for ep8 --ep_scope=global and ep2 + etp4. For the real cell's config verdict, build a ParallelismConfig with its world_size and gpus_per_node on CPU.

After step 1

CheckExpectedIf not
Peak memory (EfficiencyCallback)within ~15% of the matrix cellrecheck P_e/P_ne; the replicated P_r term is the usual surprise
grad_normfinite, same order as a 35B-A3B run of the same recipesee Debugging
moe/* metrics present and non-zeroMoEMetricsCallback logs them under aux_loss (BalancingBiasCallback under a bias-update mode)this checkpoint keeps its expert fields on text_config, and resolve_expert_count / resolve_router_topk resolve the text config before reading them. Silent zeros mean the model is not being seen as MoE and moe_balancing has resolved to none — the router is training unbalanced
Packing took the varlen pathno dense-mask warning from select_data_collatora non-varlen _attn_implementation pays a dense mask over the flattened batch, and on some families it also stops isolating documents (Document isolation). Pin flash_attention_2
Expert balance (MoEMetricsCallback, or BalancingBiasCallback under a bias-update mode)no expert at 0 tokens across a stepraise router_aux_loss_coef under aux_loss; under bias_update_transient raise router_balancing_rate — but see the 512-expert scale caveat in Callbacks before doubling it
Achieved TFLOPScompare against the MFU registry at 35B-A3B. Its rows are EP-only; tok/s/GPU on an ep8 + etp8 cell reads lower by construction (eight ETP partners share one batch), so compare TFLOPS, which counts each rank's own computea 2×+ shortfall on EFA is the proxy-Gin latency wall: narrow the dispatch group (ep8 + etp8)
Step time variancestable after 3 warmup stepsa slow rank is usually a degraded NVLink lane or a straggler NIC
First checkpointwritten, and re-loadable with from_pretrained — on the save_sharded_ep cells only after merge_ep_shards.py; the artifact itself is per-rank (format: "ep_sharded") and every loader refuses it unmerged. The gathered ep8+etp8 row loads directlyraise DIST_NCCL_TIMEOUT_MINUTES before blaming the fabric

Source on GitHubUpdated 18 November 2026