# Large-Scale Scenarios — Qwen3.5-397B-A17B

Per-rank memory arithmetic, reachable parallelism layouts and launch recipes for
**`Qwen/Qwen3.5-397B-A17B`** on **2, 3, 4 and 8 nodes of 8 × B300 (288 GB)**. Every constraint here
is enforced in code and named by its symbol; the memory columns are arithmetic from the shape below,
so treat them as a plan, not a measurement. Mechanism pages:
[Multi-Node](https://whitecircle.com/halo/docs/agent-docs/parallelism/multi-node.md), [Expert Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/expert-parallelism.md),
[Launch Recipes](https://whitecircle.com/halo/docs/agent-docs/parallelism/launch-recipes.md), [Qwen3.5](https://whitecircle.com/halo/docs/agent-docs/models/qwen3_5.md),
[GPU Training Theory](https://whitecircle.com/halo/docs/agent-docs/reference/gpu-training-theory.md#why-n-gpus-is-not-n-throughput).

Two facts set every cell and pull against each other. `nvlink_domain_size` decides whether EP's
per-layer all-to-all crosses the slow fabric. The routed experts are 386.5 B of the 396.8 B total,
only ever sharded by `X = ep_size × expert_tp_size`, so they need `X ≥ 16` at 8 B/param to fit at
all.

On an 8-GPU domain `X ≥ 16` forces the EP group across the domain boundary.
[Pipeline parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md) is the other way across it: whole-node stages keep EP
on NVLink, at costs this model feels ([Pipeline-parallel cells](#pipeline-parallel-cells)).

## First question: how wide is the NVLink domain?

`nvlink_domain_size` (auto = `gpus_per_node`; set `NVLINK_DOMAIN_SIZE` on NVL72) is the locality unit
every validator divides by, and it decides which axis is allowed onto RDMA. The dispatcher keys the
decision off topology alone: `EPConfig.spans_domains = num_domains > 1 and not node_local`
(`halo/distributed/expert_parallel/config.py`), on which the elastic backend sets `EP_DISABLE_GIN=0`, Gin on.

`num_domains` counts NVLink **domains**, not OS nodes, so a group inside one NVL72 rack stays on NVLink
even though it spans ~18 hosts.

What EP at `ep_scope=global` across domains puts on that fabric per microbatch, at this model's 60
MoE layers, is **240** all-to-alls over the EP group: forward dispatch + combine per MoE layer, and
their backward mirrors — arithmetic from the layer count and the autograd graph, not a benchmark.

Four per layer, not two, because `DeepEPDispatchFunction.backward` *is* a combine and
`DeepEPCombineFunction.backward` *is* a dispatch (`halo/distributed/expert_parallel/deepep/dispatcher.py`).
Gradient checkpointing does not multiply the count — recompute replays cached results rather than
re-issuing the collective, and that cache is a memory term of its own
([GC replay cache](#per-rank-memory-model)).

The cost is worse than the count suggests: cross-domain dispatch is **latency**-bound, so its fixed
per-collective cost does not amortize as the message grows
([DeepEP → AWS EFA](https://whitecircle.com/halo/docs/agent-docs/infrastructure/deepep.md#expert-parallelism-over-aws-efa)).

**The fork:**

- **Domain = 8** (a standard 8 × B300 node). Where a model fits at `ep_group_size <= 8`, node-local
  EP groups replicate across domains as DP replicas, their cross-replica average deferred to a
  post-backward sweep ([Multi-Node](https://whitecircle.com/halo/docs/agent-docs/parallelism/multi-node.md#deferred-cross-replica-sync)); that puts no
  per-layer traffic on RDMA.

    At 397B that does not fit (386 GB of experts per rank), so the EP group must cross the domain
    boundary at global scope, or PP must split the layers across domains; when the fabric is
    latency-bound, narrow the dispatch group with EP+ETP ([Recommended cell](#recommended-cell)).

- **Domain = 72** (GB200/GB300 NVL72). A node-scope EP group rides NVLink at any width that *divides*
  the declared domain, `requires_rdma` stays false, and wide EP becomes the attractive shape.

    On this model that means declaring a **64**-wide domain, not 72
    ([Recommended cell](#recommended-cell)); the fabric check and the bring-up advice are in
    [Scale & Limits](https://whitecircle.com/halo/docs/agent-docs/reference/scale-and-limitations.md).

## Model shape

Source: `hf_hub_download("Qwen/Qwen3.5-397B-A17B", "config.json")`, revision
`8472618112abcbd45acbcdc58436aff4233c23f7`. The checkpoint is a VLM —
`architectures: ["Qwen3_5MoeForConditionalGeneration"]`, `model_type: qwen3_5_moe` — and every field
below is from `text_config` (`qwen3_5_moe_text`).

| Field | Value | Field | Value |
|---|---|---|---|
| `num_hidden_layers` | 60 (all MoE, `mlp_only_layers: []`) | `num_attention_heads` | 32 |
| `hidden_size` | 4096 | `num_key_value_heads` | 2 |
| `num_experts` | **512** | `head_dim` | 256 |
| `num_experts_per_tok` | 10 | `attn_output_gate` | `true` (q_proj emits 2 × `NH·head_dim`) |
| `moe_intermediate_size` | 1024 | `vocab_size` | 248320 |
| `shared_expert_intermediate_size` | 1024 (one shared expert per layer) | `tie_word_embeddings` | **false** (top level) |
| `intermediate_size` | absent — no dense FFN | `max_position_embeddings` | 262144 |

`layer_types` is `["linear_attention"]*3 + ["full_attention"]` repeated 15× —
**45 gated-DeltaNet layers + 15 full-attention layers**, `full_attention_interval: 4`. The linear
layers carry `linear_num_value_heads: 64`, `linear_num_key_heads: 16`, head dims 128,
`linear_conv_kernel_dim: 4`.

Parameter counts derived from those fields (fused expert layout
`gate_up_proj [512, 2048, 4096]` + `down_proj [512, 1024, 4096]`):

| Group | Params | Sharded by |
|---|---:|---|
| Routed experts `P_e` | **386.55 B** | `X = ep_size × expert_tp_size` |
| Gated-DeltaNet (45) | 5.31 B | FSDP2 only |
| Full attention (15) | 1.57 B | FSDP2, or TP |
| `embed_tokens` + `lm_head` (untied) | 2.03 B | FSDP2 only |
| Vision tower | 0.45 B | FSDP2 only |
| **Shared experts + routers `P_r`** | **0.88 B** | **nothing** — see below |
| Non-expert `P_ne` (all four rows above + norms) | **10.25 B** | |
| **Total** | **396.80 B** | |
| Active per token (top-10 + shared + attention + head) | **17.35 B** | |

**`P_r` is the trap.** `EPQwen3_5MoELayer` owns the router `gate`, the `shared_expert` and
`shared_expert_gate`; the whole layer lands in FSDP2's ignored set (`_fsdp_excluded_params`), TP
never visits it (`apply_tp_to_attention_only`), and EP does not shard it — `replicated_named_params`
grad-syncs it by hand. So **0.88 B stays as a full 8 B/param copy on every rank ≈ 7 GB that no axis
shrinks.** Under `fp32_router` its 0.13 B of router weights are fp32 at 16 B/param, +1 GB.

## Per-rank memory model

`P_f = P_ne − P_r = 9.37 B` is the FSDP-managed non-expert share, `F` the FSDP shard width for it.
`F` is **not** `data_parallel_size`. It defaults to every rank of the job (`F = stage_world_size`), so
`ep4 + etp8` on 4 nodes still shards non-expert params 32-way even at `DP = 4`; ETP partners hold
identical replicas of them.

Two topologies narrow `F`. Under TP the `(dp, tp)` mesh drops it to `DP`. Under
`EPConfig.is_deferred_dp` (multi-group EP across domains, the `× N replicas` rows below) FSDP shards
over the **EP group**, so `F = ep_group_size` and the cross-replica average moves to the
post-backward sweep ([Multi-Node](https://whitecircle.com/halo/docs/agent-docs/parallelism/multi-node.md#deferred-cross-replica-sync)).

| Term | Bytes per rank | 397B value |
|---|---|---|
| Routed experts | `8·P_e/X` | 193 / 97 / 48 GB at `X` = 16 / 32 / 64 |
| Replicated router + shared expert | `8·P_r`, plus `8·P_router` under `fp32_router` | **7 GB, always** (8 GB with `fp32_router`) |
| FSDP non-expert, ZeRO-2 (default) | `2·P_f + 8·P_f/F` | 19 GB resident + 5 / 2 / 1 GB at `F` = 16 / 32 / 64 |
| Activations, GC on | `(2·L·H + 6·k·I_moe)·b·S` | **4.5 GB** at `b1 s8192`, **18.1 GB** at `b1 s32768` |
| GC replay cache (EP) | `L_moe·b·S·(r·(2·H_pad + 12·k) + 2·H_pad)` | **35 / 40 / 42 GB** at `b1 s8192`, `ep` = 16 / 32 / 64; 4× at `s32768` |
| Load-time peak (separate) | `2·(P_ne + P_e/ep_size)` | see the `Load` column |

`8 B/param` = 2 weights + 2 grads + 4 optimizer state; `AdamWBF16` (`halo/optimizers/adamw_bf16.py`)
allocates `exp_avg` and `exp_avg_sq` with `zeros_like(p)`, so both are bf16 and there is no master
copy and no error-feedback buffer. `fp32_grad_reduce` upcasts only inside the reduce — no storage.
The stock AdamW is refused under EP unless `fp32_non_ep_params` is set ([why](https://whitecircle.com/halo/docs/agent-docs/optimization/bf16-optimizer.md#usage)),
and fp32 expert masters (`fp32_experts`) would take the expert term to 16 B/param, where no cell below
survives.

The activation term is one bf16 layer input per layer (`2·L·H·b·S`) plus the recomputed layer's
expert intermediates (`6·k·I_moe·b·S` — gate_up, SiLU product, down input at top-`k` = 10). The
45 linear layers additionally hold an fp32 recurrent state of
`64 × 128 × 128 × 4 B ≈ 4 MB` per sequence per layer (≈190 MB at `b=1`).

The GC replay cache is what lets the recompute skip the all-to-all
([DeepEP → How the toolkit drives it](https://whitecircle.com/halo/docs/agent-docs/infrastructure/deepep.md#how-the-toolkit-drives-it)): each
MoE layer keeps its first pass's received tokens (`[R, H_pad]` bf16), their top-`k` expert ids (int64)
and weights (fp32), and its combined output (`[b·S, H_pad]` bf16), and every layer's cache is alive
when the backward starts. DeepEP sends a token once per destination rank, so `R = r·b·S` with `r` the
distinct ranks a token's top-`k` experts land on: `ep·(1 − C(E − E/ep, k) / C(E, k))` under uniform
routing, 7.7 / 8.8 / 9.4 at `ep` = 16 / 32 / 64. `H_pad` is the hidden rounded up to 256 on the
elastic backend and the hidden itself on legacy (both 4096 here). Under EP+ETP `r` counts the
`ep_size`-wide dispatch group; under PP each in-flight microbatch holds its own. At this model it is
8–9× the activation term, and it decides the 2-node cells.

**ZeRO-3 would remove the 19 GB resident term but is rejected** under EP and under TP+DP
(`_reject_fsdp_settings`) — every layout on this page is EP, so
`fsdp_reshard_after_forward: false` is mandatory, not a choice.

**Budget.** A B300 reports 288 GB (275,040 MiB), and reserved memory is what counts against it. Reserved
runs above allocated: [Nemotron-H's 32,000-token recipe](https://whitecircle.com/halo/docs/agent-docs/models/nemotron-h.md#memory) allocates 227–235 GB and
reserves 268–283 GB, 99.4% of the device on its top rank. These plans count allocated memory, so cells are ⚠ between
230 and 260 GB, ✗ above.

**The DeepEP arena is a separate term**, paid once for the model rather than once per layer. It is
receive-side and linear in the dispatch-group width: at this model's `hidden=4096`, `top_k=10` an
`ep64` group at 8192 tokens per rank costs **4.06 GiB** on top of the columns below. Formula and
growth law: [DeepEP → Buffer sizing](https://whitecircle.com/halo/docs/agent-docs/infrastructure/deepep.md#buffer-sizing). A build or grow takes
1.5x the capacity needed, so the arena can sit up to 1.5x above that figure.

It does not shape the choice of `ep_size` at any cell here; a wide dispatch group keeps its
expert-memory saving instead of spending it back. The one exception is `HALO_EP_CAPACITY_DEDUP=0`,
which gives every layer a private arena and multiplies that figure by the 60 MoE layers to
243.8 GiB, past the card.

When the arena does not fit, the failure is a CUDA OOM inside `ElasticBuffer` at the **first MoE
forward**, after the whole model has loaded, with nothing in the message naming EP capacity.

## What loads at all

Only three loaders shard at load time: `WeightAction.EXPERT_SHARD`
(`halo/model_loading/lazy_safetensors/`, planned by `WeightPlanner` in
`halo/model_loading/parallel/lazy_plan.py`), the PP stage loader (its stage's layers, plus the EP
slice under PP+EP) and HF `tp_plan="auto"` for **dense** models.

Every other branch of `_dispatch_model_loading` materializes all 396.8 B on one GPU: **794 GB, never
starts**. That kills plain FSDP2, pure TP (MoE TP has no shard-aware load), pure ETP (`ep_size == 1`
sets `experts_per_rank = num_experts`, so every rank reads every expert) and pure CP before any
memory argument. A dense model meets the same ceiling: plain FSDP2/DDP loads it whole onto every GPU,
and TP is the route past one GPU ([Data Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/data-parallelism.md#fsdp2-strategy-by-mode)).

The EP lazy loader slices by **`ep_size`, not `X`** — ETP partners share `dispatch_ep_rank` and load
the same experts at full intermediate width. The intermediate dim is split later, at EP patching
(`_init_fused_glu_params` / `_store_separate_glu_params`,
`halo/distributed/expert_parallel/moe_layer/compute.py`), which also holds one layer's transposed full-width
copy alongside the new shard. So `ep2 + etp8` has the same steady state as `ep16` and a **407 GB**
load peak.

## Scenario matrix

`X = ep_size × expert_tp_size`. `Steady` = experts + replicated + FSDP non-expert + activations +
GC replay cache at `b1`. `Load` = the separate per-rank peak while weights stream in. All EP cells need
`512 % ep_size == 0`, rejected by `reject_unsupported_model` off `config.json`
at the top of the model load — after the checkpoint download, before the process groups, the meta
shell or any weight read.

Read the `Layout` column for the fabric too: on an 8-GPU domain every **`global`** row above one node
puts its dispatch and combine on RDMA, 240 times per microbatch; every **`node`** row keeps them on
NVLink.

| Nodes | Layout | X | DP | Experts | Non-exp | Steady 8k | Steady 32k | Load | Verdict |
|---|---|---:|---:|---:|---:|---:|---:|---:|---|
| **1** (8) | `ep8` node | 8 | 8 | 386 | 35 | **>420** | — | 117 | ✗ experts alone are 1.5× HBM |
| 1 | `ep4`+`etp2` · `ep2`+`etp4` | 8 | 4 · 2 | 386 | 35 | **>420** | — | 214 · 407 | ✗ same expert share — `X` cannot exceed the 8 GPUs |
| **2** (16) | **`ep16` global** | 16 | 16 | 193 | 31 | **263** | 383 | 69 | ✗ the replay cache takes full fine-tuning past the budget; frozen layers or `s4096` (244, ⚠) fit |
| 2 | `ep16` global + `tp8` | 16 | 2 | 193 | 55 | **288** | 407 | 69 | ✗ TP collapses `F` to 2 |
| 2 | `ep2` + `etp8` global | 16 | 2 | 193 | 31 | 240 | 291 | **407** | ✗ load peak |
| 2 | `ep8` node · `ep4` node · `ep2` node | ≤8 | 16 | 386 | 35–63 | **>420** | — | 117–407 | ✗ experts alone exceed HBM |
| 2 | `pp2` × `ep8` node | 8 per stage | 8 | 206 | 27 | 268 | 373 | 66 | ✗ stage 0 holds 32 of the 60 layers and two microbatches' replay caches ([PP cells](#pipeline-parallel-cells)) |
| **3** (24) | `ep3/6/12/24` global | 3–24 | 24 | — | — | — | — | — | ✗ `512 % ep_size ≠ 0` at load ([The 3-node case](#the-3-node-case)) |
| 3 | `ep2/4/8` node | ≤8 | 24 | 386 | 35–63 | **>420** | — | 117–407 | ✗ experts alone exceed HBM |
| 3 | `ep3` + `etp8` global | 24 | 3 | 129 | 29 | 178 | 240 | **278** | ✗ `512 % 3 ≠ 0`, and the `ep_size`-governed load peak would not fit either |
| 3 | `ep2/4/8/16` global | — | — | — | — | — | — | — | ✗ rejected at config (`_reject_ep_group`) |
| 3 | `pp3` × `ep8` node | 8 per stage | 8 | 155 | 22 | 216 | 334 | 51 | ✓ at 8k, the one legal 3-node layout, at the [PP costs](#pipeline-parallel-cells); ✗ at 32k |
| **4** (32) | **`ep32` global** | 32 | 32 | 97 | 28 | **169** | 303 | 45 | ✓ |
| 4 | `ep4` + `etp8` global | 32 | 4 | 97 | 28 | 149 | 221 | **214** | ⚠ load peak, experimental axis |
| 4 | `ep32` global + `tp8` | 32 | 4 | 97 | 39 | 180 | 314 | 45 | ✓ but strictly worse than `ep32` |
| 4 | `ep16` global × 2 replicas | 16 | 32 | 193 | 31 | 263 | 383 | 69 | ✗ as `ep16` at 2 nodes; both groups still span all 4 domains (4 contiguous ranks per domain — column-block layout), so dispatch stays on RDMA. The win is the narrower 16-rank dispatch group, not domain locality |
| 4 | `pp4` × `ep8` node | 8 per stage | 8 | 103 | 16 | 154 | 258 | 36 | ✓ EP on NVLink and no 8192 tokens/rank cap, at the [PP costs](#pipeline-parallel-cells); ⚠ at 32k |
| **8** (64) | **`ep64` global** | 64 | 64 | 48 | 27 | **122** | 263 | 33 | ✓ memory to spare; tokens/rank still capped at 8192 |
| 8 | `ep8` + `etp8` global | 64 | 8 | 48 | 27 | 108 | 206 | 117 | ✓ EFA-latency pick, experimental axis |
| 8 | `ep32` global × 2 replicas | 32 | 64 | 97 | 28 | 169 | 303 | 45 | ✓ narrower EP group |
| 8 | `ep16` global × 4 replicas | 16 | 64 | 193 | 31 | 263 | 383 | 69 | ✗ as `ep16` at 2 nodes |
| any | plain FSDP2 · pure TP · pure ETP · pure CP | — | — | — | — | — | — | **794** | ✗ no shard-aware load |
| any | any EP + CP | — | — | — | — | — | — | — | ✗ `reject_unsupported_ulysses_model` |

**One node cannot train this model, in any shape.** Routed experts are only ever divided by `X`,
which on 8 GPUs is at most 8, so every legal single-node shape puts the same 386 GB of expert state
on each GPU against a B300's 288 GB.

Two nodes with `ep16 --ep_scope=global` is the floor, where full fine-tuning fits only with frozen
layers or at `s4096`; four is the first comfortable size. Freezing
the experts is the single-node escape: LoRA on EP experts, or `unfreeze_layers_patterns`, drops them
from 8 to 2 bytes/param. That is arithmetic, not a validated cell.

**`Steady 32k` is not reachable on the `global` rows.** Cross-node EP dispatches
`per_device_train_batch_size × max_length` tokens per rank in one MoE forward (under PP, one microbatch's rows × `max_length`).
`reject_unsupported_model` refuses a declared budget above
`HALO_DEEPEP_GIN_MAX_TOKENS_PER_RANK` (8192) before any weight is read, and the dispatcher re-checks at
buffer sizing; above it a proxy-GIN dispatch wedges in transit instead of erroring.

So every `ep_scope=global` row is capped at 8192 tokens/rank whatever its memory column says. The
`node`-scope rows have no such cap, though intra-node ep8 at ≥~64k tokens/rank needs `ep_buffer_backend: legacy`
([DeepEP → Transport backend](https://whitecircle.com/halo/docs/agent-docs/infrastructure/deepep.md#transport-backend)).

**TP is a net loss on this model.** `apply_tp_to_attention_only` shards only q/k/v/o of the 15
full-attention layers: 1.57 B of 10.25 B. It leaves the 45 gated-DeltaNet layers untouched, and
leaves embeddings and `lm_head` replicated by design. `_find_attention` probes
`self_attn`/`attention`/`attn`, and the linear block is `linear_attn`, so the loop skips those layers
and warns which ones it left replicated.

What TP does do is divide `data_parallel_size`, which divides `F`, so it *raises* the non-expert
term.

**CP is rejected outright.** `reject_unsupported_ulysses_model`
(`halo/distributed/context_parallel/model_rules.py`) raises on `Qwen3_5MoeGatedDeltaNet` and, on a meta
shell, on any `linear_attention` entry in `layer_types`: `causal_conv1d_fn` needs a cross-rank halo
and `chunk_gated_delta_rule` is a recurrent scan. Long context comes from the activation headroom at
4–8 nodes, not from CP.

## The 3-node case

24 GPUs is three NVLink domains, and nothing in the code requires a power of two — `world_size %
nvlink_domain_size == 0` passes and `ParallelismConfig` validates world 24 exactly as it does world
16. The model is what fails, in two independent ways.

**Global-scope EP on 24 must be a multiple of 3.** `CrossNodeLayout` (built by
`_reject_ep_group`) requires `ep_group_size % num_domains == 0` with `num_domains = 3`, and
`nvlink_domain_size % members_per_domain == 0`, leaving exactly `ep_size ∈ {3, 6, 12, 24}`. `512` is
`2⁹`, so **every one of them fails `512 % ep_size == 0`**.

`reject_unsupported_model` raises that off `config.json`, at the top of the model load and
before any process group. `_reject_ep_group` refuses the rest at global scope in order:
`ep_size=16` first, on `reject_cross_node_ep_group` (`24 % 16 ≠ 0`), then `ep_size ∈ {2, 4, 8}` on
`CrossNodeLayout`, because they cannot tile 3 domains equally.

The single EP+ETP shape at world 24 is `ep_size=3, expert_tp_size=8`, which dies on the same
divisor.

**Node-scope EP caps `X` at 8.** `_reject_ep_group` requires `nvlink_domain_size % ep_group_size ==
0`, so `ep_size ∈ {2, 4, 8}` — and `8·P_e/8 = 386 GB` of experts per rank before anything else. The
same ceiling makes node-scope EP useless at 2, 4 and 8 nodes too.

**No EP layout survives without PP; at 24 GPUs only `pp3 × ep8` fits outside experimental PP+ETP.** Each stage is one
node holding 24, 20 or 16 of the 60 layers, so `ep8` node-local divides the 512 experts and stage 0
carries 155 GB of them ([Pipeline-parallel cells](#pipeline-parallel-cells)). `pp3 × etp8` passes the
config gate with the same expert share per rank, but PP+ETP is verified on tiny models only; `pp3 × ep2`
passes and does not fit. Without PP, use 2 nodes
(`ep16` global) or 4 (`ep32` global).

## Pipeline-parallel cells

`Qwen3_5MoeTextModel` is [PP-registered](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md#model-support), so whole-node stages
with `ep8` inside each keep every dispatch on NVLink; only the boundary activations cross RDMA. The
default split is head-weighted on the 4-layer `layer_types` period — `[32, 28]` at `pp2`,
`[24, 20, 16]` at `pp3`, `[16, 16, 16, 12]` at `pp4` — so stage 0 is the largest, and its matrix row
changes in three ways:

- the experts and the replicated router and shared expert hold only the stage's layers:
  `8·P_e·L_stage/60/ep_size`;
- the FSDP term adds the unsharded stage gradient the schedule holds across the microbatch loop:
  `4·P_f,stage + 8·P_f,stage/F`, with `F` the stage's 8 ranks;
- under 1F1B the first stage keeps up to `pp_size` microbatches' layer inputs and GC replay caches in
  flight.

What PP buys here: the 8192 tokens/rank cross-node dispatch ceiling does not apply (node-local `ep8`
still needs `ep_buffer_backend: legacy` past ~64k tokens per rank), and 3 nodes become usable. What it
costs, and why an EP layout stays the recommendation wherever one fits:

- **No router balancing.** The split severs the aux-loss path and Qwen3.5 has no checkpoint slot for a
  bias, so the run takes `moe_balancing: none`: 512 experts at top-10 train unbalanced
  ([PP + EP](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md#pp--ep-moe)).
- **The bubble.** `9 × (pp_size − 1)` microbatches hold it at 10%, and `per_device_train_batch_size`
  must be a multiple of the microbatch count ([Configuration](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md#configuration)).
- **The PP refusals.** `save_sharded_ep`, PEFT and expert LoRA, and layer-indexed
  `unfreeze_layers_patterns` are refused, and `fp32_grad_reduce` doubles the held gradient
  ([Rejected combinations](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md#rejected-combinations)).

## Recommended cell

On an 8-GPU domain every EP-only layout puts the dispatch on RDMA — pick the widest EP group that
loads, and narrow the dispatch group (EP+ETP) when the fabric is latency-bound. PP keeps it on NVLink
at the [costs above](#pipeline-parallel-cells).

| Nodes | Cell | Crosses on | Steady `b1 s8192` | Why |
|---|---|---|---:|---|
| 2 | `ep16 --ep_scope=global` + `unfreeze_layers_patterns` | RDMA all-to-all | ≈155 GB partial (263 GB full FT, ✗) | the only layout that loads; full fine-tuning fits only at `s4096`, no headroom for 32k or `b>1` |
| 3 | `pp3` × `ep8` node, `moe_balancing: none` | boundary activations only | 216 GB | the one layout that fits outside experimental PP+ETP ([The 3-node case](#the-3-node-case)); trains unbalanced |
| 4 | **`ep32 --ep_scope=global`** | RDMA all-to-all | 169 GB | first comfortable point on memory — but the binding limit is the 8192 tokens/rank cross-node dispatch ceiling, not the 90 GB spare |
| 8 | **`ep64 --ep_scope=global`** | RDMA all-to-all | 122 GB | memory headroom to spare, same 8192 tokens/rank ceiling on `b × max_length`; `ep8 + etp8` if the fabric is EFA |

**On NVL72, declare a 64-GPU domain, not 72.** `_reject_ep_group` requires
`nvlink_domain_size % ep_group_size == 0`, and `72 % 64 = 8`.

At `NVLINK_DOMAIN_SIZE=72` the only EP widths that survive validation are `ep2` and `ep72`:
`ep4`/`ep8` fall to the racy-EP guard (scope-blind on a single domain), `ep16`/`ep32`/`ep64` to the
divisibility rule. `512 % 72 ≠ 0` then kills `ep72` at model load. **A 512-expert model has no legal
EP shape on a full 72-wide rack.**

The working recipe is `NVLINK_DOMAIN_SIZE=64` over 16 of the rack's 18 compute trays, with
`ep_size=64, ep_scope=node`: one dispatch group per rack, 8 experts/rank, ~48 GB of expert state,
`requires_rdma` false. It costs 8 of 72 GPUs, 11% idle. The general form
— the domain is the tiling unit, so the legal `ep_group_size` values are its divisors — is on
[Multi-Node](https://whitecircle.com/halo/docs/agent-docs/parallelism/multi-node.md#gb200gb300-nvl72-multi-node-nvlink).

Scaling out, `world=512` over 8 racks at domain 64 accepts every node-scope `ep_size` dividing 64
(the racy guard is inert across multiple domains); `ep64` gives 8 EP groups, one per rack, averaged by
the deferred cross-replica sweep. Like the rest of this page, that is rank math and arithmetic: a plan,
not a measurement.

**Frozen layers are the lever at 2 nodes.** `unfreeze_layers_patterns` leaves untouched parameters at
2 B/param — weights only, no gradient and no optimizer state. Unfreezing the top 20 of 60 layers plus
routers and `lm_head` takes the expert term from `8·P_e/X` to `4·P_e/X` — 193 → 97 GB at `X=16`.

**When the fabric is the bottleneck, narrow the dispatch group.**
Cross-node EP over EFA is proxy-Gin latency-bound
([DeepEP → AWS EFA](https://whitecircle.com/halo/docs/agent-docs/infrastructure/deepep.md#expert-parallelism-over-aws-efa)).

`ep64` gives each rank 56 cross-node dispatch partners; `ep8 + expert_tp_size=8` gives the same `X=64` with **7**,
because `_reject_expert_tp` forces exactly one ETP group per NVLink domain
(`expert_tp_size == members_per_domain`, `ep_size == num_domains`). EP+ETP is an experimental axis —
its expert-TP reduce runs in token space outside the DeepEP dispatch→combine span.

On InfiniBand with IBGDA the latency wall does not exist; use plain `ep64` — but size the microbatch against the arena
term above, which `ep8 + etp8` does not pay (its dispatch group is 8 wide, so its arena is an eighth
of `ep64`'s at the same tokens/rank).

## Launch recipes

One `torchrun` per node. No 397B config ships — derive one from the 122B sibling per
[YAML knobs](#yaml-knobs).

```bash
torchrun --nnodes=$NNODES --node_rank=$NODE_RANK --nproc_per_node=8 \
  --rdzv_backend=c10d --rdzv_endpoint=$MASTER_ADDR:29500 \
  scripts/training/sft.py <your-397b-config>.yaml <layout flags>
```

| Nodes | Layout flags |
|---|---|
| 2 | `--expert_parallel_size=16 --ep_scope=global` |
| 3 | `--pipeline_parallel_size=3 --expert_parallel_size=8 --ep_scope=node`, with the PP keys of [Pipeline-parallel cells](#pipeline-parallel-cells) (`moe_balancing: none`, `max_length`, `pipeline_microbatches`) |
| 4 | `--expert_parallel_size=32 --ep_scope=global` |
| 8 | `--expert_parallel_size=64 --ep_scope=global` |
| 8, EFA | `--expert_parallel_size=8 --expert_tensor_parallel_size=8 --ep_scope=global` |

### Environment

The fabric env (EFA vs InfiniBand, proxy GIN) is the standard one from
[Multi-Node → RDMA fabrics](https://whitecircle.com/halo/docs/agent-docs/parallelism/multi-node.md#rdma-fabrics), the rest from
[Launch Recipes](https://whitecircle.com/halo/docs/agent-docs/parallelism/launch-recipes.md#environment-variables). What this model changes:

```bash
export DIST_NCCL_TIMEOUT_MINUTES=90    # default 30; a 397B gathered save takes minutes
export DIST_STORE_TIMEOUT_HOURS=8      # default 4; a single-rank 794 GB download can outlast it
# GB200/GB300 NVL72 only — 64, not 72: ep_group_size must DIVIDE the domain, and 512 % 72 != 0.
# Launch on 16 of the rack's 18 compute trays; the other 8 GPUs sit idle.
export NVLINK_DOMAIN_SIZE=64
```

`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` composes with the `ElasticBuffer` and answers
allocator fragmentation under variable-shape packing. Under EP it costs throughput, so set it only
when the run does not fit without it
([measured](https://whitecircle.com/halo/docs/agent-docs/optimization/throughput-benchmarks.md#allocator-expandable-segments)).

### YAML knobs

Base the config on `examples/sft/qwen3_5/qwen3.5-122b-a10b-ep.yaml` — the closest shipped shape.
Values that differ from it or are load-bearing at 397B:

| Knob | Value | Why |
|---|---|---|
| `expert_parallel_size` / `ep_scope` | per the table above / `global` | node-scope EP caps `X` at 8 and cannot hold `P_e` |
| `attn_implementation` | `flash_attention_2` | unset, these packed rows load FA2 on Blackwell too (unpacked rows take FA4, [Qwen3.5 → Configs](https://whitecircle.com/halo/docs/agent-docs/models/qwen3_5.md#configs)) and Hopper picks FA3; the pin keeps Hopper on FA2 as well. An explicit FA2 is kept as-is |
| `packing` / `padding_free` | `true` / `false` | fixed-length rows, as the shipped recipe |
| `fsdp_reshard_after_forward` | `false` | ZeRO-3 is rejected under EP |
| `gradient_checkpointing` | `true` | the activation column assumes it |
| `bf16` + `optim` | as shipped | AdamWBF16 auto-enables whenever `bf16: true` and `optim` is `adamw_torch_fused`/`adamw_torch` outside accelerate-managed DDP — both are already the defaults, so neither key has to be written to get the 8 B/param cost. `optim: muon` or `flash_adamw` opts out |
| `fp32_experts` / `fp32_non_ep_params` | **omit** | the 35B configs set them; fp32 masters would take the expert term to 16 B/param and OOM every cell |
| `text_only_model` | `true` | loads `Qwen3_5MoeForCausalLM`, which drops the vision tower and MTP tail from the build and the export ([Qwen3.5 → EP layer](https://whitecircle.com/halo/docs/agent-docs/models/qwen3_5.md#ep-layer)). `aux_loss` trains on either class |
| `moe_balancing` | `aux_loss` | trains and exports by construction. Strict `bias_update` raises: Qwen3.5 has no checkpoint slot for a routing bias ([Callbacks](https://whitecircle.com/halo/docs/agent-docs/training-methods/callbacks.md#moe-balancing-modes)) |
| `router_balancing_rate` | well below `1e-3`, only under `bias_update_transient` (the 122B recipe's mode) | refused under an explicit `aux_loss` or `none`. The sign step lands on softmax probabilities, whose uniform scale is `1/512` here — the default γ is ~50% of it |
| `fp32_router` | `true` | +1 GB per rank (the 0.13 B router weights at 16 B/param, not in the matrix columns), and 512 experts at top-10 is where bf16 routing logits start to tie |
| `fp32_grad_reduce` | `true` | bf16 sums lose precision at 16–64 ranks; no storage cost outside PP, where it doubles the held stage gradient |
| `use_grouped_gemm` / `use_liger_kernel` | `true` / `true` (the defaults) | the fused `[512, 2048, 4096]` layout is what `grouped_mm` wants on SM100 |
| `liger_kernel_config.fused_linear_cross_entropy` | `true` | opt-in: this family defaults to Liger's plain cross-entropy. The fused loss never materializes `[b, S, 248320]` — 4 GB per copy at `S=8192` |
| `max_concurrent_loading` | leave unset | node-local wave gate; unset adapts to the node (`min(4, max(1, local_world_size // 2))` — 4 on an 8-GPU node, 2 on a 4-GPU tray), and any explicit value is used verbatim. The lazy EP path bypasses it, so at load it bounds only the EP fallback; a resume's optimizer-shard reads take it too |
| `save_sharded_ep` | `true` on the pure-EP cells (`ep16`/`ep32`/`ep64`) | every rank writes its own shard instead of funneling 794 GB through one. Requires a single EP group spanning all ranks (`ep_group_size == world_size` — exactly these cells) and a shared output filesystem; **rejected under ETP**, so the `ep8+etp8` cell takes the layer-streaming gathered save instead. Merge before serving ([Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#expert-parallelism-ep-eptp-epcp)) |
| `per_device_train_batch_size` | `1` | the activation column is `b=1`; raise only at 8 nodes and only after step 1's peak is known |
| `max_length` | `8192` on every `ep_scope=global` cell | doubling it doubles the activation column — but on cross-node EP the hard limit is the 8192 tokens/rank dispatch ceiling above, not memory. Raise the ceiling only after validating a larger dispatch end-to-end on your fabric ([DeepEP → AWS EFA](https://whitecircle.com/halo/docs/agent-docs/infrastructure/deepep.md#expert-parallelism-over-aws-efa)) |
| `unfreeze_layers_patterns` | required at 2 nodes | see [Recommended cell](#recommended-cell) |
| `save_total_limit: 1` + `save_only_model: true` | as shipped | a bf16 checkpoint is ~800 GB per save |

Pre-shard the corpus with `scripts/before_training/prepare_dataset.py --num_shards N`,
`N >= data_parallel_size` — at 64 ranks the on-the-fly path materializes the dataset per rank.

## Other 300B-class checkpoints

The same arithmetic (8 B/param on the trained experts, `X = ep_size × expert_tp_size`, the
non-expert weights gathered whole under ZeRO-2) for the three other checkpoints in the 200–320B
class, on 8 × B300 ([Budget](#per-rank-memory-model)). Every figure is a plan, not
a measurement, except the three GLM-5.3-Flash rows marked measured ([rates](https://whitecircle.com/halo/docs/agent-docs/models/glm5-next.md#throughput)).

| Checkpoint | Routed experts | Cell | Expert leg / rank | Static / rank | What binds |
|---|---:|---|---:|---:|---|
| **GLM-5.3-Flash** (321B; 42 MoE layers × 288 × `M=2048, H=4096`) | 304 B | 2 × 8, `ep16 --ep_scope=global` | 19.0 B → 152 GB | Measured, GC on: 68.8 GiB after load, 212.7–214.1 GiB peak, 1,504 tok/s/GPU | the 8192 tokens/rank cross-node dispatch ceiling. Needs the bf16 conversion (~650 GB) |
| GLM-5.3-Flash | 304 B | 2 × 8, `pp2 × ep8` | 19.0 B → 152 GB | Measured, GC on: 57.1 / 54.0 GiB after load, 218.0–218.7 / 197.5–197.9 GiB peak, 2,423 tok/s/GPU | padded rows only, as packing is refused; 1.61× `ep16` |
| GLM-5.3-Flash | 304 B | 2 × 8, `ep8` with DP | 38.0 B → 304 GB | Measured: out of memory in the first optimizer step, 243–250 GiB allocated of 267.7 | optimizer state on 40.2B trainable per rank: weights and gradients fit, AdamWBF16's moments do not |
| GLM-5.3-Flash on EFA | 304 B | 2 × 8, `ep2 --expert_tensor_parallel_size=8 --ep_scope=global` | same `X=16` | ~200 GB by arithmetic | one cross-node dispatch partner per rank instead of 8 (one ETP group per domain, [EP+ETP](https://whitecircle.com/halo/docs/agent-docs/parallelism/expert-tensor-parallelism.md#process-groups-epetp-combo)); experimental axis |
| **Step-3.7-Flash** (198B; 42 MoE layers × 288 × `M=1280, H=4096`) | 190 B | 1 × 8, `ep8` | 23.8 B → 190 GB | ~213 GB (+15 GB gathered non-expert, +6 GB shards) | ~75 GB left on the 288 GB device for activations and the allocator's reserve at `max_length: 8192` with checkpointing — marginal, unmeasured |
| Step-3.7-Flash | 190 B | 2 × 8, `ep16 --ep_scope=global` | 11.9 B → 95 GB | ~115 GB | the 8192 tokens/rank dispatch ceiling |
| **DeepSeek-V4-Flash** (284B; 43 MoE layers × 256 × `M=2048, H=4096`, 3 of them hash-routed) | 277 B | 2 × 8, `ep16 --ep_scope=global` | 17.3 B → 138 GB | ~155 GB | the 8192 tokens/rank dispatch ceiling. Needs the bf16 conversion (~580 GB output) |

The `ep16` rows all sit on cross-node EP and inherit its `per_device_train_batch_size × max_length
≤ 8192` contract.

## Pre-flight checklist

**Before the launch**

- `512 % ep_size == 0`, `expert_tp_size | moe_intermediate_size`, TP head divisibility and the
  declared tokens/rank budget are checked off `config.json` before any weight is read
  (`reject_unsupported_model`, at the top of `load_distributed_model`, after
  the checkpoint download); the Ulysses and PP structure refusals run on a meta-device shell, also
  before any weight is read.
- Scratch: a single checkpoint is ~800 GB with `save_only_model: true` (as shipped), ~2.4 TB with
  optimizer shards (AdamWBF16 adds 4 B/param).

    Rotation deletes the previous checkpoint only after the new one completes, so
    `save_total_limit: 1` still transiently holds **both**: budget ~1.6 TB free per save, ~4.8 TB with
    optimizer shards. The trainer checks the peak against `output_dir`'s volume at construction
    ([Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#warm-restart-vs-exact-resume-torchrun)). Confirm the volume with `findmnt`/`df -h` before pointing `HF_HOME`,
    `HF_DATASETS_CACHE`, `TMPDIR` and `HALO_DATA_ROOT` at it.

    On the `save_sharded_ep` cells budget for the merge too: `merge_ep_shards.py` streams (peak host
    RAM ≈ one merged MoE layer, not the ~800 GB artifact), and `--delete_input_shards` frees the
    per-rank shards only once the merged checkpoint is complete, so the volume briefly holds both.

- Host RAM at load: the EP lazy path reads safetensors straight to GPU and needs almost none. Every
  slice is a view over the mmapped shard, so anonymous RSS stays near the process baseline (about
  one tensor at a time) and `max_concurrent_loading` never applies.

    What `free` shows is page cache the size of the checkpoint (~800 GB here; 398 / 628 / 582 GB for
    the Step-3.7 / GLM-5.3 / DeepSeek-V4 Flash artifacts of
    [the 300B table](#other-300b-class-checkpoints)), shared by the node's ranks and reclaimable. Each
    rank's file-backed RSS reaches its dense + own expert slices while its shard handles are open. Not
    a leak.

    The gate+up fan-in's 2× transient lives on the GPU (~1 GB per layer). If `ep_lazy_loading` is off
    or the checkpoint layout is lazy-incompatible, the fallback stages the full model in CPU RAM per
    concurrent loader; set `max_concurrent_loading: 1` when `free -g` is tight. `fp32_experts` doubles
    the expert bank it stages ([EP → Model loading](https://whitecircle.com/halo/docs/agent-docs/parallelism/expert-parallelism.md#model-loading)).

- Host RAM at save and resume: a run that keeps optimizer state (`save_only_model: false`) passes
  each rank's optimizer shard through host RAM: ~28 GB/rank at `ep64`, ~102 GB/rank at `ep16`. At `ep16`
  an 8-GPU node holds ~410 GB during a save (the default `max_concurrent_loading` admits 4 ranks at a
  time) and ~820 GB during a resume, where every rank keeps its copy until the collective apply.

    Every recommended cell runs **one** EP group, so each rank's expert state is unique; a job wide enough to
    run several EP groups (they are DP replicas) writes that half once per replica group instead
    ([Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#warm-restart-vs-exact-resume-torchrun)).

    A `save_sharded_ep` save, with or without optimizer state, also holds each rank's whole expert slice in
    host RAM at once, unthrottled: ~48 GB/rank and ~386 GB per 8-GPU node at `ep16`, halving at each
    doubling of `ep_size`. Rank 0 holds ~20 GB more for the replicated half.

- Hardware gates on every node: `scripts/profiling/nvlink_health.py` (non-zero exit = a degraded
  lane that silently caps or hangs the EP all-to-all), `nvidia-smi` (8 GPUs, near-zero used memory,
  no ECC/Xid), and the fabric check for your interconnect
  ([RDMA fabrics](https://whitecircle.com/halo/docs/agent-docs/parallelism/multi-node.md#rdma-fabrics)).

- Dataset shards: `num_shards >= data_parallel_size`, which is `stage_world_size / max(tp, cp, etp)`,
  not the GPU count.

- Dry-run with the smaller Qwen3.5 MoE sibling (`Qwen/Qwen3.5-35B-A3B`) before the real weights load. One
  8-GPU host cannot form a recommended cell, since each needs its own node count. Two `torchrun` agents on
  that host ([`--nnodes=2 --nproc_per_node=4`](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md#measured-throughput)) train `pp2` ×
  `ep4` and pass the config gates for `ep8 --ep_scope=global` and `ep2` + `etp4`. For the real cell's
  config verdict, build a `ParallelismConfig` with its `world_size` and `gpus_per_node` on CPU.

**After step 1**

| Check | Expected | If not |
|---|---|---|
| Peak memory (`EfficiencyCallback`) | within ~15% of the matrix cell | recheck `P_e`/`P_ne`; the replicated `P_r` term is the usual surprise |
| `grad_norm` | finite, same order as a 35B-A3B run of the same recipe | see [Debugging](https://whitecircle.com/halo/docs/agent-docs/reference/debugging.md) |
| `moe/*` metrics present and non-zero | `MoEMetricsCallback` logs them under `aux_loss` (`BalancingBiasCallback` under a bias-update mode) | this checkpoint keeps its expert fields on `text_config`, and `resolve_expert_count` / `resolve_router_topk` resolve the text config before reading them. Silent zeros mean the model is not being seen as MoE and `moe_balancing` has resolved to `none` — the router is training unbalanced |
| Packing took the varlen path | no dense-mask warning from `select_data_collator` | a non-varlen `_attn_implementation` pays a dense mask over the flattened batch, and on some families it also stops isolating documents ([Document isolation](https://whitecircle.com/halo/docs/agent-docs/data/collators.md#document-isolation-under-packing)). Pin `flash_attention_2` |
| Expert balance (`MoEMetricsCallback`, or `BalancingBiasCallback` under a bias-update mode) | no expert at 0 tokens across a step | raise `router_aux_loss_coef` under `aux_loss`; under `bias_update_transient` raise `router_balancing_rate` — but see the 512-expert scale caveat in [Callbacks](https://whitecircle.com/halo/docs/agent-docs/training-methods/callbacks.md#balancingbiascallback) before doubling it |
| Achieved TFLOPS | compare against the [MFU registry](https://whitecircle.com/halo/docs/agent-docs/optimization/throughput-benchmarks.md) at 35B-A3B. Its rows are EP-only; tok/s/GPU on an `ep8 + etp8` cell reads lower by construction (eight ETP partners share one batch), so compare TFLOPS, which counts each rank's own compute | a 2×+ shortfall on EFA is the proxy-Gin latency wall: narrow the dispatch group (`ep8 + etp8`) |
| Step time variance | stable after 3 warmup steps | a slow rank is usually a degraded NVLink lane or a straggler NIC |
| First checkpoint | written, and re-loadable with `from_pretrained` — on the `save_sharded_ep` cells only **after** `merge_ep_shards.py`; the artifact itself is per-rank (`format: "ep_sharded"`) and every loader refuses it unmerged. The gathered `ep8+etp8` row loads directly | raise `DIST_NCCL_TIMEOUT_MINUTES` before blaming the fabric |
