Multi-Node Launch Recipes
Runnable launches for multi-node training: torchrun and SLURM commands, EP checkpoint save/load and filesystem setup. For the process-group layout, node-local vs cross-node scope, and the ParallelismConfig API, see Multi-Node Training. For SkyPilot cluster provisioning, see SkyPilot.
These recipes assume N × 8-GPU nodes on an RDMA fabric (InfiniBand/RoCE or AWS EFA — RDMA fabrics). Launch one torchrun per node (it forks the 8 per-GPU ranks); EP all-to-all uses NVLink within a node and RDMA only when ep_scope="global" makes the EP group span nodes. A ready-to-run example covering cross-node EP, node-local EP + DP, EP+TP, and the NVL72 variant ships at examples/sft/gptoss/gptoss-20b-multinode-ep.yaml; the two pipeline-parallel recipes, one stage per node, launch the same way (Pipeline Parallelism → Launching).
torchrun
One torchrun per node, incrementing --node_rank. The config sets the parallelism layout — it ships expert_parallel_size: 16 / ep_scope: global, cross-node EP across both nodes — and every parallelism field overrides on the command line:
- Node-local EP + DP:
--expert_parallel_size=8 --ep_scope=node --use_grouped_gemm=true. The config pins the per-expert loop forep16, and atep8that loop is 28% slower (14,402 against 20,066 tok/s/GPU on one 8× B300 host).
Bash
# Node 0 (master): --node_rank=0 ; Node 1: --node_rank=1
torchrun \
--nnodes=2 \
--node_rank=0 \
--nproc_per_node=8 \
--master_addr=$MASTER_ADDR \
--master_port=$MASTER_PORT \
scripts/training/sft.py \
examples/sft/gptoss/gptoss-20b-multinode-ep.yamlOn one node the launch drops --nnodes, --node_rank and --master_*: torchrun --nproc_per_node=8 scripts/training/sft.py <config> --expert_parallel_size=8 --use_grouped_gemm=true. Each axis is one --*_parallel_size flag, in the YAML or on the CLI, combinable where Supported combinations allows, and each needs torchrun (Two launchers): accelerate launch, detected through ACCELERATE_MIXED_PRECISION or ACCELERATE_USE_FSDP (is_accelerate_launch), raises on any of them. A plain python launch trains on the one GPU it binds however many are visible: init_training_script pins n_gpu to 1, so HF never wraps the model in nn.DataParallel. halo launch is single-node only.
On a multi-socket host, add --numa-binding node to each torchrun: it runs every rank on its GPU's NUMA node, a gain on host-bound TP, CP and small-batch EP runs (NUMA binding).
The log-writing rank (global rank 0 on a shared output filesystem, each node's local rank 0 otherwise) tees its stdout and stderr to <output_dir>/log/run.log, so a detached run keeps its console without a redirect. Redirecting the launch (> "$D/<job>.log" 2>&1, as the detached Docker launch does) also captures the other ranks, such as a crash off rank 0. A log directory or tee that cannot be set up is warned on that rank, and the run continues without the file.
SLURM
Exactly one task per node — srun launches one torchrun per node, which forks the per-GPU ranks. --ntasks-per-node=8 would launch 8 torchruns × 8 = 64 procs/node.
A bare srun --ntasks-per-node=N <script> — one task per GPU, no torchrun — is refused at startup: SLURM declares the world through SLURM_NTASKS but supplies no env:// rendezvous, so MASTER_ADDR/MASTER_PORT are unset and no process group can be built. Use the recipe below, or export both identically on every task.
Ranks must sit gpus_per_node per machine in rank order: every node-local EP/CP/TP/ETP group is a contiguous rank block, so ParallelismConfig rejects a round-robin placement (srun --distribution=cyclic with an exported rendezvous). One torchrun per node, or --distribution=block, satisfies it.
Bash
#!/bin/bash
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-node=8
export MASTER_ADDR=$(scontrol show hostname $SLURM_NODELIST | head -n 1)
export MASTER_PORT=29500
srun bash -c 'torchrun \
--nnodes=$SLURM_NNODES \
--node_rank=$SLURM_NODEID \
--nproc_per_node=$SLURM_GPUS_PER_NODE \
--master_addr=$MASTER_ADDR \
--master_port=$MASTER_PORT \
scripts/training/sft.py \
examples/sft/gptoss/gptoss-20b-multinode-ep.yaml'The single quotes make each task expand its own $SLURM_NODEID. Unquoted, the batch shell expands it once, to 0, so every node claims rank 0 and the rendezvous times out with DistStoreError.
Other schedulers
There is no scheduler-specific integration. Any orchestrator (Kubernetes, Kubeflow PyTorchJob,
Nomad, Ray, plain SSH) launches the same way: one torchrun per node with its --node_rank, all
pointed at one rendezvous endpoint. The toolkit reads rank and topology only from the standard
torchrun and SLURM_* variables, so map your scheduler's node index onto --node_rank — Kubernetes
JOB_COMPLETION_INDEX, SkyPilot SKYPILOT_NODE_RANK, Nomad NOMAD_ALLOC_INDEX within one group (the
shipped specs instead declare one group per rank and set NODE_RANK literally —
Nomad).
Managed cloud recipes: SkyPilot (AWS EFA / Nebius IB) and RunPod (InfiniBand pods).
Environment variables
Bash
export DIST_SHARED_FILESYSTEM=1 # default; "0" for per-node local storage
export DIST_NCCL_TIMEOUT_MINUTES=60 # NCCL watchdog; toolkit default 30 (PyTorch's own is 10)
export DIST_STORE_TIMEOUT_HOURS=4 # default; bounds c10d-store waits (main-first download, load gate, checkpoint write joins)
export NCCL_SOCKET_IFNAME=<your fast NIC> # multi-homed node: `ib0` on IB, the ENA iface on AWS
# The image bakes the IB/RoCE NCCL defaults and CUDA_DEVICE_MAX_CONNECTIONS=1
# (agent-docs/infrastructure/docker.md#baked-in-environment-variables); InfiniBand/RoCE needs no fabric env.
# export NVLINK_DOMAIN_SIZE=72 # GB200/GB300 NVL72 only; leave unset on ≤8-GPU nodes
# export NCCL_DEBUG=INFO # optional; overrides the image's WARNThe per-fabric launch env — AWS EFA's plugin variables, the proxy GIN cross-node EP needs over EFA, a rollout server on another node: Multi-Node → RDMA fabrics.
DIST_NCCL_TIMEOUT_MINUTES is the one NCCL watchdog setting: the process group starts before the
config is parsed, so a training config that sets ddp_timeout away from its default is refused. Set
it alike on every node. init_distributed pins it as the default for every later
dist.new_group(). EP/CP/PP subgroups pass it explicitly (new_group or split_group); the pin is what carries it onto the
DTensor meshes, since init_device_mesh takes no timeout — that is the DP/HSDP/TP axes. It does
not bound the DeepEP dispatch/combine kernels, which carry the ElasticBuffer GPU-side barrier
rather than the PyTorch watchdog.
Inter-node EP runs the DeepEP V2 NCCL Gin (RDMA) backend; intra-node EP runs the non-Gin NVLink
path. The dispatcher sets EP_DISABLE_GIN from the EP topology (honoring an explicit value).
EP_SUPPRESS_NCCL_CHECK=1 is baked into both images as an ENV and must reach the process
environment: DeepEP latches it at import deep_ep, so a Python-level write is too late. Without it
DeepEP's duplicate-NCCL guard flags the image's HPC-X transport plugin as a second NCCL runtime and
fails the import.
Checkpointing with EP
save_sharded_ep picks the save mode, and a sharded save merges before loading:
Bash
python scripts/after_training/merge_ep_shards.py --input_dir checkpoint/ --output_dir merged/The two modes, save-rank memory, and every save_sharded_ep rejection live on Expert Parallelism → Checkpointing; resume semantics are on Checkpoints & Resume.
File system
Set DIST_SHARED_FILESYSTEM=1 (default) for shared storage (NFS, Lustre, GPFS) or 0 for per-node local storage, and split the read and write sides with DIST_INPUT_SHARED_FILESYSTEM / DIST_OUTPUT_SHARED_FILESYSTEM where they differ (per-node input caches beside shared-mount checkpoints on a slow NFS export). What per-node output storage changes for a parallel run: Multi-Node → Shared vs node-local storage; the coordination and the startup probes: Filesystem Handling.
Weights take ~2 bytes/param in bf16 (7B MoE ≈ 14 GB, 20B ≈ 40 GB, 70B ≈ 140 GB), the size of a save_model export; per-rank sharded is roughly full / ep_size. A training checkpoint keeps fp32 masters (fp32_router, fp32_experts, fp32_non_ep_params) at 4 bytes per element and adds the optimizer state (4 bytes/param for AdamWBF16) unless save_only_model: true.
Preflight
EP all-to-all rides NVLink inside a node, so a marginal lane silently caps throughput or hangs a run.
scripts/profiling/nvlink_health.py reads nvidia-smi counters (no GPU allocation, safe beside a live
job), separates hard faults from FEC-absorbed churn, and exits non-zero on an unhealthy or down link;
run it before a long launch:
Bash
python scripts/profiling/nvlink_health.py # summary, exit 1 if unhealthy or downPre-tokenized, per-rank sharded datasets (num_shards >= data_parallel_size):
Data Loading → Pre-processed (sharded) datasets.
Symptoms and fixes for a failed launch (rendezvous, RDMA, config rejections):
Troubleshooting.
Source on GitHubUpdated 18 November 2026