Multi-Node EP Training on RunPod InfiniBand Clusters
A manual runbook for multi-node EP on RunPod pods with InfiniBand; the repo ships no RunPod tooling.
With only EP active, data_parallel_size = world_size — EP is orthogonal to DP
(Stacking the dimensions). Theory:
Multi-Node Parallelism.
Prerequisites
- 2+ RunPod pods with InfiniBand, same data center / IB fabric, 8 GPUs each (H100/H200 SXM).
- A persistent volume mounted at
/workspace(500 GB+) for weights, HF cache, checkpoints. - The training image. Set the pod's Container Image to
public.ecr.aws/whitecircle/halo:hopper(anonymous, no registry credentials), or push your own build to your own registry (Docker Guide, AWS Auth). - HuggingFace token for gated models, optionally a WandB key.
The halo:hopper image ships everything the run needs, including the Mellanox NCCL defaults
(Docker). Override the CMD with sleep infinity — the default CMD prints a
banner and exits, which stops a RunPod pod.
Network setup
Designate one pod as Node 0 and take its IB-routable IP as MASTER_ADDR (not the public SSH IP):
Bash
hostname -I | awk '{print $1}' # IB-routable IP
ibstat # expect State: Active
ib_write_bw -d mlx5_0 # server on Node 0
ib_write_bw -d mlx5_0 <NODE0_IP> # client on Node 1 — expect close to the port's line rate
# (HDR 200 Gb/s ≈ 24 GB/s); an order of magnitude below it
# means the fabric is not what the pod advertisesInstall diagnostics if missing: apt-get install -y infiniband-diags perftest libibverbs-dev rdmacm-utils
— ibstat/perfquery come from infiniband-diags, ib_write_bw from perftest. The training image
installs none of them; it ships the NCCL fabric transports, not the fabric diagnostics.
Set only the environment-specific variables; the image covers the rest:
DIST_NCCL_TIMEOUT_MINUTES is the one watchdog setting (Launch Recipes → Environment variables lists the groups it covers and the DeepEP barrier it does not).
A bare NCCL_TIMEOUT does nothing — PyTorch does not read it.
Per-pod setup
Bash
cd /workspace
git clone https://github.com/whitecircle/halo.git code && cd code
hf auth login --token $HF_TOKEN
wandb login $WANDB_API_KEY # optional
python -c "import torch; print(torch.__version__, torch.version.cuda)"
python -c "import deep_ep; print('DeepEP OK')"
python -c "from flash_attn_interface import flash_attn_func; print('FA3 OK')"
# Optional: pre-fetch the model on every pod (with DIST_SHARED_FILESYSTEM=0 each pod's local rank 0
# fetches it at load otherwise)
export HF_HOME=/workspace/hf_cache
python -c "from huggingface_hub import snapshot_download; snapshot_download('openai/gpt-oss-20b', ignore_patterns=['*/*'])"EP topologies (2 nodes × 8 GPUs)
- Node-local EP: each node runs its own EP group over NVLink; only gradient AllReduce crosses nodes. Simplest to debug.
- Cross-node EP: one EP group spans all 16 GPUs; the expert all-to-all uses the NCCL Gin backend over IB RDMA. Each GPU holds fewer experts → more memory for activations.
- EP+TP: a single global EP group across nodes; the rules: Topology rules.
The flags and their defaults: Configuration reference → ParallelismConfig.
Launch
Run on each pod simultaneously (use tmux). Both pods must use identical --expert_parallel_size,
--ep_scope, --master_addr, --master_port, --nnodes; only --node_rank differs.
Bash
#!/bin/bash
set -e
export DIST_NCCL_TIMEOUT_MINUTES=30
export HF_HOME=/workspace/hf_cache
export DIST_SHARED_FILESYSTEM=0 # RunPod pods have independent storage
NODE_RANK=${1:?Usage: bash launch.sh <node_rank> <master_ip>}
MASTER_ADDR=${2:?Usage: bash launch.sh <node_rank> <master_ip>}
MASTER_PORT=${3:-29500}
cd /workspace/code
torchrun \
--nnodes=2 --node_rank=$NODE_RANK --nproc_per_node=8 \
--master_addr=$MASTER_ADDR --master_port=$MASTER_PORT \
scripts/training/sft.py examples/sft/gptoss/gptoss-20b-multinode-ep.yaml \
--expert_parallel_size=8 --ep_scope=node --use_grouped_gemm=true \
--output_dir=/workspace/checkpoints/run-001Run bash launch.sh 0 <master_ip> on Node 0 and bash launch.sh 1 <master_ip> on Node 1. Swap the
parallelism flags from the topology table; add --resume_from_checkpoint=true to resume.
The YAML sets save_only_model: true (warm restart, smaller checkpoints); save_on_each_node needs
no key, since DIST_SHARED_FILESYSTEM=0 above auto-forces it.
Checkpoints and resume
Formats, save modes, and resume mechanics are owned by Checkpoints & Resume. The RunPod-specific parts:
Gathered (default) is the multi-node choice. It all-gathers expert weights and writes sharded safetensors loadable with
from_pretrained(), and every pod's save rank writes its own copy.--save_sharded_epraises at trainer construction on a multi-node non-shared FS. Per-rank shards are keyed by global rank and scatter across the pods' local disks (the index and non-expert params land only on the rank-0 pod).No single pod holds a mergeable checkpoint, and
scripts/after_training/merge_ep_shards.pytakes one input dir. Where it does run (single node, or a shared FS) it buys write bandwidth, since every rank writes its own slice. The price is a checkpoint that must be merged before resume or serving, and it saves no host memory.Resume needs no rsync. With
DIST_SHARED_FILESYSTEM=0the mixin forcessave_on_each_node=True, so each pod's local rank 0 writes a complete checkpoint to its own disk; re-runtorchrunon every pod with--resume_from_checkpoint=true.Trained EP/CP weights reload with no manual step: the training scripts load the policy from the resumed checkpoint, so leave
model_name_or_pathat the base, which reference and teacher models still resolve (why).WANDB_RUN_IDis hashed fromoutput_dir+ launch timestamp and broadcast from rank 0, so it is consistent across a run's ranks but not across re-runs. Export the sameWANDB_RUN_IDon every pod to resume into one WandB run.
Monitoring
Bash
nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total,temperature.gpu --format=csv -l 5
watch -n 2 "perfquery -x | grep -E 'XmtData|RcvData'" # IB throughput countersOnly rank 0 prints training output. For comm issues set NCCL_DEBUG=INFO and NCCL_DEBUG_SUBSYS=ALL, then
look for NCCL INFO NET/IB (connection up) and NCCL WARN (fallbacks/timeouts).
Troubleshooting
Pods that never rendezvous, InfiniBand down, a resume that finds no checkpoint on one pod, and the rest: Troubleshooting. Cross-node dispatch is per-operation latency-bound, so a slow cross-node EP run is a case for node-local EP or a larger batch (DeepEP).
Related pages
Source on GitHubUpdated 18 November 2026