SkyPilot Deployment
Automated cluster provisioning for MoE training on AWS (EFA) and Nebius (InfiniBand). Four tasks live at launcher-configs/skypilot/{aws,nebius}/{nodelocal,crossnode}-ep.yaml, one per cloud fabric and EP topology, each defaulting to GPT-OSS 120B; the presets retarget them at launch, and other MoE families swap CONFIG_PATH.
Install and authenticate
Bash
pip install "skypilot-nightly[nebius,aws]" # nightly has best H200 availabilityNebius: install the CLI (curl -sSL https://storage.ai.nebius.cloud/nebius/install.sh | bash), then nebius init and run nebius-setup.sh from nebius-solution-library/skypilot. AWS: aws configure. Verify with sky check.
Launch
Every task needs two edits first, or the launch burns a provisioning cycle before failing:
set resources.image_id (uncomment it, or pass --image-id; Docker image — the setup:
block exits non-zero when flash_attn / deep_ep are missing) and set the S3 mount bucket
(--env HALO_BUCKET=<your-bucket>) to your bucket (Storage). Each task also pins its
provider's region (us-east-1 on AWS, eu-north1 on Nebius) — change it, or delete the key for
cross-region failover, to match your quota.
Bash
sky launch -c oss-120b launcher-configs/skypilot/nebius/crossnode-ep.yaml \
--secret HF_TOKEN --secret WANDB_API_KEY # --secret masks tokens in logsAs shipped, nodelocal-ep runs EP=8 at node scope on one B200:8 node (halo:blackwell) and crossnode-ep runs EP=16 at global scope on two H200:8 nodes (halo:hopper). Each task drives its run from envs: CONFIG_PATH (the training config) plus MODEL, EXPERT_PARALLEL_SIZE, EXPERT_PARALLEL_SCOPE, OUTPUT_DIR and WANDB_PROJECT, forwarded as --model_name_or_path / --expert_parallel_size / --ep_scope / --output_dir / --project_name. Every envs: key takes a launch-time --env KEY=VALUE and every resources field its sky launch flag (--gpus, --num-nodes, --memory, --disk-size, --network-tier, --image-id), so no retarget edits the YAML. Point MODEL at a stage-1 output_dir to chain stages, and retarget OUTPUT_DIR and WANDB_PROJECT alongside MODEL, or the run writes into the shipped checkpoint directory and project.
The two crossnode-ep files are not interchangeable: beyond cloud/region they differ in the RDMA fabric setup below (fabric env, the /dev/gdrdrv device, the setup: probe), and no launch flag supplies the device or the probe. The two nodelocal-ep files differ only in the EFA env the AWS run: block exports on more than one node.
Presets
The 120B and 122B models run nodelocal-ep on Blackwell only (B200's 180 GB/GPU is the minimum, B300 has headroom; lower max_length on OOM) and need the shipped memory: 512+ and disk_size: 3000. Both gpt-oss presets load the release as OpenAI ships it, openai/gpt-oss-120b by default and openai/gpt-oss-20b for the 20B row, its experts decoded at load (GPT-OSS → Configs). The 20B preset keeps EP inside each node and adds cross-node data parallelism over the fabric, hence --network-tier best (RDMA fabric).
Cross-node expert parallelism
Cross-node EP distributes MoE experts across nodes — at EP=16 each rank owns num_experts / 16 experts
(GPT-OSS-120B's 128 experts → 8 per rank, node 0 holding 0–63 and node 1 holding 64–127) — reached via
NVLink within a node and InfiniBand/EFA RDMA between them, fitting models larger than one node.
Set in the SkyPilot YAML envs:
YAML
envs:
EXPERT_PARALLEL_SIZE: "16" # EP group size across both nodes
EXPERT_PARALLEL_SCOPE: "global" # cross-node (not node-local)
MAX_LENGTH: "8192" # cross-node dispatch ceiling (~8k tokens/rank over Gin); pins any overridden config to it
DIST_SHARED_FILESYSTEM: "1" # default; "0" for local-only FSBoth run: blocks are a plain torchrun --nnodes=$SKYPILOT_NUM_NODES ... scripts/training/sft.py $CONFIG_PATH
that forwards --expert_parallel_size / --ep_scope (and, in crossnode-ep, --max_length) from these envs — it does not use the
halo CLI. MAX_LENGTH is a crossnode-ep knob: the training configs are tuned for a single NVLink
domain, while cross-node dispatch is capped at 8192 tokens/rank
(DeepEP → AWS EFA) and is rejected above it.
RDMA fabric: AWS EFA vs Nebius InfiniBand
The Nebius (IB) tasks keep the image's InfiniBand defaults. A multi-node AWS (EFA) run sets the EFA
launch env: crossnode-ep in its envs, AWS nodelocal-ep in its run: block only when
SKYPILOT_NUM_NODES > 1, so one node stays NVLink-only. NCCL_PROTO is left to the plugin's own probe.
YAML
envs: # AWS crossnode-ep; nodelocal-ep exports the first two on >1 node
NCCL_NET_PLUGIN: ofi
NCCL_NET: Libfabric
NCCL_GIN_TYPE: "2" # cross-node EP only (proxy GIN)Every task sets config.docker.run_options: ["--ipc=host", "--ulimit", "memlock=-1", "--ulimit", "stack=67108864", "--shm-size=128g"]; only AWS crossnode-ep prepends
"--device", "/dev/gdrdrv" — node-local EP needs neither proxy GIN nor GDRCopy.
crossnode-ep sets resources.network_tier: best so SkyPilot provisions the EFA/InfiniBand interfaces; nodelocal-ep omits it, so a multi-node nodelocal-ep launch passes --network-tier best.
On AWS, SkyPilot attaches only one EFA interface unless aws.use_internal_ips: true is set in ~/.sky/config.yaml, which needs a private subnet plus aws.use_ssm or aws.ssh_proxy_command; without it, all ranks share one NIC. A task file cannot set it.
On Nebius that flag is the only fabric guard: its tasks name no NCCL_NET, so a multi-node nodelocal-ep launched without it runs the cross-node gradient sync over TCP sockets with no error (AWS's NCCL_NET=Libfabric fails the launch instead).
GIN-plugin prerequisites and the measured B300 EFA numbers: DeepEP → Expert parallelism over AWS EFA. Per-fabric env and verification (fi_info -p efa vs ibstat): Multi-Node → RDMA fabrics.
NVL72 racks (GB200/GB300)
On NVL72 the NVLink domain spans the rack (72 GPUs across ~18 OS nodes), so node-local grouping should target the rack: set NVLINK_DOMAIN_SIZE: "72" and EXPERT_PARALLEL_SCOPE: "node" in envs. Standard 8-GPU hosts need no change — the default reads gpus_per_node. See Multi-Node → NVL72. The published images are x86_64-only, so the rack's Grace (aarch64) hosts need an arm64 build first (Docker).
NCCL collective timeout
Model loading rarely holds a collective for long: the source agreement and the eager loads join over the c10d store (DIST_STORE_TIMEOUT_HOURS), and the default lazy EP load reads on every rank at once (Multi-Node → Model loading). A gathered save does: the non-writing ranks wait in its per-layer expert gathers while the save rank streams each flush to disk (Checkpoints → Multi-node). Every shipped crossnode-ep task pins DIST_NCCL_TIMEOUT_MINUTES: "60" in envs; raise it there if a save is slower still.
Storage
Every shipped task declares two mounts:
YAML
file_mounts:
/workspace: {source: ., mode: COPY} # the repo
/data: {source: "s3://${HALO_BUCKET}/halo", mode: MOUNT} # bucket from envs; --env HALO_BUCKET=... retargetsHALO_BUCKET is the placeholder you must set (--env HALO_BUCKET=<your-bucket>); HF_HOME,
HALO_DATA_ROOT (the toolkit scratch root) and OUTPUT_DIR all resolve under /data.
Pre-upload large models (>10GB) to S3 once. /data is the bucket's halo/ prefix, so sync below it:
Bash
aws s3 sync models/qwen3.5-122b-a10b s3://<your-bucket>/halo/models/qwen3.5-122b-a10b/Then point the run at it with --env MODEL=/data/models/qwen3.5-122b-a10b; the shipped tasks default
MODEL to a Hub id and keep only their writes under /data (OUTPUT_DIR: /data/checkpoints/...).
For MOUNT_CACHED reads, add a second file_mounts entry for the model prefix and point MODEL at
it, keeping checkpoints on the MOUNT. Alternatively download from HF at runtime with HF_HOME on a
MOUNT_CACHED bucket so the cache survives restarts.
In multi-node training a MOUNT checkpoint bucket is shared across all nodes; any node can save and
reload after preemption. Pull results with aws s3 sync,
rsync -Pavz oss-120b:/data/checkpoints/ ./, or SSH; manage buckets with sky storage ls / delete.
Docker image
resources.image_id is not set in the shipped YAMLs — uncomment it or pass --image-id. Each carries
it as a commented line under resources naming the prebuilt image for its shipped GPUs, which pulls
anonymously (no AWS account, no registry login).
Each task's setup: block fails fast when flash_attn / deep_ep are not importable, so a from-source
build on a bare node is not a fallback; the prebuilt image is required. The hopper image serves
H100/H200 (crossnode-ep, the GPT-OSS 20B preset) and blackwell B200/B300 (nodelocal-ep as shipped). To run your own build
instead, push it to a registry the nodes can reach and point image_id there (Docker).
YAML
resources:
image_id: docker:public.ecr.aws/whitecircle/halo:hopperCluster management
Bash
sky status # list clusters
sky logs <cluster> --follow # logs
ssh <cluster> # SSH in
sky stop <cluster> # stop, preserve state
sky down <cluster> # terminateSpot instances (AWS): spot with auto-recovery needs managed jobs — sky jobs launch with use_spot: true (job_recovery: failover); on a plain sky launch cluster spot preemption is not recovered. Auto-stop idle clusters with sky launch --idle-minutes-to-autostop 30. Check pricing with sky show-gpus H200.
Troubleshooting
- Cross-node EP hangs: verify RDMA —
ibstat(Nebius) orfi_info -p efa(AWS); on AWS also confirm/dev/gdrdrvreached the container. - NCCL errors: check the NCCL env vars in the YAML and the launch's
--env; inspect setup withsky logs <cluster> --status.
OOM, NCCL timeouts and config rejections: Troubleshooting.
Resources
Source on GitHubUpdated 18 November 2026