# Multi-node training

Multi-node training runs one training job across several machines. Each node starts its own processes, the processes join one group over the network, and the GPUs exchange data over the cluster's fabric.

## What it needs

- **A launcher on every node.** Each machine starts its own processes, one per GPU, with the same arguments. They meet at a shared address and port and form one group.
- **A fast fabric.** Inside a node the GPUs talk over NVLink. Between nodes they use the network: an RDMA (remote direct memory access) fabric such as InfiniBand, which bypasses the operating system's network stack, or plain TCP over Ethernet, which is much slower.
- **Storage.** Either a shared filesystem that every node sees, or a local disk on each node.

## What crosses the network

The network is the slow link, so a good layout sends little over it. Replicated [data parallelism](/glossary/data-parallelism) sends only gradients between nodes. [Tensor parallelism](/glossary/tensor-parallelism) talks inside every layer and stays within a node.

## In Halo

Multi-node training is one `torchrun` per node, each inside its own container running the same image:

```bash
torchrun --nnodes=2 --node_rank=$NODE_RANK --nproc_per_node=8 \
    --master_addr=$MASTER_ADDR --master_port=29500 \
    scripts/training/sft.py examples/sft/gptoss/gptoss-20b-multinode-ep.yaml
```

Only `--node_rank` differs between nodes. `MASTER_ADDR` has to be an address the compute fabric can route. `DIST_SHARED_FILESYSTEM=1`, the default, means a shared filesystem; set it to `0` for per-node disks. InfiniBand and RoCE run on the image's defaults, but only AWS EFA (Elastic Fabric Adapter) is validated by real multi-node runs. There is no built-in scheduler: SkyPilot and Nomad templates are included. See [Clusters and multi-node](/halo/docs/clusters).

