FSDP (Fully Sharded Data Parallel) is PyTorch's method for training a model across GPUs without a full copy on each. Parameters, gradients and optimizer state are split into shards, and a layer's weights are gathered only while that layer runs.
How it works
Plain data parallelism keeps the whole model, its gradients and its optimizer state on every GPU. FSDP gives each GPU one shard of each: with eight GPUs, an eighth.
A layer needs its full weights to run, so the GPUs work layer by layer:
- Before a layer runs, the GPUs all-gather its parameter shards into a full copy.
- The layer computes, and the full copy is freed.
- In the backward pass the layer is gathered again and its gradient is reduce-scattered, so each GPU keeps only its gradient shard.
- The optimizer updates each GPU's shard locally.
What it costs
FSDP trades memory for communication: a gather and a reduce for each layer on every step. Each GPU still keeps its own activations, so it does not help with one sequence that is too long. That takes context parallelism.
In Halo
FSDP2, the second version of PyTorch's implementation, is the default: -n 8 gives FSDP2 sharding across 8 GPUs. Any Hugging Face AutoModelForCausalLM runs under it.
For multi-node data parallelism, --use_hsdp shards within the NVLink domain (the node, on a typical 8-GPU host) and replicates across domains, so only the gradient all-reduce crosses the fabric. It is rejected with expert, tensor and expert-tensor parallelism. See Parallelism.
