# What is FSDP?

FSDP (Fully Sharded Data Parallel) is PyTorch's method for training a model across GPUs without a full copy on each. Parameters, gradients and optimizer state are split into shards, and a layer's weights are gathered only while that layer runs.

## How it works

Plain [data parallelism](/glossary/data-parallelism) keeps the whole model, its gradients and its optimizer state on every GPU. FSDP gives each GPU one shard of each: with eight GPUs, an eighth.

A layer needs its full weights to run, so the GPUs work layer by layer:

1. Before a layer runs, the GPUs all-gather its parameter shards into a full copy.
2. The layer computes, and the full copy is freed.
3. In the backward pass the layer is gathered again and its gradient is reduce-scattered, so each GPU keeps only its gradient shard.
4. The optimizer updates each GPU's shard locally.

## What it costs

FSDP trades memory for communication: a gather and a reduce for each layer on every step. Each GPU still keeps its own activations, so it does not help with one sequence that is too long. That takes [context parallelism](/glossary/context-parallelism).

## In Halo

FSDP2, the second version of PyTorch's implementation, is the default: `-n 8` gives FSDP2 sharding across 8 GPUs. Any Hugging Face `AutoModelForCausalLM` runs under it.

For [multi-node](/glossary/multi-node-training) data parallelism, `--use_hsdp` shards within the NVLink domain (the node, on a typical 8-GPU host) and replicates across domains, so only the gradient all-reduce crosses the fabric. It is rejected with expert, tensor and expert-tensor parallelism. See [Parallelism](/halo/docs/parallelism).

