Data parallelism trains one model on several GPUs by giving each GPU a different slice of the batch. Every GPU computes gradients on its own slice, and the gradients are averaged into a single update.
How it works
Every GPU runs the same model on different rows. Each does its own forward and backward pass, the gradients are averaged across the GPUs, and every GPU applies the same update, so the copies stay identical.
The effective batch grows with the GPU count: per-GPU batch × GPUs × gradient accumulation steps.
Limits
Data parallelism adds throughput, not capacity. In its plain form every GPU holds the whole model, its gradients and its optimizer state, so the model has to fit on one GPU. FSDP (Fully Sharded Data Parallel) lifts that limit by sharding all three across the GPUs.
It also does nothing for one sequence that is too long. That takes context parallelism.
In Halo
Data parallelism is the default, sharded with FSDP2. -n sets the number of GPUs:
Bash
halo launch sft examples/sft/qwen3/qwen3-4b-ultrachat.yaml -n 8When context, tensor or expert-tensor parallelism is on, the data-parallel size is what is left over:
data_parallel_size = world_size / max(cp_size, tp_size, expert_tp_size)
Expert parallelism is orthogonal: its GPUs are also data-parallel ones. The effective batch is per_device_train_batch_size × gradient_accumulation_steps × data_parallel_size. See Parallelism.
