Parallelism
Plain data parallelism needs no flags. halo launch <method> <config> -n 8 (or
torchrun --nproc_per_node=8) shards the model across 8 GPUs with FSDP2. Each
GPU loads the whole model before the wrap, so a dense model must fit on one GPU
at its load dtype; past that, TP shards it as it loads
(Data Parallelism ↗).
Add a mode below only when the model or the sequence does not fit.
EP combines with one of TP, CP or ETP (EP+TP for MoE plus attention sharding, EP+CP for MoE plus long context). PP combines with EP or pure ETP inside each stage.
The data-parallel size is what PP, CP, TP and ETP leave; EP does not reduce it, because an EP rank is also a data-parallel rank (formula ↗).
How EP groups work

Each token goes to the rank that holds its expert and comes back to the rank it came from.

Ranks form dispatch groups of ep_size. Ranks that hold the same expert slice
are DP replicas, and their expert gradients are averaged after the backward
pass. Most of the rules below are about how these groups may be shaped.
Rules that save you a wasted run
Halo validates the layout at startup and rejects an invalid shape with a message that names the rule. Most checks run at config time. The CP and TP model checks (head counts, attention classes) run on a meta-device copy of the model before any weight loads, and LoRA under TP fails at trainer construction.
- Allowed combinations are in the Supported matrix.
- Within one NVLink domain, EP fills the domain or uses
ep_size=2. EP=4 on 8 GPUs is rejected, and so isep4 + tp2; for a 4-way expert split on 8 GPUs, useep4 + etp2(why ↗). - Every other layout rule (what stays inside one NVLink domain, which is
the node, or the rack on NVL72 with
NVLINK_DOMAIN_SIZE=72; EP+CP; HSDP; the FSDP knobs) is in Topology rules ↗. - LoRA, QLoRA and the trainers each mode covers are in the Supported matrix and its Training methods table.
Context parallelism

CP splits the sequence across ranks and switches to a head split for the
attention itself, so the result is exact. The cost is two all-to-alls per
attention layer. The head counts must divide by cp_size; the collator pads the
sequence length to fit.
Pipeline parallelism

pipeline_parallel_size splits the decoder layers into stages, each a whole
number of NVLink domains, so only boundary activations cross nodes. A run needs
at least two nodes, and pp2 under one torchrun on an 8-GPU node is rejected;
the figure's one-host rows run two 4-GPU torchrun agents, which count as two
nodes. Inside a stage it runs
FSDP2, EP or pure ETP. The methods and families it covers, multimodal text towers
and Qwen3.8-Flash-Next included, are in the
Supported matrix; any other model is
refused by name before a weight loads.
- When it pays. Between machines on a fast link, PP is not a speedup for a model
that fits on one machine: it runs 4.0% below
dp16and 15.9% belowep8with DP. It leads on a slow link (13.8× overdp16forced onto plain TCP sockets), over layouts that cross the link with experts (3.82× overep16), and where experts and optimizer state do not fit one machine: GLM-5.3-Flash runs 1.61× overep16, andep8with DP runs out of memory (rows ↗). - Batches.
pipeline_microbatchesreplaces gradient accumulation and must divideper_device_train_batch_size. Every batch pads tomax_length, so usepackingwith one packed row per microbatch. Inkling, GLM-5 Next and DeepSeek-V4 refuse packing under PP and train on padded rows. - Gradient checkpointing. Turn it off where memory allows. The shipped
recipes do, and run 19–23% faster than with it, at about twice the allocated
peak. Reentrant checkpointing is refused;
use_reentrant: falseworks. - Refused: LoRA and any PEFT model, QLoRA, tied embeddings,
accelerate launch,padding_free,torch_compile, Muon and FlashAdamW,save_sharded_ep, image data, andaux_losswith a live coefficient, so a MoE without an exportable bias trains atmoe_balancing: none(every refusal and why ↗).
Launch recipes and per-node merges: Clusters.
Picking a shape
Sharding costs throughput, so use the least that fits (Performance).
Deep dives: Expert ↗ · Context ↗ · Tensor ↗ · Expert-Tensor ↗ · Pipeline ↗ · Multi-Node ↗.
Source on GitHubUpdated 18 November 2026