Parallelism

Plain data parallelism needs no flags. halo launch <method> <config> -n 8 (or torchrun --nproc_per_node=8) shards the model across 8 GPUs with FSDP2. Each GPU loads the whole model before the wrap, so a dense model must fit on one GPU at its load dtype; past that, TP shards it as it loads (Data Parallelism ↗). Add a mode below only when the model or the sequence does not fit.

NeedModeFlag
Shard MoE experts across GPUsEP--expert_parallel_size=N
Shard each expert's FFNETP--expert_tensor_parallel_size=N
Shard dense weights (attention and FFN)TP--tensor_parallel_size=N
Split a long sequence across GPUsCP--context_parallel_size=N
Split a model's layers across nodesPP--pipeline_parallel_size=N
Cut cross-node FSDP traffic (DP or CP)HSDP--use_hsdp=true

EP combines with one of TP, CP or ETP (EP+TP for MoE plus attention sharding, EP+CP for MoE plus long context). PP combines with EP or pure ETP inside each stage.

The data-parallel size is what PP, CP, TP and ETP leave; EP does not reduce it, because an EP rank is also a data-parallel rank (formula ↗).

How EP groups work

Tokens routed to the ranks owning their experts and returned to the rank they came from
Tokens routed to the ranks owning their experts and returned to the rank they came from

Each token goes to the rank that holds its expert and comes back to the rank it came from.

Two EP dispatch groups inside one NVLink domain, with the ranks holding the same expert slice forming DP replicas
Two EP dispatch groups inside one NVLink domain, with the ranks holding the same expert slice forming DP replicas

Ranks form dispatch groups of ep_size. Ranks that hold the same expert slice are DP replicas, and their expert gradients are averaged after the backward pass. Most of the rules below are about how these groups may be shaped.

Rules that save you a wasted run

Halo validates the layout at startup and rejects an invalid shape with a message that names the rule. Most checks run at config time. The CP and TP model checks (head counts, attention classes) run on a meta-device copy of the model before any weight loads, and LoRA under TP fails at trainer construction.

  • Allowed combinations are in the Supported matrix.
  • Within one NVLink domain, EP fills the domain or uses ep_size=2. EP=4 on 8 GPUs is rejected, and so is ep4 + tp2; for a 4-way expert split on 8 GPUs, use ep4 + etp2 (why ↗).
  • Every other layout rule (what stays inside one NVLink domain, which is the node, or the rack on NVL72 with NVLINK_DOMAIN_SIZE=72; EP+CP; HSDP; the FSDP knobs) is in Topology rules ↗.
  • LoRA, QLoRA and the trainers each mode covers are in the Supported matrix and its Training methods table.

Context parallelism

Ulysses attention: each rank holds a sequence chunk, all-to-all swaps it for a head slice of the full sequence, then swaps back
Ulysses attention: each rank holds a sequence chunk, all-to-all swaps it for a head slice of the full sequence, then swaps back

CP splits the sequence across ranks and switches to a head split for the attention itself, so the result is exact. The cost is two all-to-alls per attention layer. The head counts must divide by cp_size; the collator pads the sequence length to fit.

Pipeline parallelism

Tokens/s/GPU with and without pipeline parallelism: on two 8-GPU machines over a fast link PP runs 4.0% below dp16 and 15.9% below ep8 with DP; forced onto plain TCP sockets it leads dp16 by 13.8×; on one machine it runs 8.5% below dp8 and 9.2% above ep8; on GLM-5.3-Flash it leads ep16 by 1.61×
Tokens/s/GPU with and without pipeline parallelism: on two 8-GPU machines over a fast link PP runs 4.0% below dp16 and 15.9% below ep8 with DP; forced onto plain TCP sockets it leads dp16 by 13.8×; on one machine it runs 8.5% below dp8 and 9.2% above ep8; on GLM-5.3-Flash it leads ep16 by 1.61×

pipeline_parallel_size splits the decoder layers into stages, each a whole number of NVLink domains, so only boundary activations cross nodes. A run needs at least two nodes, and pp2 under one torchrun on an 8-GPU node is rejected; the figure's one-host rows run two 4-GPU torchrun agents, which count as two nodes. Inside a stage it runs FSDP2, EP or pure ETP. The methods and families it covers, multimodal text towers and Qwen3.8-Flash-Next included, are in the Supported matrix; any other model is refused by name before a weight loads.

  • When it pays. Between machines on a fast link, PP is not a speedup for a model that fits on one machine: it runs 4.0% below dp16 and 15.9% below ep8 with DP. It leads on a slow link (13.8× over dp16 forced onto plain TCP sockets), over layouts that cross the link with experts (3.82× over ep16), and where experts and optimizer state do not fit one machine: GLM-5.3-Flash runs 1.61× over ep16, and ep8 with DP runs out of memory (rows ↗).
  • Batches. pipeline_microbatches replaces gradient accumulation and must divide per_device_train_batch_size. Every batch pads to max_length, so use packing with one packed row per microbatch. Inkling, GLM-5 Next and DeepSeek-V4 refuse packing under PP and train on padded rows.
  • Gradient checkpointing. Turn it off where memory allows. The shipped recipes do, and run 19–23% faster than with it, at about twice the allocated peak. Reentrant checkpointing is refused; use_reentrant: false works.
  • Refused: LoRA and any PEFT model, QLoRA, tied embeddings, accelerate launch, padding_free, torch_compile, Muon and FlashAdamW, save_sharded_ep, image data, and aux_loss with a live coefficient, so a MoE without an exportable bias trains at moe_balancing: none (every refusal and why ↗).

Launch recipes and per-node merges: Clusters.

Picking a shape

Sharding costs throughput, so use the least that fits (Performance).

Deep dives: Expert ↗ · Context ↗ · Tensor ↗ · Expert-Tensor ↗ · Pipeline ↗ · Multi-Node ↗.

Source on GitHubUpdated 18 November 2026