Pipeline Parallelism
PP splits a model's decoder layers into contiguous stages, each owning a rank block. It is the outermost parallelism dimension and the only one designed to cross NVLink domains: the only traffic leaving a domain is the pipeline's point-to-point boundary activations. Between machines on a fast link it is not a speedup for a model that fits on one machine (measured).
The expert axes are the only other sharding PP composes with: EP, or pure ETP, never both. FSDP2 data parallelism runs inside each stage. Every other pairing is refused by the capability matrix. The bubble arithmetic and the bandwidth argument: GPU Training Theory §9.
Why PP is the cross-node dimension
Per-step traffic that crosses RDMA, by mode:
When that trade pays, and when it does not (inside one domain, or on an NVL72 rack): When pipeline parallelism is worth it.
Launching
Set pipeline_parallel_size on a script whose trainer declares PP_REFUSAL = None: SFT, SMPO, DPO,
KTO, offline GRPO, Bradley-Terry reward and classification
(Trainer support). Any other script refuses it before a model loads with
<Trainer> does not support Pipeline Parallelism (PP): <its PP_REFUSAL>. The model's backbone must be
registered (Model support); any other is refused by name on the meta shell, on every
rank, before a weight is read.
Launch one torchrun per node (halo launch is single-node). Every stage is a whole number of NVLink
domains holding at least two ranks, so a run spans at least two domains — two nodes at the default
domain sizing. Two recipes ship, both one stage per node on 2 × 8 GPUs:
A per-node output filesystem leaves each node only its own stage's parts; merge the node
directories with scripts/after_training/merge_pp_stages.py before from_pretrained
(Per-node export).
Rank layout
PP carves the world first; every other mode runs unchanged inside one stage:
Text
stage_world_size = world_size / pipeline_parallel_size
pp_rank = global_rank // stage_world_sizeA pipeline chain is the set of ranks holding the same intra-stage position across every stage.
Chain members are stage_world_size apart, share a data-parallel coordinate, and consume the same
batch — stage 0 reads input_ids, the last stage reads labels. The trainer's dataloader shards by
the stage-local DP rank, never the global rank.
Stage boundaries fall on NVLink-domain boundaries (stage_world_size % nvlink_domain_size == 0,
enforced at config time). Every EP/TP/ETP/CP group is a domain-local rank block, so this keeps them
strictly inside one stage. pp_size must therefore divide the domain count: pp2 on one 8-GPU node
leaves stage_world_size = 4 and is rejected at the default domain sizing. Single-rank stages are
rejected too (the stage FSDP wrap assumes DP ≥ 2).
Configuration
The four knobs (pipeline_parallel_size, pipeline_microbatches, pipeline_schedule,
pipeline_split), their defaults and constraints:
Configuration reference → ParallelismConfig.
How they interact with the trainer's batch settings:
- The microbatches are the gradient accumulation: an unset
pipeline_microbatchestakesgradient_accumulation_steps, setting both raises, and the trainer then runs withgradient_accumulation_steps=1. Under1f1ba count belowpp_sizeis raised to it, with a warning. per_device_train_batch_sizeis the chain's rows per optimizer step, split into the microbatches, so it must be a multiple of their count (the recipes run 8 rows in 8 microbatches).data_parallel_sizeis divided bypipeline_parallel_size: a whole chain consumes one batch.- A
pipeline_splitsums to the layer count with boundaries on thelayer_typesperiod (Split-offset constraint). max_lengthmust be set: the pipeline freezes its P2P shapes on the first step and pads every batch to it. Offline GRPO has no such key and pads tomax_prompt_length + max_completion_length(both required). Amax_lengthat the model's context window is warned, since every row then pays full-window compute. The train dataloader runs withdrop_last=True, forced and logged when the config leaves it off.
Microbatch count sizes the bubble. Below 4 × pp_size microbatches the trainer warns: at
2 × pp_size the 1F1B bubble (pp − 1)/(m + pp − 1) is a fifth of the step at pp_size = 2 and grows
toward a third with pp_size. Plan for 9 × (pp_size − 1) microbatches to hold it at 10%.
Raising pp_size does not shrink the bubble. With one row per microbatch (what packing requires)
and no ETP, the count is m = T × pp_size for T = global_batch_tokens / (world_size × max_length),
so the 1F1B bubble (pp − 1)/(m + pp − 1) rises with pp_size, toward 1/(1 + T). The global
batch you accept sets the bubble, not the stage count: at world 512, max_length 8192, pp8, a
4.2M-token step is m = 8 and a 47% bubble; 10% needs m = 63, ~33M tokens per step. Only an
interleaved (virtual-stage) schedule changes this, and none is wired.
Stage partition. The default split is head-weighted: the last stage carries the lm_head
matmul + CE (for Qwen3-8B, ~3 decoder layers' worth of compute), so its layer budget shrinks by that
cost, snapped to the layer_types period, and the earlier stages absorb the difference. Any
partition is gradient-exact; only throughput changes. The chosen partition is logged at startup.
Model support
A stage is the real backbone with its layer list sliced, the final norm neutralized on non-last
stages, the input embedding (found through get_input_embeddings, with any norm it carries) dropped
off non-first stages, and non-first stages driven through inputs_embeds, so each family's rotary
embeddings, mask construction, layer_types handling and EP layers are inherited.
Families are registered in PP_FAMILY_SPECS (halo/distributed/pipeline_parallel/families.py), one
PPFamilySpec per family keyed by its backbone class. A spec with validated=True is registered: its
stages reproduce the unsplit model on the pipeline GPU tier, and
tests/cpu/conventions/test_pp_family_roster.py holds every such family to a GPU row that runs it, a
MoE one also with its experts distributed. A spec with a refusal is refused with its reason, and a
backbone no spec registers is refused by name —
<Backbone> is not registered for pipeline parallelism (PP_FAMILY_SPECS); registered: ....
reject_unsupported_pp_structure (model_rules.py) checks, in order, the decoder-layer list, the
family refusal, the tie gate, the head transform, live MTP layers and the registration, so a tied or
MTP-carrying checkpoint names that reason first.
Models with live MTP tail layers are refused: MTP re-embeds input_ids mid-backbone, a second
cross-stage stream. Live means built — the decoder list is longer than num_hidden_layers — so an MTP
count the native families ship as checkpoint metadata (num_nextn_predict_layers,
mtp_num_hidden_layers, num_mtp_layers; transformers drops those weights at load) passes.
Hyper-connection families carry the widened residual between layers: DeepSeek-V4 and GLM-5 Next a
[batch, seq, hc_mult, hidden] stream collapsed by hc_head, Qwen3.8-Flash-Next a flat
[batch, seq, hc_count * hidden] one collapsed by hyper_connection_mixer (its only post-layers module, declared
as its final norm). Each spec neutralizes the collapse off the last stage like the norm, and binds a mid-chain
forward that feeds the received stream to the layer loop instead of re-widening it, refusing anything but the
full stream. All three select per-layer inputs by the layer's own attributes (layer_types_rebase_safe), so any
split offset is exact.
Token ids on later stages
A family whose layers re-read the token ids mid-network declares the backbone kwarg it takes them under
(PPFamilySpec.token_ids_kwarg): Qwen3.8-Flash-Next's per-layer embeddings (PLE) look their n-grams up from
ple_input_ids. Every rank of a chain already holds the batch, so the runner hands the non-first stages of such a
family the microbatch's input_ids as a token_ids kwarg, split with the rows, and nothing crosses a boundary;
a stage that should take them and gets none raises. DeepSeek-V4 declares none: its hash_moe layers stay on
stage 0.
Boundary cost
One bf16 tensor crosses each stage edge per microbatch per direction (the activation forward, its
gradient back) and nothing else: masks, position_ids and labels are re-sliced on every rank from
the batch it already holds, and shape inference exchanges meta tensors once.
Forward plus backward doubles each cell; 1f1b waits on every P2P batch before the dependent
compute, so those figures are exposed, not overlapped. torch keeps 2 × pipeline_microbatches
boundary buffers per mid-chain stage for the life of the run: 16 GiB at 8 microbatches of 32k tokens
on the hc_mult=4 stream (4 GiB for a plain residual). Only pipeline_parallel_size changes the
traffic; the cut position does not.
Measured between two B300 nodes over EFA, NCCL send/recv from each rank to its peer on the other node moves 92.8 GB/s per GPU pair at 256 MiB and 96.2 GB/s at 1 GiB, and keeps that rate with all eight pairs sending at once (776 GB/s in total). That is about twice the EFA column, so the two crossings above take 2.9 ms and 11.2 ms there.
Split-offset constraint
GptOss, Qwen3, Qwen3.5/3.6, Step-3.7, Laguna and Inkling select each layer's attention mask by position in the
layer list. A stage starting off a whole number of layer_types periods would give sliding-window layers a
full causal mask, so validate_partition (partition.py) raises at split time, on every rank; the
hyper-connection families (layer_types_rebase_safe) are exempt. The period is the
smallest shift p with layer_types[p:] == layer_types[:-p], so a ragged list (Step-3.7-Flash: 45
layers = 11 whole full,s,s,s periods plus one) still has period 4.
That caps the stage count at pp_size <= num_layers // period. The default splitter snaps every
boundary to the period and leaves the ragged tail on the last stage (45 layers at period 4, pp2 →
[24, 21]); a manual pipeline_split gets no such help: at 60 layers a hand-written [30, 30]
is rejected where the default [32, 28] passes.
PP + EP (MoE)
EP dispatch/combine stays inside each stage — the EP groups carry the stage's rank offset — while only P2P activations cross stages. This is the cross-node shape for a large MoE.
- Every stage must carry at least one MoE layer at
ep_group_size > 1(EP and PP+ETP): a dense stage runs a different collective program than its MoE peers and deadlocks at the gradient norm, so such a partition raises at setup. Choose apipeline_parallel_sizeorpipeline_splitthat reaches the MoE layers on every stage. - The stage FSDP wrap is EP-aware: EP layers are FSDP-ignored (
ep_size>1) or sharded as stage-mesh DTensors (ep_size==1+fsdp_shard_ep1_experts);fsdp_shard_ep1_experts: falseis rejected under PP. - EP gradient sync runs in one deferred post-backward sweep (
EPConfig.sweep_reduces_across_replicas), since per-backward hooks would fire on every microbatch and re-scale grads already synced. The sweep is off atep_size == 1with the defaultfsdp_shard_ep1_experts, where FSDP2's own reduce-scatter syncs the experts. Router and replicated averages reduce over the stage's rank block, never the world. - The gradient norm is exact and rank-uniform: one world all-reduce of each rank's local squared norm where every stage parameter is a DTensor; otherwise stage-scoped reduces, then one sum across the pipeline chain.
- Router balancing:
bias_updateworks on a family whose EP layers export the bias (the counter all-reduce is stage-scoped), andbias_update_transientis refused; the HF aux-loss path is severed by the split, soaux_losswith a liverouter_aux_loss_coefis rejected at split time, the refusal namingbias_updateonly where it exports.MoEMetricsCallbackis skipped under PP. - Gradient checkpointing works: the recompute replays each frame's DeepEP dispatch/combine instead of
issuing a second one, so it composes with the schedule's in-flight microbatches
(EP checkpoint scope). The DeepEP
capacity generation is registered on
PipelineStageModule, so every microbatch sizes its own dispatch. - Equivalence gates: gpt-oss against a bf16 ep1 twin (
tests/gpu/parallelism/pp/test_pp_ep_vs_unsplit.py, both sides on the unfused expert-output sum, since the fused un-permute rounds a rank's local top-k sum once), and every other registered MoE family against its unsplit ep1 twin through the real trainer atep2(test_pp_family_vs_reference.py; fp32 under ETP, and for a hyper-connection family wherever no DeepEP dispatch runs, since its base and scale gradients sit 4 to 5 orders under their layer's largest and bf16 accumulation order decides them; a bf16ep2row leaves those parameters unscored and the fp32 ETP row covers them). One released checkpoint has run the split as a correctness smoke: Qwen3-30B-A3B-Instruct-2507 atpp2 × ep4withmoe_balancing: nonetracksep8over 20 steps (mean relative loss gap 3.4e-4, largest 1.16e-3), as twotorchrunagents on one host.
PP + ETP (MoE)
Pure expert-tensor parallelism (expert_parallel_size: 1, expert_tensor_parallel_size: k) inside
each stage: every rank holds every expert, sharded k-way along the FFN intermediate dim, and
SumAcrossGroup sums the partners' outputs in token space. The stage keeps stage_world_size / k
data-parallel replicas.
- ETP partners hold shards of one expert and consume the same batch, so they are not DP
replicas: the deferred sweep divides the expert leg by
stage_world_size / expert_tp_size. - The ETP all-reduce is model math inside the forward/backward and is not deferred — safe only
because
ep_size == 1leaves no DeepEP combine to interleave with. Adding EP on top is rejected. - The equivalence gate (
tests/gpu/parallelism/pp/test_pp_etp_vs_unsplit.py) runs in fp32: ETP reorders the summation producing each expert's output, which in bf16 flips near-tied top-k picks with no sync bug present. Save, export (each expert whole, under global layer names) and exact resume are covered on GPU (test_pp_ep_resume.py --mode etp2) and on CPU (tests/cpu/checkpoint/test_pp_sharded_save_roundtrip.py).
Rejected combinations
The first five are refused by the capability-matrix allowlist before any rank math; the rest by their own gate.
Fused head and loss on the last stage
The last stage folds lm_head and cross-entropy into its forward and returns the summed loss, so the
[microbatch, seq, vocab] logits plane and its gradient are never materialized. The head runs over
token chunks of max(1, 128M // vocab_size) rows (logit_chunk_rows), each under a non-reentrant
checkpoint. At V=201088, S=8192 and one microbatch row that drops 3.29 GB of bf16 logits plus as
much logit gradient; the loss and activation gradients are unchanged, while the head weight's gradient
is reassociated per chunk (~7e-7 relative in fp32, ~1e-2 in bf16).
On by default; HALO_PP_FUSED_HEAD_LOSS=0 restores the logits path. PipelineLossAdapter.stage_loss
fuses only the causal-LM CE with raw-logit predictions (the default causal_lm_token_loss, no
predictions_fn) — SFT alone; every other trainer declares its own token loss. The forward-only
pass that serves compute_metrics un-fuses and returns real logits. The gate
(tests/gpu/parallelism/pp/test_pp_fused_head_loss.py, pp2×dp2, tiny Qwen3, fp32, seq 1024, vocab
65536, with and without GC) holds loss within 1e-6 relative and every gradient within 1e-5, at
last-stage peak memory 1045.9 → 470.9 MiB.
The last stage applies what the family's *ForCausalLM.forward does around lm_head — Cohere's
logit_scale, Gemma's softcap, Granite's division — to the logits it returns and inside the fused
loss: the transform the chunked GRPO log-probs apply, verified against the family's forward
(halo/models/head_transform.py). A family whose forward it does not reproduce is refused on every
rank.
Attention, masks, and checkpointing
- Padding masks are exact. Mask construction reads only
inputs_embeds, the 2D mask andposition_ids, so a mid-stage forward builds a bitwise identical mask (equivalence-gated on ragged right-padded batches under eager, SDPA and FA4 varlen). - FA4 kernels are pre-warmed by the loader before the split, so no mid-schedule JIT stall.
- Liger
cross_entropyandfused_linear_cross_entropyare force-disabled under PP: a stage forward runs without labels, so FLCE would claim a saving it does not deliver, andcross_entropyrebindsF.cross_entropyprocess-wide under the last-stage loss. The stage loss chunks its own fp32 CE (Liger). - Gradient checkpointing (non-reentrant) is grad-exact: the per-microbatch phase holds FSDP's reduction off, so recompute cannot trigger a premature reduce-scatter.
- The stage holds its unsharded gradient across the microbatch loop. The schedule reduce-scatters
once per step, so each rank keeps a full gradient of its stage's FSDP-managed parameters until
then: 2 B/param in bf16, 4 B/param under
fp32_grad_reduceorfp32_non_ep_params(warned at config time). - Turn GC off where memory allows, dense or with EP. A microbatch's activations are already
1/mof a step. TRL'sSFTConfigturnsgradient_checkpointingon by default; both shipped recipes set it off (Measured throughput).
Measured throughput
Between machines on a fast link, PP is not a speedup for a model that fits on one machine: it runs 4.0% below dp16 on Qwen3-8B and 15.9% below ep8 with DP on gpt-oss-20b. It leads where the alternative sends attention or experts across the link, where the link is slow, and where a model's experts and optimizer state do not fit on one machine. On one machine it gains only with EP, by halving the dispatch group.
Halo 1.2.0, B300, tokens per second per GPU over the steady steps, one run per arm. The arms of a block train the same rows at the same tokens per GPU per step. Peaks are allocated GiB, the highest rank of each stage. Every gpt-oss row runs grouped GEMM and moe_balancing: bias_update, and the Qwen3-8B and gpt-oss rows run packed HuggingFaceH4/ultrachat_200k.
A PP rate counts attended tokens. A PP batch is padded to max_length, and the count reads its attention_mask, or the row_lengths of packed rows, so the rate EfficiencyCallback logs equals the one of any other layout on the same rows. Halo 1.2.0 counted every padded row and logged 17,709 for the 17,608 below, 15,820 for 15,715, and 2,556 for GLM-5.3-Flash's 2,423; the PP rates on this page divide the attended tokens of the matching non-PP arm, on the same rows, by PP's step time.

Two 8-GPU machines, fast link (EFA, one stage per machine), max_length 4096 on Qwen3-8B and 8192 on gpt-oss-20b, m=8 (a 1/9 bubble), median of 10 steps:
- Against DP, with any experts local, PP runs 4.0% and 1.3% below
dp16, GC off then on, and 15.9% and 14.3% belowep8with DP. - Against the layouts that cross the link it leads: 1.63× and 2.22× over
dp8 × tp2, 3.82× and 3.36× overep16, whose dispatch crosses it in every MoE layer. Both PP rates repeat within 0.4%. - PP's memory saving falls on its last stage: 0.68× the DP row's peak on Qwen3-8B and 0.45× on gpt-oss-20b, GC off. The first stage holds 0.98× and 0.84×.
- GC off raises PP's own rate 18.9% on Qwen3-8B and 22.8% on gpt-oss-20b, for 1.9× and 2.1× the allocated peak. The whole-device peaks are 49.2 GiB on stage 0 and 35.7 GiB on stage 1 for Qwen3-8B, 70.4 and 40.8 GiB for gpt-oss-20b.
A slow link. The same two machines forced onto plain TCP sockets, Qwen3-8B, GC off: pp2 × dp8 holds 15,414, 0.875× its fast-link rate, while dp16 falls to 1,117, 0.061× of its own. PP leads by 13.8×.
A model that does not fit on one machine's expert group. GLM-5.3-Flash full fine-tune, two 8-GPU machines, GC on, m=8, one padded row of about 7,750 tokens per microbatch, stages of 24 and 21 layers: pp2 × ep8 runs 2,423 at 12.82 s a step, 1.61× the 1,504 at 20.63 s of ep16 global, at 218.7 / 197.9 GiB against 214.1. On torch's general attention kernel and upstream's hyper-connections the same arms read about 1,060 and 913, a 1.16× lead (GLM-5 Next → Throughput). ep8 with DP runs out of memory in the first optimizer step: at 40.2B trainable per rank, AdamWBF16's state takes the allocation to 243–250 GiB of 267.7.
Qwen3.5 and 3.6, which fit either way. The text tower through its class, two 8-GPU machines, max_length 8192, packed, m=8, moe_balancing: none, median of steps 4–13 on the same rows. Qwen3.5-35B-A3B with GC off: pp2 × ep8 attends 15,306 tok/s/GPU against 15,758 for ep8 + DP (0.97×), at 126.7 / 62.7 GiB against 148.7. With GC on: 12,202 against 12,994 (0.94×), at 59.5 / 39.1 GiB against 91.0. Qwen3.6-35B-A3B with GC off: 15,202 against 15,698 (0.97×), at 126.5 / 63.0 GiB against 149.0. PP costs 3 to 6% here and pays in memory, the far stage most. A one-step save merged with merge_pp_stages.py, took its tower back with reattach_vision_tower.py and loaded as the composite with no key missing or unexpected.
Two more text towers, on GC. pp2 × ep8 at m=8 on Qwen3.8-Flash-Next and Nemotron 3.5 Super VL runs out of memory in the first step with GC off, at about 260 of 267.7 GiB with eight full graphs live on a stage. With GC on, 13 steps ran at 16.89 and 13.69 s a step, 1,940 and 2,394 tok/s/GPU as logged by 1.2.0, which counted padding, at 115.7 / 78.2 and 100.9 / 96.8 GiB.
One 8-GPU machine, GC off, max_length 8192, 524k tokens (64 rows) per step, m=16 (a 1/17 bubble), median of steps 6–15. A single torchrun refuses an NVLink domain smaller than its node (nvlink_domain_size must be a multiple of gpus_per_node), so the pp2 rows run two torchrun agents in one container, --nnodes=2 --nproc_per_node=4 with --node_rank=0 on CUDA_VISIBLE_DEVICES=0,1,2,3 and --node_rank=1 on 4,5,6,7: each agent is a 4-GPU node to the toolkit, and each half is one stage. The traffic between the halves stays inside the host, over NCCL shared memory, so these rows price the schedule and the narrower groups, not an inter-node link. The dp8 and ep8 rows run one agent on the host's 8-GPU domain.
- Dense:
pp2 × dp4runs 8.5% belowdp8, with nothing to offset the bubble. - With EP:
pp2 × ep4runs 9.2% aboveep8: halving the dispatch group cuts the DeepEP cost by more than the bubble adds. Both gpt-oss rates still rise over that window, so read the gap as a median of one run each.
Packing
packing: true composes with PP and is the way to avoid full-window compute on short rows: PP pads
every batch to max_length. Each stage rebuilds the segment markers a model's conv / linear-attention layers
read (segment_markers_for: seq_idx for GatedDeltaNet, LFM-2 ShortConv and Nemotron-H's Mamba, cu_seq_lens for
the delta rule) from its microbatch's position_ids, as the packing collator builds them for a flattened row, so
Qwen3.5 / 3.6, Qwen3.8-Flash-Next and Nemotron-H keep each document of a packed row apart; GatedDeltaNet needs its segment-aware kernels for it,
as off PP. Packing is refused, whatever allow_cross_document_linear_attention says, where the layers read no
boundary at all (find_unmarked_sequence_mixer: KDA, Lightning Attention-2, Inkling's short convolutions), whose
state would carry from each document into the next. In the roster that is GLM-5 Next and Inkling, whose padded
rows hold one document each and still run; DeepSeek-V4 refuses multi-document rows on every path
(Document isolation).
reject_pipeline_packed_rows (halo/models/segment_markers.py) holds the rule; the collator factory
applies it when PP keeps the packed rows, and reject_pp_collator again to the trainer's final collator,
whatever built it.
Under PP the packing collator keeps its [B, L] rows instead of flattening to [1, B*L]: the
schedule splits dim 0 into microbatches, so the microbatch split is the flattening, and each
microbatch reaches the stage as [1, max_length]. One requirement, enforced at setup:
YAML
packing: true
per_device_train_batch_size: 4 # must equal the microbatch count
pipeline_microbatches: 4A microbatch carrying more than one packed row leaks attention across documents under FlashAttention,
which derives cu_seqlens from position_ids only at batch size 1 (~1.9 nats of logprob error); the
gate does not vary by backend (Document isolation).
The trailing pad region is inert: its positions restart at 0 every PAD_TAIL_SEGMENT_CHUNK (256) tokens,
so it forms a few segments of its own with ignored labels.
Each stage builds those markers once per microbatch from one host read of the row
(PipelineStageModule._segment_kwargs in halo/distributed/pipeline_parallel/stage.py), and on a flash backend
the varlen set with them: cu_seq_lens_q/k and the longest segment as a host-int max_length_q/k. The schedule
splits only batch-dim tensors, so the markers the packing collator builds for a flattened row cannot ride in with
the batch; without the varlen set every attention layer derives it from position_ids again, three host syncs
per layer and again in the recompute. A masked or one-document microbatch gets none, nor does a non-flash
backend on a family without markers.
Per-step metrics
Only the last stage computes the loss, while HF logs from global rank 0 on the first stage, so the
trainer broadcasts the last stage's per-step scalars down the chain (one fp32 vector per step,
PipelineLossAdapter.metrics_fn); every chain rank logs the same values.
Evaluation
Evaluation runs through the schedule on every PP trainer: eval_loss reports normally and every rank
logs the identical broadcast value. Without compute_metrics the schedule computes its own
microbatched loss; with it, one forward-only pass yields the last stage's merged output, from which
both the loss and the predictions derive. A forward-only drive skips the schedule's trailing gradient
reduction, so evaluation does not reshard the stage per batch.
compute_metrics is served where a pipeline can reproduce the trainer's prediction convention and
rejected at construction elsewhere:
per_device_eval_batch_size == per_device_train_batch_sizewhenever the run evaluates, since the P2P shapes are frozen. Partial final batches are row-padded with all-ignore rows contributing 0, then trimmed before scoring.- SFT's
preprocess_logits_for_metricsruns on the last stage before the broadcast, so it must be row-independent (argmax and top-k are); SFTcompute_metricswithout it is refused. - The evaluation gather is scoped to the data-parallel replicas, since every chain rank returns the same predictions.
- DPO and KTO require their precomputed reference columns in the eval dataset too.
Loading
Loading is stage-aware: each rank builds the whole model shell on the meta device, plans the
checkpoint against that key space, keeps only the plans for its stage's decoder layers — re-basing
layers.5 on disk to layers.1 on a stage starting at layer 4 — and streams exactly those tensors
from safetensors (load_pp_stage_model, halo/model_loading/parallel/pp.py, narrowing the
whole-model plan through StageKeyFilter). Peak
per-rank load memory is
Text
P_ne_layers / pp_size + embeddings + final norm + head (+ the EP expert slice / ep_size)Embeddings, the final norm and the head are materialized on every stage and dropped by
build_pipeline_stage right after. PP+EP composes as a filter: the PP layer filter narrows the EP
planner's output.
The structure gate runs on the meta shell before any weight is read and before the loader's first
cross-rank verdict, so an unregistered family, a refused one or a tied checkpoint fails every rank
alike. The coverage gate raises if any parameter or persistent buffer is left on the meta device, or
a re-based key names nothing on the sliced stage; both verdicts are stage-local, so they are joined
and raised on every rank. State the checkpoint does not carry (a score head on a base LM) follows
the EP lazy loader's random-init policy (EP → Model loading).
The stage loader reads configured FP32 masters (fp32_non_ep_params, fp32_router, fp32_experts)
at FP32 (Load precision). PP requires a safetensors
checkpoint; init_from_scratch, QLoRA loading and lowp_precision != bf16 are rejected. The stage
load runs on every rank at once; max_concurrent_loading does not apply
(Multi-Node → Model loading).
Checkpoints and resume
save_pp_checkpoint (halo/checkpoint/parallel/pp.py) writes one complete-tensor part set per stage
under the unsplit model's hub names plus a merged standard index, so a directory on a shared output
filesystem loads with plain from_pretrained and serves like an EP-gathered save. The layout, the
name revert and the index exchange: Checkpoints → What lands on disk.
The input embedding, with any norm it carries, lives on stage 0 alone, so no two stages write it. A stage holding
an external table (Qwen3.8-Flash-Next's n-gram tables) writes it beside its parts under the unsplit model's names,
in files under its own part prefix, and the stage-aware loader attaches it by those names, so a resume reads each
stage's table from its own files.
The writer set follows the output filesystem (DIST_OUTPUT_SHARED_FILESYSTEM, falling back to
the DIST_SHARED_FILESYSTEM umbrella):
Because the parts carry the unsplit model's names, the weights are topology-independent: a
complete directory resumes onto a different pp_size. The optimizer shards are stage-local:
their restore is gated on the topology fingerprint and pp_stage_partition (both in
optimizer_meta.pt), and any drift raises. PP+EP loads its expert weights only at construction, so
resume through the training scripts, which repoint the load at the checkpoint
(Resume by parallelism mode).
Every PP trainer's save, resume and export runs on GPU through the real trainer, on one host. SFT
resumes in tests/gpu/parallelism/pp/test_pp_resume.py (dense pp2), exports in
test_pp_sft_trainer_e2e.py, and does both with experts in test_pp_ep_resume.py (gpt-oss pp2 × ep2
and pp2 × etp2); on per-node storage its checkpoint is also merged with merge_pp_stages.py and resumed
from the merge, and the Qwen3-8B recipe saved, resumed, merged and loaded with from_pretrained on two
nodes (Verified topologies). The other trainers run in
test_pp_trainer_checkpoints.py, one row per trainer and two for offline GRPO (with and without the KL
reference that rides each checkpoint as reference_logps.pt); each resume there restores the optimizer
bit for bit and replays the unbroken run.
Per-node export
On per-node output storage each node's directory holds the full index but only the parts of the stages it ran, plus the wrapper parts, and only its own ranks' optimizer shards and RNG states. It resumes only the topology and rank placement that wrote it; a node asked for another stage's layers fails on every rank, naming the placement, and every after-training tool refuses the directory, naming its absent parts. To export, or to resume from one directory, copy each node's directory onto one host and merge:
Bash
python scripts/after_training/merge_pp_stages.py \
--input_dir /copies/node0/checkpoint-500 /copies/node1/checkpoint-500 \
--output_dir /models/mergedThe inputs must carry identical indexes that name stage parts, so an ordinary checkpoint is refused, and
one global_step, since two steps of a run share an index and an optimizer meta and only
trainer_state.json tells them apart. Each part, a stage's table files included, comes from the first input
holding it; every
input's optimizer shards, RNG and prefetch states come along with one optimizer_meta.pt, hard-linked
when the output is on their filesystem (so the optimizer state is not stored twice) and copied with
their mode otherwise, and the config, tokenizer, trainer state and scheduler come from the first input.
A part no input holds, metas or steps that differ, a node with no trainer state beside one that has it, a
per-rank file held at two sizes or a rank whose shard no input holds is refused before the first write; the
index lands last. The merged directory loads with
from_pretrained, uploads like a shared-filesystem save, and resumes exactly on the save's topology.
A linked file outlives its input's deletion, and a later training save into either directory replaces it
rather than writing through the link
(Checkpoints & Resume).
Constraints inherited from torch.distributed.pipelining
- Fixed shapes. Input shapes freeze on the first step; a later shape raises
PipeliningShapeError. The trainer pads every batch tomax_length. - Bare tensors only. Every stage returns a tensor. Per-example side tensors (advantages, reference log-probs, pool positions) ride the runner's dict-target side channel, split in lockstep with the rows.
- Microbatch invariance. Per-microbatch losses must sum to the full-batch loss, so a whole-batch denominator is passed as a step-level normalizer.
PipelineRunnerdrives torch's private FSDP2 and schedule members, so construction asserts their presence: a torch upgrade that renames one raises immediately, naming PP as needing re-validation.
Trainer support
PP is declare-to-enable per trainer (like SUPPORTS_CP): PP_REFUSAL is the mechanism a refusing
trainer's error names, None where PP is supported. An enabled trainer declares its
pipeline-loss contract as a PipelineLossAdapter (_pp_loss_adapter(),
halo/trainers/parallelism/pipeline_loss_adapter.py). _configure_pipeline_parallel calls
_pp_loss_adapter() once, before anything reads the adapter, so an adapter builder may seed the
per-step state its loss closes over.
The refusals are pure functions in halo/trainers/parallelism/pipeline_rules.py. The training-argument,
PEFT and multimodal gates run before the stage split, with each trainer's own
_reject_unsupported_pp_options (DPO and KTO: keyword model/args, no live ref_model, the
precompute-only reference, compute_metrics); the collator, pad-token and EP-coverage gates run after
super().__init__. The rank-local verdicts (image evidence, a stage with no EP layer) are joined
across ranks.
Which trainers enable PP: Trainer Compatibility. The enabled trainers' pipeline-loss contracts:
A refusing trainer's error names its PP_REFUSAL: embedding (the in-batch similarity matrix is the
loss), SDPG (a second, differently-shaped same-model forward), online and async GRPO (the weight sync
sends the full named_parameters() from one rank set, which no PP rank holds), teacher and self
distillation (a second forward per microbatch through a distinct frozen teacher).
Code map
halo/distributed/pipeline_parallel/: families.py (the family specs and the hyper-connection
stream forwards), model_rules.py (the structure and run-settings gates), partition.py (the split
and validate_partition), stage.py (build_pipeline_stage and the stage↔global naming),
stage_losses.py, groups.py (chain and stage groups) and runner.py (PipelineRunner over torch's
Schedule1F1B / ScheduleGPipe). The stage-aware loader is halo/model_loading/parallel/pp.py; the
trainer side is PipelineTrainerMixin (halo/trainers/parallelism/pipeline_trainer.py, inert at
pp_size == 1), pipeline_rules.py, pipeline_loss_adapter.py and the stage clip in
grad_clipping.py.
Source on GitHubUpdated 18 November 2026