Context Parallelism (CP)
CP splits long sequences across GPUs using Ulysses sequence parallelism with all-to-all communication in attention layers. Use it when a sequence is too long to fit in single-GPU activation memory. CP runs standalone or combined with EP (MoE models). Why sequence length, not parameter count, drives that memory: GPU Training Theory §8.
data_parallel_size = world_size / cp_size; gradient sync always spans all world_size ranks.
CP is not compatible with TP — both partition the same contiguous rank blocks, and TP has
already cut this rank's head count that Ulysses redistributes over.
How Ulysses works
Naive sequence chunking breaks cross-chunk attention. Ulysses instead redistributes tensors around
the attention kernel: each GPU holds S/cp_size tokens but all heads, one all-to-all carrying Q, K
and V swaps that to the full S tokens but H/cp_size heads for the attention compute, and a second
swaps the output back. The backward runs the same two in reverse.
Multi-head attention is independent across heads, which is what makes this exact.
Packing Q, K and V into one exchange takes two collectives per layer each way where one per tensor takes three. On 8× B300 (1.2.0, three rounds) that runs Qwen3-8B CP2 at s4096 4.4% and gpt-oss-20b ep8 × cp2 3.7% faster, and leaves long sequences flat (cp4 at s16384 −0.4%, ep8 × cp8 at s32768 −0.1%).


Implementation
UlyssesModel (halo/distributed/context_parallel/ulysses_model.py) is the entry point: the CP
loaders construct it over the loaded model and the trainer adopts its CPConfig. The all-to-all
(UlyssesAllToAll, all_to_all.py) is a custom autograd function that packs every tensor it carries
into one buffer, row by destination rank, and runs one all_to_all_single: sequence_to_heads scatters
the heads of Q, K and V and gathers the sequence (GQA's K/V and MLA's narrower V keep their own head
counts and head dims), heads_to_sequence swaps the output back, and the backward runs the opposite
swap on the gradients. The per-family attention wrappers in
families.py subclass UlyssesAttention (ulysses_attention.py), whose one forward path runs RoPE
(and any position-dependent scale) on the local chunk before the all-to-all, so no position gather,
then Flash Attention with native GQA in [B, S, H, D], with zero transposes. The MLA families
(GLM4 MoE Lite, Mistral4) share MLAUlyssesAttention: RoPE on the rope half of each head, and an
_attend that right-pads V to qk_head_dim where v_head_dim differs, runs flash, and crops back.
Loss computation
CP computes loss per chunk with two corrections:
- Boundary tokens — the causal label shift (
logits[:-1]predictslabels[1:]) loses one prediction per chunk. Non-final ranks use their last logit to predict the first token of the next chunk. - Globally-normalized sum loss — completion-only training distributes tokens unevenly across
chunks, so per-rank
reduction='mean'biases gradients. Each rank computesloss_i = cp_size * local_sum / global_tokenswithglobal_tokensthe all-reduced non-masked token count. After FSDP2's1/cp_sizegradient averaging every token contributes equally. The fp32local_sumruns over checkpointed chunks of the rank's logits (chunked_token_sum, as the pipeline's last stage scores them), so no fp32 copy of the chunk's logits plane is built.
That form is rank-varying and exists for the gradient path only. Under torch.no_grad() (the eval
loop) the wrapper returns the rank-uniform group value instead — the CE sum and the aux term are
all-reduced over the CP group — because HF's DP-scoped metric gather keeps a single CP sibling's
copy as eval_loss, and a chunk-partial copy would bias it by that chunk's share of loss tokens.
The MoE router aux loss takes no cp_size factor. Each rank's load_balancing_loss_func sees
only its chunk's router logits, so the term is chunk-local — the load and the mean router
probability of that chunk — and the FSDP average over CP ranks trains the mean of the per-chunk
terms at router_aux_loss_coef. That is not the whole-sequence term (a product of means does not
split across chunks); a cp_size factor would only scale it cp_size×.
The GRPO scoring seam calls forward_hidden_states on the CP wrapper, which returns this rank's
backbone hidden states without constructing vocabulary-wide logits. cp_shift_against_full_labels
pairs them with the full row's next-token labels, and the vocab-chunked output head scores those
targets. Offline GRPO's per-row token-objective sums (and SMPO's per-sequence sums) reduce through
cp_sum_rows in halo/distributed/context_parallel/sequence_chunks.py, an autograd-aware SUM whose
backward supplies the CP factor canceled by world-wide mean gradient synchronization. They must
not also use the SFT loss's explicit cp_size multiplier. Offline GRPO uses this path for
training, evaluation and reference scoring; online and environmental GRPO do not declare CP. Its
reference and resume contract is in
Offline GRPO.
Fused head loss
Without it, each rank scores its chunk from the model's bf16 logits plane, cast to fp32 one checkpointed chunk at a
time (Loss computation), so the plane and its gradient stay in bf16. When the run
resolves Liger's fused_linear_cross_entropy on for the family (requested in liger_kernel_config, or the family
default: GLM-4.7-Flash), a training forward instead keeps one position's logits (logits_to_keep=1), takes the
backbone's final hidden states, and scores them through Liger's fused linear cross-entropy under the family's
verified head transform (fused_head_ce_sum in halo/model_loading/liger/cross_entropy.py: hidden and logit scale
folded onto the hidden states, softcap in the kernel, vocabulary cut on the weight). No logits plane is built. The
weight gradient accumulates in fp32. The boundary pairing, the token normalization and the aux term are the
logits loss's. The forward returns logits=None and the chunk's token_accuracy, so SFT logs accuracy but no
entropy, as FLCE does without CP. On GLM-4.7-Flash at ep8 × cp4 and 65,536 tokens (1.2.0) the peak falls from 53.30 to
46.99 GiB (−11.8%), step-1 loss within 7e-8 relative.
Eval forwards keep the logits loss. A training forward without labels returns the full logits, with one warning:
only a labelled forward (SFT) runs the fused loss, and SMPO scores its pairs from those logits. A forward that cannot
take the fused path keeps the logits loss, with one warning:
no logits_to_keep parameter (Bailing's remote code), a PEFT tuner or a non-nn.Linear module on the output
embedding, a head path the transform probe cannot verify, or a logit scale over a biased head. The model's own head
stays unpatched under CP, since the wrapper never hands it labels.
A model that returns an aux_loss while its config declares no router_aux_loss_coef raises — a
stand-in weight would train a different objective than the same config without CP. Set the field
through model_init_kwargs, or turn the aux loss off for that family with moe_balancing.
Requirements
NVLink-local topology — cp_size must divide the NVLink domain (gpus_per_node on a standard
node, the rack on NVL72) and cannot exceed it; Ulysses' all-to-alls per attention layer must stay
on NVLink. cp_size=3 on an 8-GPU domain is rejected at config time, before the model loads
(_reject_cp_locality).
Attention impl — a real Flash Attention is required. reject_unsupported_ulysses_model raises
CPConfigError for any _attn_implementation outside SUPPORTED_ATTN_IMPLEMENTATIONS (FA2,
FA3, FA4): eager, sdpa and a hub kernel label are rejected, and flex_attention is
auto-switched to the arch's flash kernel (FA4 on Blackwell, FA3 on Hopper, else FA2;
Flash Attention → Context Parallelism) by
load_distributed_model when CP is active. A wrapper declaring
REQUIRES_FLASH_ATTN_LABEL = False waives the check (Bailing; see
bailing.md).
That check reads the declared implementation. The kernel CP calls is arch-matched, with FA4 refused where
supports_fa4_backward refuses it (GLM-4 MoE Lite's latent attention in every run, as a precaution,
head_dim-256 attention in a deterministic one,
Flash Attention → Supported backends), and
under CP the loader labels such a model with that kernel (FA2 on Blackwell) where a deterministic run
would otherwise fall back to SDPA, so the auto-detected default passes the check. Leave
attn_implementation at it.
The Ulysses call hands that kernel the deterministic flag transformers passes its own flash calls
(FLASH_ATTENTION_DETERMINISTIC, which full_determinism sets before the load), so a deterministic run
gets the kernel's deterministic backward under CP too.
Head divisibility — cp_size must divide both the Q and the KV head count. GPT-OSS (64 Q, 8 KV):
CP=8 → 8 Q / 1 KV; CP=4 → 16 Q / 2 KV; CP=3 rejected.
These model checks, the sequence-axis refusals below included, run on a meta-device shell of the model
before any weight is read (reject_unsupported_structure), and again on the loaded model at the CP wrap.
Sequence length must be a multiple of cp_size. scripts/training/sft.py sets pad_to_multiple_of=cp_size
and the collator factory routes to a padding collator that rounds each batch up — no manual collator
construction. A ragged batch that reaches SFT's compute_loss anyway is right-padded there, which needs
the tokenizer's pad_token_id: a processing_class without one raises rather than padding with
vocabulary token 0.
Right padding only. The Ulysses path ignores attention_mask: it calls dense flash attention
with causal= and has no varlen path, so it tolerates only padding a causal mask already ignores.
A left-padded batch is rejected: every real token would attend the leading pads and the loss would
silently differ from the same batch without CP. Every loader of a CP trainer collates through
LeftPaddingGuard, which checks each batch's attention_mask on the host before it moves to the
device, so the CP forward reads nothing back from the GPU to check it. The wrapped model receives the
chunk's mask only where its router aux loss reads it (output_router_logits, which keeps padding out of
the aux statistics); elsewhere it gets none, and transformers' flash mask construction reads nothing back.
SMPO's collator left-pads prompts and emits no attention_mask, so the guard has nothing to check;
its chosen|rejected concat, which builds the forward's rows on the device, flushes every row left,
so every pad trails the completion and any batch size runs.
Offline GRPO's CP collator right-pads each joined prompt+completion row (Offline GRPO).
Supported model architectures
Source of truth: ULYSSES_ATTENTION_MAP in halo/distributed/context_parallel/families.py, built
by walking the UlyssesAttention subclass tree and reading each wrapper's HF_MODULE_NAMES (a
duplicate HF name raises). The validator and the patcher both read that map — there is no separate
accept list to keep in sync.
Every other attention class raises CPConfigError, and a model carrying state along the sequence
axis (linear attention, sequence-axis convolutions, window pooling) is refused by class name even
where its attention is wrapped — Qwen3.5/3.6 on every released checkpoint. Per family, with the reason
on its page: Compatibility matrix.
CP with EP
EP is orthogonal to DP; CP reduces it. So data_parallel_size = world_size / cp_size whether EP is
on or off. CP does its all-to-all in attention and EP its DeepEP dispatch/combine in the MoE
block — different points in the forward pass, no conflict.
Orthogonal mode (cp_size == world_size): one CP group over all ranks, DP=1. Partial mode:
world_size / cp_size CP groups, each on a different batch.
CP always requires gradient sync when world_size > 1, even at DP=1 — each rank holds partial
gradients from its chunk. DistributedTrainerMixin applies FSDP2 across all ranks; for EP+CP the
same wrapper runs with EP params in ignored_params.
EP+CP needs node-local EP filling the NVLink domain: on 8 GPUs ep8/cp2 → DP4, ep8/cp4 → DP2, ep8/cp8 → DP1 (Topology rules).
Pure CP on a MoE still gets EP layers. load_cp_model builds an ep_size == 1 EP
config whenever needs_ep_layers holds and the model declares experts, so the
MoE blocks run the grouped-GEMM expert path rather than the stock
per-expert loop — needs_ep_layers forces Liger's routed-expert swiglu/geglu off either way, so
without it the model would pay that cost for no speedup. Dense models under CP get no EP config.
Usage
Bash
# CP-only (any supported model, long sequences)
torchrun --nproc_per_node=4 scripts/training/sft.py \
examples/sft/qwen3/qwen3-4b-ultrachat.yaml --context_parallel_size=4 --packing=false
# EP+CP (MoE, node-local EP with ep_group_size equal to the NVLink domain)
torchrun --nproc_per_node=8 scripts/training/sft.py \
examples/sft/gptoss/gptoss-20b-multinode-ep.yaml \
--expert_parallel_size=8 --ep_scope=node --use_grouped_gemm=true --context_parallel_size=2 --packing=falseBoth example configs set packing: true (refused under CP), and the gpt-oss one sets
ep_scope: global (refused under CP), so the commands override them. The gpt-oss one also pins the per-expert loop for
its ep16, so its command turns grouped GEMM back on.
Programmatic: parallelism_config=ParallelismConfig(ep_size=8, cp_size=8).
Checkpoint saving
save_cp_checkpoint serves dense CP runs only. An MoE carries EP layers (grouped GEMM is
on by default), and the saver ladder checks for EP layers first, so EP+CP and even ep_size=1
MoE take the EP save — which strips the .original_attention. prefix itself.
trainer.save_model(output_dir) gathers on every rank and the FS-aware save rank writes.
_find_cp_wrapper() finds the wrapper on the compile-unwrapped model, directly or under PEFT. The
wrapper's state_dict() remaps .original_attention. keys to standard paths and drops the
duplicate dense .mlp.{gate,up,down}_proj keys only on layers carrying routed experts — genuinely
dense layers keep their dense MLP weights.
On resume CP is a Path B mode: the trainer skips the checkpoint weight-reload (the CP wrapper
changes the module tree). The training scripts repoint model_name_or_path at the checkpoint so
load_distributed_model loads the trained weights at construction; a model not constructed from
the checkpoint raises rather than silently continuing on its current weights.
Configured FP32 masters are restored before the CP wrapper is installed (Load precision).
Trainer state is restored, and so are LoRA adapters: the saved CP-normalized keys are remapped back onto the live wrapped names, and a wholesale key miss raises.
load_best_model_at_end is refused at construction for CP full fine-tunes (base weights only load
at construction, so the best checkpoint cannot be reloaded in place). See
Checkpoints & Resume.
Optimizations
Native GQA holds K and V at their own head count (no repeat_kv), and the [B, S, H, D] layout drops
the transposes. The all-to-alls run in native BFloat16, with Q, K and V in one buffer.
torch.compile traces the exchange without a graph break, though the reentrant checkpointing CP
forces still breaks the graph; its throughput effect under CP is not measured. Use Liger (default on).
Limitations
Trainers. CP is declare-to-enable (SUPPORTS_CP, default False). Offline GRPO
under CP requires full fine-tuning: it rejects PEFT and native expert LoRA at any kl_beta, and an
explicit ref_model, since it scores the run-start reference through the distributed policy
(reject_offline_cp_options). A trainer that does not declare CP raises at construction, and its entry script
rejects --context_parallel_size > 1 earlier through
parallelism_config_from_args(..., trainer_cls=...), which reads the same flag. Full matrix:
Trainer Compatibility. Nothing
inspects a trainer's loss for CP-safety, so a new trainer must verify its own objective before
declaring the flag.
Models. Only the wrappers in the table above; anything else
raises CPConfigError. Linear-attention, sequence-axis convolution and window-pooling blocks
are rejected by class name even when the surrounding attention is wrapped.
Generation. UlyssesModel.generate() raises: each rank holds one sequence chunk and the
Ulysses attention has no KV-cache path. Generate from a saved checkpoint without CP.
Axis combinations. CP composes with EP only (allowlist). EP alone OOMs at long sequences (it shards experts, not activations) — that is the case CP exists for.
Knobs. Everything below raises unless the verdict says otherwise.
Adding a new model
Subclass
UlyssesAttentioninfamilies.py. The default_project_qkvreads plainq_proj/k_proj/v_projinto[B, S, H, D], the default_apply_rotaryis rotate-half (apply_rope) and the output projection isOUTPUT_PROJECTION = "o_proj"; override only what the family does differently (GptOss split-halves rotary, Cohere2 interleaved fp32, Bailing's fused projection anddense).For an MLA family subclass
MLAUlyssesAttentioninstead: it owns the geometry, the rope-half_apply_rotaryand the V-padding_attend.Declare the HF attention class name in the wrapper's
HF_MODULE_NAMES.ULYSSES_ATTENTION_MAPis read off the subclass tree, so defining the class is the whole registration.Test:
Bash
torchrun --nproc_per_node=2 \
tests/gpu/parallelism/cp/test_cp_correctness.py
# EP+CP, if MoE (2-GPU minimal repro: EP=CP=world=2):
torchrun --nproc_per_node=2 \
tests/gpu/parallelism/combined/test_ep_cp_correctness.pyFurther CP tests live in tests/gpu/parallelism/cp/. Add the family to FAMILIES in
tests/common/equivalence/cp_equivalence.py and set ulysses_cp=True on its TinyFamily:
tests/cpu/conventions/test_tiny_family_roster.py fails on a wrapper no roster family builds.
Background: DeepSpeed-Ulysses.
Source on GitHubUpdated 18 November 2026