Context Parallelism (CP)

CP splits long sequences across GPUs using Ulysses sequence parallelism with all-to-all communication in attention layers. Use it when a sequence is too long to fit in single-GPU activation memory. CP runs standalone or combined with EP (MoE models). Why sequence length, not parameter count, drives that memory: GPU Training Theory §8.

data_parallel_size = world_size / cp_size; gradient sync always spans all world_size ranks. CP is not compatible with TP — both partition the same contiguous rank blocks, and TP has already cut this rank's head count that Ulysses redistributes over.

How Ulysses works

Naive sequence chunking breaks cross-chunk attention. Ulysses instead redistributes tensors around the attention kernel: each GPU holds S/cp_size tokens but all heads, one all-to-all carrying Q, K and V swaps that to the full S tokens but H/cp_size heads for the attention compute, and a second swaps the output back. The backward runs the same two in reverse. Multi-head attention is independent across heads, which is what makes this exact.

Packing Q, K and V into one exchange takes two collectives per layer each way where one per tensor takes three. On 8× B300 (1.2.0, three rounds) that runs Qwen3-8B CP2 at s4096 4.4% and gpt-oss-20b ep8 × cp2 3.7% faster, and leaves long sequences flat (cp4 at s16384 −0.4%, ep8 × cp8 at s32768 −0.1%).

Ulysses attention: RoPE runs on the local chunk, one all-to-all on Q, K and V scatters heads and gathers the sequence so each rank attends over the full sequence with 16 of 64 heads, and an all-to-all on the output restores the per-rank chunk layout
Ulysses attention: RoPE runs on the local chunk, one all-to-all on Q, K and V scatters heads and gathers the sequence so each rank attends over the full sequence with 16 of 64 heads, and an all-to-all on the output restores the per-rank chunk layout
An SFT training step under CP: the wrapper narrows the batch to one contiguous sequence chunk per rank, embeddings and every decoder layer run on that chunk (Ulysses attention, then the MLP or MoE), the head scores the chunk from sharded logits or fused with the loss with no logits plane, and the loss is scaled by cp_size over the CP group's all-reduced token count
An SFT training step under CP: the wrapper narrows the batch to one contiguous sequence chunk per rank, embeddings and every decoder layer run on that chunk (Ulysses attention, then the MLP or MoE), the head scores the chunk from sharded logits or fused with the loss with no logits plane, and the loss is scaled by cp_size over the CP group's all-reduced token count

Implementation

UlyssesModel (halo/distributed/context_parallel/ulysses_model.py) is the entry point: the CP loaders construct it over the loaded model and the trainer adopts its CPConfig. The all-to-all (UlyssesAllToAll, all_to_all.py) is a custom autograd function that packs every tensor it carries into one buffer, row by destination rank, and runs one all_to_all_single: sequence_to_heads scatters the heads of Q, K and V and gathers the sequence (GQA's K/V and MLA's narrower V keep their own head counts and head dims), heads_to_sequence swaps the output back, and the backward runs the opposite swap on the gradients. The per-family attention wrappers in families.py subclass UlyssesAttention (ulysses_attention.py), whose one forward path runs RoPE (and any position-dependent scale) on the local chunk before the all-to-all, so no position gather, then Flash Attention with native GQA in [B, S, H, D], with zero transposes. The MLA families (GLM4 MoE Lite, Mistral4) share MLAUlyssesAttention: RoPE on the rope half of each head, and an _attend that right-pads V to qk_head_dim where v_head_dim differs, runs flash, and crops back.

Loss computation

CP computes loss per chunk with two corrections:

  1. Boundary tokens — the causal label shift (logits[:-1] predicts labels[1:]) loses one prediction per chunk. Non-final ranks use their last logit to predict the first token of the next chunk.
  2. Globally-normalized sum loss — completion-only training distributes tokens unevenly across chunks, so per-rank reduction='mean' biases gradients. Each rank computes loss_i = cp_size * local_sum / global_tokens with global_tokens the all-reduced non-masked token count. After FSDP2's 1/cp_size gradient averaging every token contributes equally. The fp32 local_sum runs over checkpointed chunks of the rank's logits (chunked_token_sum, as the pipeline's last stage scores them), so no fp32 copy of the chunk's logits plane is built.

That form is rank-varying and exists for the gradient path only. Under torch.no_grad() (the eval loop) the wrapper returns the rank-uniform group value instead — the CE sum and the aux term are all-reduced over the CP group — because HF's DP-scoped metric gather keeps a single CP sibling's copy as eval_loss, and a chunk-partial copy would bias it by that chunk's share of loss tokens.

The MoE router aux loss takes no cp_size factor. Each rank's load_balancing_loss_func sees only its chunk's router logits, so the term is chunk-local — the load and the mean router probability of that chunk — and the FSDP average over CP ranks trains the mean of the per-chunk terms at router_aux_loss_coef. That is not the whole-sequence term (a product of means does not split across chunks); a cp_size factor would only scale it cp_size×.

The GRPO scoring seam calls forward_hidden_states on the CP wrapper, which returns this rank's backbone hidden states without constructing vocabulary-wide logits. cp_shift_against_full_labels pairs them with the full row's next-token labels, and the vocab-chunked output head scores those targets. Offline GRPO's per-row token-objective sums (and SMPO's per-sequence sums) reduce through cp_sum_rows in halo/distributed/context_parallel/sequence_chunks.py, an autograd-aware SUM whose backward supplies the CP factor canceled by world-wide mean gradient synchronization. They must not also use the SFT loss's explicit cp_size multiplier. Offline GRPO uses this path for training, evaluation and reference scoring; online and environmental GRPO do not declare CP. Its reference and resume contract is in Offline GRPO.

Fused head loss

Without it, each rank scores its chunk from the model's bf16 logits plane, cast to fp32 one checkpointed chunk at a time (Loss computation), so the plane and its gradient stay in bf16. When the run resolves Liger's fused_linear_cross_entropy on for the family (requested in liger_kernel_config, or the family default: GLM-4.7-Flash), a training forward instead keeps one position's logits (logits_to_keep=1), takes the backbone's final hidden states, and scores them through Liger's fused linear cross-entropy under the family's verified head transform (fused_head_ce_sum in halo/model_loading/liger/cross_entropy.py: hidden and logit scale folded onto the hidden states, softcap in the kernel, vocabulary cut on the weight). No logits plane is built. The weight gradient accumulates in fp32. The boundary pairing, the token normalization and the aux term are the logits loss's. The forward returns logits=None and the chunk's token_accuracy, so SFT logs accuracy but no entropy, as FLCE does without CP. On GLM-4.7-Flash at ep8 × cp4 and 65,536 tokens (1.2.0) the peak falls from 53.30 to 46.99 GiB (−11.8%), step-1 loss within 7e-8 relative.

Eval forwards keep the logits loss. A training forward without labels returns the full logits, with one warning: only a labelled forward (SFT) runs the fused loss, and SMPO scores its pairs from those logits. A forward that cannot take the fused path keeps the logits loss, with one warning: no logits_to_keep parameter (Bailing's remote code), a PEFT tuner or a non-nn.Linear module on the output embedding, a head path the transform probe cannot verify, or a logit scale over a biased head. The model's own head stays unpatched under CP, since the wrapper never hands it labels.

A model that returns an aux_loss while its config declares no router_aux_loss_coef raises — a stand-in weight would train a different objective than the same config without CP. Set the field through model_init_kwargs, or turn the aux loss off for that family with moe_balancing.

Requirements

NVLink-local topology — cp_size must divide the NVLink domain (gpus_per_node on a standard node, the rack on NVL72) and cannot exceed it; Ulysses' all-to-alls per attention layer must stay on NVLink. cp_size=3 on an 8-GPU domain is rejected at config time, before the model loads (_reject_cp_locality).

Attention impl — a real Flash Attention is required. reject_unsupported_ulysses_model raises CPConfigError for any _attn_implementation outside SUPPORTED_ATTN_IMPLEMENTATIONS (FA2, FA3, FA4): eager, sdpa and a hub kernel label are rejected, and flex_attention is auto-switched to the arch's flash kernel (FA4 on Blackwell, FA3 on Hopper, else FA2; Flash Attention → Context Parallelism) by load_distributed_model when CP is active. A wrapper declaring REQUIRES_FLASH_ATTN_LABEL = False waives the check (Bailing; see bailing.md).

That check reads the declared implementation. The kernel CP calls is arch-matched, with FA4 refused where supports_fa4_backward refuses it (GLM-4 MoE Lite's latent attention in every run, as a precaution, head_dim-256 attention in a deterministic one, Flash Attention → Supported backends), and under CP the loader labels such a model with that kernel (FA2 on Blackwell) where a deterministic run would otherwise fall back to SDPA, so the auto-detected default passes the check. Leave attn_implementation at it.

The Ulysses call hands that kernel the deterministic flag transformers passes its own flash calls (FLASH_ATTENTION_DETERMINISTIC, which full_determinism sets before the load), so a deterministic run gets the kernel's deterministic backward under CP too.

Head divisibility — cp_size must divide both the Q and the KV head count. GPT-OSS (64 Q, 8 KV): CP=8 → 8 Q / 1 KV; CP=4 → 16 Q / 2 KV; CP=3 rejected.

These model checks, the sequence-axis refusals below included, run on a meta-device shell of the model before any weight is read (reject_unsupported_structure), and again on the loaded model at the CP wrap.

Sequence length must be a multiple of cp_size. scripts/training/sft.py sets pad_to_multiple_of=cp_size and the collator factory routes to a padding collator that rounds each batch up — no manual collator construction. A ragged batch that reaches SFT's compute_loss anyway is right-padded there, which needs the tokenizer's pad_token_id: a processing_class without one raises rather than padding with vocabulary token 0.

Right padding only. The Ulysses path ignores attention_mask: it calls dense flash attention with causal= and has no varlen path, so it tolerates only padding a causal mask already ignores. A left-padded batch is rejected: every real token would attend the leading pads and the loss would silently differ from the same batch without CP. Every loader of a CP trainer collates through LeftPaddingGuard, which checks each batch's attention_mask on the host before it moves to the device, so the CP forward reads nothing back from the GPU to check it. The wrapped model receives the chunk's mask only where its router aux loss reads it (output_router_logits, which keeps padding out of the aux statistics); elsewhere it gets none, and transformers' flash mask construction reads nothing back.

SMPO's collator left-pads prompts and emits no attention_mask, so the guard has nothing to check; its chosen|rejected concat, which builds the forward's rows on the device, flushes every row left, so every pad trails the completion and any batch size runs.

Offline GRPO's CP collator right-pads each joined prompt+completion row (Offline GRPO).

Supported model architectures

Source of truth: ULYSSES_ATTENTION_MAP in halo/distributed/context_parallel/families.py, built by walking the UlyssesAttention subclass tree and reading each wrapper's HF_MODULE_NAMES (a duplicate HF name raises). The validator and the patcher both read that map — there is no separate accept list to keep in sync.

Attention classWrapperFamily specificsPage
GptOssAttentionGptOssUlyssesAttentionsplit-halves RoPE; sinks must be resetgpt-oss.md
Qwen3MoeAttention (Qwen3 MoE)Qwen3MoeUlyssesAttentionQ/K RMSNormqwen3.md
Qwen3Attention (Qwen3 dense)Qwen3MoeUlyssesAttention (reused)Q/K RMSNormqwen3.md
Qwen3VLTextAttention (Qwen3-VL text branch)Qwen3MoeUlyssesAttention (reused)Q/K RMSNormqwen3.md
Glm4MoeLiteAttentionGlm4MoeLiteUlyssesAttentionDeepSeek-V3 MLA: separate qk/v head dimsglm4.md
Mistral4AttentionMistral4UlyssesAttentionMLA: separate qk/v head dims, llama-4 scalingmistral4.md
Qwen3_5MoeAttention / Qwen3_5AttentionQwen3_5MoeUlyssesAttentionpost-attention sigmoid gate + partial RoPEqwen3_5.md
BailingMoeV2Attention / …FlashAttention2 / …SdpaAttention (Ling 2.0)BailingMoeV2UlyssesAttentionfused query_key_value, dense output projection, Q/K norm before partial RoPEbailing.md
Cohere2MoeAttentionCohere2MoeUlyssesAttentioninterleaved fp32 rotary on sliding layers only; full-attention layers are NoPEcohere2-moe.md

Every other attention class raises CPConfigError, and a model carrying state along the sequence axis (linear attention, sequence-axis convolutions, window pooling) is refused by class name even where its attention is wrapped — Qwen3.5/3.6 on every released checkpoint. Per family, with the reason on its page: Compatibility matrix.

CP with EP

EP is orthogonal to DP; CP reduces it. So data_parallel_size = world_size / cp_size whether EP is on or off. CP does its all-to-all in attention and EP its DeepEP dispatch/combine in the MoE block — different points in the forward pass, no conflict.

Orthogonal mode (cp_size == world_size): one CP group over all ranks, DP=1. Partial mode: world_size / cp_size CP groups, each on a different batch.

CP always requires gradient sync when world_size > 1, even at DP=1 — each rank holds partial gradients from its chunk. DistributedTrainerMixin applies FSDP2 across all ranks; for EP+CP the same wrapper runs with EP params in ignored_params.

EP+CP needs node-local EP filling the NVLink domain: on 8 GPUs ep8/cp2 → DP4, ep8/cp4 → DP2, ep8/cp8 → DP1 (Topology rules).

Pure CP on a MoE still gets EP layers. load_cp_model builds an ep_size == 1 EP config whenever needs_ep_layers holds and the model declares experts, so the MoE blocks run the grouped-GEMM expert path rather than the stock per-expert loop — needs_ep_layers forces Liger's routed-expert swiglu/geglu off either way, so without it the model would pay that cost for no speedup. Dense models under CP get no EP config.

Usage

Bash

# CP-only (any supported model, long sequences)
torchrun --nproc_per_node=4 scripts/training/sft.py \
    examples/sft/qwen3/qwen3-4b-ultrachat.yaml --context_parallel_size=4 --packing=false

# EP+CP (MoE, node-local EP with ep_group_size equal to the NVLink domain)
torchrun --nproc_per_node=8 scripts/training/sft.py \
    examples/sft/gptoss/gptoss-20b-multinode-ep.yaml \
    --expert_parallel_size=8 --ep_scope=node --use_grouped_gemm=true --context_parallel_size=2 --packing=false

Both example configs set packing: true (refused under CP), and the gpt-oss one sets ep_scope: global (refused under CP), so the commands override them. The gpt-oss one also pins the per-expert loop for its ep16, so its command turns grouped GEMM back on.

Programmatic: parallelism_config=ParallelismConfig(ep_size=8, cp_size=8).

Checkpoint saving

save_cp_checkpoint serves dense CP runs only. An MoE carries EP layers (grouped GEMM is on by default), and the saver ladder checks for EP layers first, so EP+CP and even ep_size=1 MoE take the EP save — which strips the .original_attention. prefix itself.

trainer.save_model(output_dir) gathers on every rank and the FS-aware save rank writes. _find_cp_wrapper() finds the wrapper on the compile-unwrapped model, directly or under PEFT. The wrapper's state_dict() remaps .original_attention. keys to standard paths and drops the duplicate dense .mlp.{gate,up,down}_proj keys only on layers carrying routed experts — genuinely dense layers keep their dense MLP weights.

On resume CP is a Path B mode: the trainer skips the checkpoint weight-reload (the CP wrapper changes the module tree). The training scripts repoint model_name_or_path at the checkpoint so load_distributed_model loads the trained weights at construction; a model not constructed from the checkpoint raises rather than silently continuing on its current weights.

Configured FP32 masters are restored before the CP wrapper is installed (Load precision).

Trainer state is restored, and so are LoRA adapters: the saved CP-normalized keys are remapped back onto the live wrapped names, and a wholesale key miss raises.

load_best_model_at_end is refused at construction for CP full fine-tunes (base weights only load at construction, so the best checkpoint cannot be reloaded in place). See Checkpoints & Resume.

Optimizations

Native GQA holds K and V at their own head count (no repeat_kv), and the [B, S, H, D] layout drops the transposes. The all-to-alls run in native BFloat16, with Q, K and V in one buffer.

torch.compile traces the exchange without a graph break, though the reentrant checkpointing CP forces still breaks the graph; its throughput effect under CP is not measured. Use Liger (default on).

Limitations

Trainers. CP is declare-to-enable (SUPPORTS_CP, default False). Offline GRPO under CP requires full fine-tuning: it rejects PEFT and native expert LoRA at any kl_beta, and an explicit ref_model, since it scores the run-start reference through the distributed policy (reject_offline_cp_options). A trainer that does not declare CP raises at construction, and its entry script rejects --context_parallel_size > 1 earlier through parallelism_config_from_args(..., trainer_cls=...), which reads the same flag. Full matrix: Trainer Compatibility. Nothing inspects a trainer's loss for CP-safety, so a new trainer must verify its own objective before declaring the flag.

Models. Only the wrappers in the table above; anything else raises CPConfigError. Linear-attention, sequence-axis convolution and window-pooling blocks are rejected by class name even when the surrounding attention is wrapped.

Generation. UlyssesModel.generate() raises: each rank holds one sequence chunk and the Ulysses attention has no KV-cache path. Generate from a saved checkpoint without CP.

Axis combinations. CP composes with EP only (allowlist). EP alone OOMs at long sequences (it shards experts, not activations) — that is the case CP exists for.

Knobs. Everything below raises unless the verdict says otherwise.

KnobUnder CPGate
packing, padding_freerejected — the Ulysses path runs a dense causal kernel with no per-document boundarieshalo/data/collators/factory.py; reject_cp_packed_collator re-checks a hand-built collator
left-padded batchesrejected on every batch, on the host at collation (not cached — a prompt-left-padding collator pads only its ragged batches)LeftPaddingGuard in context_parallel/sequence_chunks.py, applied by DataParallelLoaderMixin._loader_collate_fn
attn_implementationmust resolve to FA2/FA3/FA4; flex_attention is auto-switched with a warning, eager/sdpa rejected — except for a wrapper declaring REQUIRES_FLASH_ATTN_LABEL = False (Bailing), which runs on any labelSUPPORTED_ATTN_IMPLEMENTATIONS, reject_unsupported_ulysses_model
init_from_scratchrejected — no sharded random initmodel_loading/parallel/load_rules.py
reset_sinks: false on GPT-OSSrejected at model load — the CP attention kernels drop the sink column, so live sinks would misnormalize the softmax in every layerreject_unsupported_model, re-checked in GptOssUlyssesAttention
label_smoothing_factor > 0, loss_type: dftrejected — the Trainer pops labels and pairs full labels with this rank's chunk logitsreject_cp_trainer_args
compute_metrics, preprocess_logits_for_metricsrejected whenever set, not gated on eval_strategy (a direct evaluate()/predict() reaches the same misaligned chunk-logits path) — eval under CP is loss-onlysame
SFT's metrics (mean_token_accuracy, entropy, aux_loss, num_attended_tokens_seen)scored on the rank's chunk against the full labels, accumulated locally per micro-batch and reduced once per log, as in every mode (SFT loss and metrics); under the fused head loss, accuracy from the chunk's token_accuracy weighed by its supervised targets, and no entropyhalo/trainers/sft.py (_record_loss_metrics)
multimodal inputs (pixel_values)rejected — text-onlycontext_parallel/ulysses_model.py
fsdp_shard_ep1_experts: falserejected — the CP path shards ep1 experts unconditionally, so the flag would be a silent no-op_reject_fsdp_settings
accelerate launchrejected — CP requires torchrunreject_unsupported_run and reject_custom_parallelism_under_accelerate
gradient_checkpointingsupported; use_reentrant is forced to True (warned when the config sets false; rule)DistributedTrainerMixin._apply_training_args_overrides
use_liger_kernelsupported; the model's own cross_entropy and fused_linear_cross_entropy patches are off (an explicit cross_entropy is warned), and a resolved fused_linear_cross_entropy runs in the CP wrapper (fused head loss)halo/model_loading/liger/dispatch.py, context_parallel/ulysses_model.py
use_peft / LoRAsupported for SFT and SMPO — CP leaves attention unsharded, so adapters stay replicated; offline GRPO rejects adaptersreject_offline_cp_options (offline GRPO)
QLoRA (load_in_4bit)supported for SFT and SMPO on a dense model — CP keeps the standard loader and preserves Params4bit. On an MoE the grouped-GEMM loader takes over and rejects a quantized base (use_grouped_gemm is on by default); offline GRPO rejects adaptersmodel_loading/parallel/load_rules.py, reject_offline_cp_options
use_hsdp, fsdp_reshard_after_forwardsupported — CP is one of HSDP's two accepted paths (pure DP, CP) and one of ZeRO-3's EP-free shapes_reject_hsdp, _reject_fsdp_settings
torch_compilenot gated; the exchange traces without a graph break, throughput effect not measured—
save_sharded_eprejected on a CP run — per-rank shards carry .original_attention. keys the merge script cannot remapreject_unmergeable_ep_sharded_save

Adding a new model

  1. Subclass UlyssesAttention in families.py. The default _project_qkv reads plain q_proj/k_proj/v_proj into [B, S, H, D], the default _apply_rotary is rotate-half (apply_rope) and the output projection is OUTPUT_PROJECTION = "o_proj"; override only what the family does differently (GptOss split-halves rotary, Cohere2 interleaved fp32, Bailing's fused projection and dense).

    For an MLA family subclass MLAUlyssesAttention instead: it owns the geometry, the rope-half _apply_rotary and the V-padding _attend.

  2. Declare the HF attention class name in the wrapper's HF_MODULE_NAMES. ULYSSES_ATTENTION_MAP is read off the subclass tree, so defining the class is the whole registration.

  3. Test:

Bash

torchrun --nproc_per_node=2 \
    tests/gpu/parallelism/cp/test_cp_correctness.py
# EP+CP, if MoE (2-GPU minimal repro: EP=CP=world=2):
torchrun --nproc_per_node=2 \
    tests/gpu/parallelism/combined/test_ep_cp_correctness.py

Further CP tests live in tests/gpu/parallelism/cp/. Add the family to FAMILIES in tests/common/equivalence/cp_equivalence.py and set ulysses_cp=True on its TinyFamily: tests/cpu/conventions/test_tiny_family_roster.py fails on a wrapper no roster family builds.

Background: DeepSpeed-Ulysses.

Source on GitHubUpdated 18 November 2026