# Context Parallelism (CP)

CP splits long sequences across GPUs using Ulysses sequence parallelism with all-to-all
communication in attention layers. Use it when a sequence is too long to fit in single-GPU
activation memory. CP runs standalone or combined with EP (MoE models). Why sequence length, not
parameter count, drives that memory:
[GPU Training Theory §8](https://whitecircle.com/halo/docs/agent-docs/reference/gpu-training-theory.md#activations-the-memory-sharding-doesnt-touch).

`data_parallel_size = world_size / cp_size`; gradient sync always spans all `world_size` ranks.
CP is **not compatible with TP** — both partition the same contiguous rank blocks, and TP has
already cut this rank's head count that Ulysses redistributes over.

## How Ulysses works

Naive sequence chunking breaks cross-chunk attention. Ulysses instead redistributes tensors around
the attention kernel: each GPU holds `S/cp_size` tokens but all heads, one all-to-all carrying Q, K
and V swaps that to the full `S` tokens but `H/cp_size` heads for the attention compute, and a second
swaps the output back. The backward runs the same two in reverse.
Multi-head attention is independent across heads, which is what makes this exact.

Packing Q, K and V into one exchange takes two collectives per layer each way where one per tensor takes three. On
8× B300 (1.2.0, three rounds) that runs Qwen3-8B CP2 at s4096 4.4% and gpt-oss-20b ep8 × cp2 3.7% faster, and leaves
long sequences flat (cp4 at s16384 −0.4%, ep8 × cp8 at s32768 −0.1%).

![Ulysses attention: RoPE runs on the local chunk, one all-to-all on Q, K and V scatters heads and gathers the sequence so each rank attends over the full sequence with 16 of 64 heads, and an all-to-all on the output restores the per-rank chunk layout](https://whitecircle.com/halo/docs/assets/agent-docs/assets/diagrams/ulysses_attention_flow.png?ref=872a3e4284ef3a533c18b651f91b76adcbdcf306)

![An SFT training step under CP: the wrapper narrows the batch to one contiguous sequence chunk per rank, embeddings and every decoder layer run on that chunk (Ulysses attention, then the MLP or MoE), the head scores the chunk from sharded logits or fused with the loss with no logits plane, and the loss is scaled by cp_size over the CP group's all-reduced token count](https://whitecircle.com/halo/docs/assets/agent-docs/assets/diagrams/cp_training_step.png?ref=872a3e4284ef3a533c18b651f91b76adcbdcf306)

## Implementation

`UlyssesModel` (`halo/distributed/context_parallel/ulysses_model.py`) is the entry point: the CP
loaders construct it over the loaded model and the trainer adopts its `CPConfig`. The all-to-all
(`UlyssesAllToAll`, `all_to_all.py`) is a custom autograd function that packs every tensor it carries
into one buffer, row by destination rank, and runs one `all_to_all_single`: `sequence_to_heads` scatters
the heads of Q, K and V and gathers the sequence (GQA's K/V and MLA's narrower V keep their own head
counts and head dims), `heads_to_sequence` swaps the output back, and the backward runs the opposite
swap on the gradients. The per-family attention wrappers in
`families.py` subclass `UlyssesAttention` (`ulysses_attention.py`), whose one forward path runs RoPE
(and any position-dependent scale) on the local chunk before the all-to-all, so no position gather,
then Flash Attention with native GQA in `[B, S, H, D]`, with zero transposes. The MLA families
(GLM4 MoE Lite, Mistral4) share `MLAUlyssesAttention`: RoPE on the rope half of each head, and an
`_attend` that right-pads V to `qk_head_dim` where `v_head_dim` differs, runs flash, and crops back.

### Loss computation

CP computes loss per chunk with two corrections:

1. **Boundary tokens** — the causal label shift (`logits[:-1]` predicts `labels[1:]`) loses one
   prediction per chunk. Non-final ranks use their last logit to predict the first token of the next
   chunk.
2. **Globally-normalized sum loss** — completion-only training distributes tokens unevenly across
   chunks, so per-rank `reduction='mean'` biases gradients. Each rank computes
   `loss_i = cp_size * local_sum / global_tokens` with `global_tokens` the all-reduced non-masked
   token count. After FSDP2's `1/cp_size` gradient averaging every token contributes equally. The
   fp32 `local_sum` runs over checkpointed chunks of the rank's logits (`chunked_token_sum`, as the
   pipeline's last stage scores them), so no fp32 copy of the chunk's logits plane is built.

That form is rank-varying and exists for the gradient path only. Under `torch.no_grad()` (the eval
loop) the wrapper returns the rank-uniform group value instead — the CE sum and the aux term are
all-reduced over the CP group — because HF's DP-scoped metric gather keeps a single CP sibling's
copy as `eval_loss`, and a chunk-partial copy would bias it by that chunk's share of loss tokens.

The MoE router aux loss takes **no** `cp_size` factor. Each rank's `load_balancing_loss_func` sees
only its chunk's router logits, so the term is chunk-local — the load and the mean router
probability of that chunk — and the FSDP average over CP ranks trains the mean of the per-chunk
terms at `router_aux_loss_coef`. That is not the whole-sequence term (a product of means does not
split across chunks); a `cp_size` factor would only scale it `cp_size`×.

The GRPO scoring seam calls `forward_hidden_states` on the CP wrapper, which returns this rank's
backbone hidden states without constructing vocabulary-wide logits. `cp_shift_against_full_labels`
pairs them with the full row's next-token labels, and the vocab-chunked output head scores those
targets. Offline GRPO's per-row token-objective sums (and SMPO's per-sequence sums) reduce through
`cp_sum_rows` in `halo/distributed/context_parallel/sequence_chunks.py`, an autograd-aware SUM whose
backward supplies the CP factor canceled by world-wide mean gradient synchronization. They must
**not** also use the SFT loss's explicit `cp_size` multiplier. Offline GRPO uses this path for
training, evaluation and reference scoring; online and environmental GRPO do not declare CP. Its
reference and resume contract is in
[Offline GRPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/offline.md#reference-model).

### Fused head loss

Without it, each rank scores its chunk from the model's bf16 logits plane, cast to fp32 one checkpointed chunk at a
time ([Loss computation](#loss-computation)), so the plane and its gradient stay in bf16. When the run
resolves Liger's `fused_linear_cross_entropy` on for the family (requested in `liger_kernel_config`, or the family
default: GLM-4.7-Flash), a training forward instead keeps one position's logits (`logits_to_keep=1`), takes the
backbone's final hidden states, and scores them through Liger's fused linear cross-entropy under the family's
verified head transform (`fused_head_ce_sum` in `halo/model_loading/liger/cross_entropy.py`: hidden and logit scale
folded onto the hidden states, softcap in the kernel, vocabulary cut on the weight). No logits plane is built. The
weight gradient accumulates in fp32. The boundary pairing, the token normalization and the aux term are the
logits loss's. The forward returns `logits=None` and the chunk's `token_accuracy`, so SFT logs accuracy but no
entropy, as FLCE does without CP. On GLM-4.7-Flash at ep8 × cp4 and 65,536 tokens (1.2.0) the peak falls from 53.30 to
46.99 GiB (−11.8%), step-1 loss within 7e-8 relative.

Eval forwards keep the logits loss. A training forward without labels returns the full logits, with one warning:
only a labelled forward (SFT) runs the fused loss, and SMPO scores its pairs from those logits. A forward that cannot
take the fused path keeps the logits loss, with one warning:
no `logits_to_keep` parameter (Bailing's remote code), a PEFT tuner or a non-`nn.Linear` module on the output
embedding, a head path the transform probe cannot verify, or a logit scale over a biased head. The model's own head
stays unpatched under CP, since the wrapper never hands it labels.

A model that returns an `aux_loss` while its config declares no `router_aux_loss_coef` **raises** — a
stand-in weight would train a different objective than the same config without CP. Set the field
through `model_init_kwargs`, or turn the aux loss off for that family with `moe_balancing`.

## Requirements

**NVLink-local topology** — `cp_size` must divide the NVLink domain (`gpus_per_node` on a standard
node, the rack on NVL72) and cannot exceed it; Ulysses' all-to-alls per attention layer must stay
on NVLink. `cp_size=3` on an 8-GPU domain is rejected at config time, before the model loads
(`_reject_cp_locality`).

**Attention impl** — a real Flash Attention is required. `reject_unsupported_ulysses_model` raises
`CPConfigError` for any `_attn_implementation` outside `SUPPORTED_ATTN_IMPLEMENTATIONS` (FA2,
FA3, FA4): `eager`, `sdpa` and a hub kernel label are rejected, and `flex_attention` is
auto-switched to the arch's flash kernel (FA4 on Blackwell, FA3 on Hopper, else FA2;
[Flash Attention → Context Parallelism](https://whitecircle.com/halo/docs/agent-docs/optimization/flash-attention.md#model-specific-handling)) by
`load_distributed_model` when CP is active. A wrapper declaring
`REQUIRES_FLASH_ATTN_LABEL = False` waives the check (Bailing; see
[bailing.md](https://whitecircle.com/halo/docs/agent-docs/models/bailing.md#cp-wrapper)).

That check reads the *declared* implementation. The kernel CP calls is arch-matched, with FA4 refused where
`supports_fa4_backward` refuses it (GLM-4 MoE Lite's latent attention in every run, as a precaution,
head_dim-256 attention in a deterministic one,
[Flash Attention → Supported backends](https://whitecircle.com/halo/docs/agent-docs/optimization/flash-attention.md#supported-backends)), and
under CP the loader labels such a model with that kernel (FA2 on Blackwell) where a deterministic run
would otherwise fall back to SDPA, so the auto-detected default passes the check. Leave
`attn_implementation` at it.

The Ulysses call hands that kernel the `deterministic` flag transformers passes its own flash calls
(`FLASH_ATTENTION_DETERMINISTIC`, which `full_determinism` sets before the load), so a deterministic run
gets the kernel's deterministic backward under CP too.

**Head divisibility** — `cp_size` must divide both the Q and the KV head count. GPT-OSS (64 Q, 8 KV):
CP=8 → 8 Q / 1 KV; CP=4 → 16 Q / 2 KV; CP=3 rejected.

These model checks, the sequence-axis refusals below included, run on a meta-device shell of the model
before any weight is read (`reject_unsupported_structure`), and again on the loaded model at the CP wrap.

**Sequence length** must be a multiple of `cp_size`. `scripts/training/sft.py` sets `pad_to_multiple_of=cp_size`
and the collator factory routes to a padding collator that rounds each batch up — no manual collator
construction. A ragged batch that reaches SFT's `compute_loss` anyway is right-padded there, which needs
the tokenizer's `pad_token_id`: a `processing_class` without one raises rather than padding with
vocabulary token 0.

**Right padding only.** The Ulysses path ignores `attention_mask`: it calls dense flash attention
with `causal=` and has no varlen path, so it tolerates only padding a causal mask already ignores.
A left-padded batch is rejected: every real token would attend the leading pads and the loss would
silently differ from the same batch without CP. Every loader of a CP trainer collates through
`LeftPaddingGuard`, which checks each batch's `attention_mask` on the host before it moves to the
device, so the CP forward reads nothing back from the GPU to check it. The wrapped model receives the
chunk's mask only where its router aux loss reads it (`output_router_logits`, which keeps padding out of
the aux statistics); elsewhere it gets none, and transformers' flash mask construction reads nothing back.

SMPO's collator left-pads prompts and emits no `attention_mask`, so the guard has nothing to check;
its chosen|rejected concat, which builds the forward's rows on the device, flushes every row left,
so every pad trails the completion and any batch size runs.

Offline GRPO's CP collator right-pads each joined prompt+completion row
([Offline GRPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/offline.md)).

### Supported model architectures

Source of truth: `ULYSSES_ATTENTION_MAP` in `halo/distributed/context_parallel/families.py`, built
by walking the `UlyssesAttention` subclass tree and reading each wrapper's `HF_MODULE_NAMES` (a
duplicate HF name raises). The validator and the patcher both read that map — there is no separate
accept list to keep in sync.

| Attention class | Wrapper | Family specifics | Page |
|---|---|---|---|
| `GptOssAttention` | `GptOssUlyssesAttention` | split-halves RoPE; sinks must be reset | [gpt-oss.md](https://whitecircle.com/halo/docs/agent-docs/models/gpt-oss.md) |
| `Qwen3MoeAttention` (Qwen3 MoE) | `Qwen3MoeUlyssesAttention` | Q/K RMSNorm | [qwen3.md](https://whitecircle.com/halo/docs/agent-docs/models/qwen3.md#qwen3-moe) |
| `Qwen3Attention` (Qwen3 dense) | `Qwen3MoeUlyssesAttention` (reused) | Q/K RMSNorm | [qwen3.md](https://whitecircle.com/halo/docs/agent-docs/models/qwen3.md#qwen3-dense) |
| `Qwen3VLTextAttention` (Qwen3-VL text branch) | `Qwen3MoeUlyssesAttention` (reused) | Q/K RMSNorm | [qwen3.md](https://whitecircle.com/halo/docs/agent-docs/models/qwen3.md#qwen3-vl) |
| `Glm4MoeLiteAttention` | `Glm4MoeLiteUlyssesAttention` | DeepSeek-V3 MLA: separate qk/v head dims | [glm4.md](https://whitecircle.com/halo/docs/agent-docs/models/glm4.md) |
| `Mistral4Attention` | `Mistral4UlyssesAttention` | MLA: separate qk/v head dims, llama-4 scaling | [mistral4.md](https://whitecircle.com/halo/docs/agent-docs/models/mistral4.md) |
| `Qwen3_5MoeAttention` / `Qwen3_5Attention` | `Qwen3_5MoeUlyssesAttention` | post-attention sigmoid gate + partial RoPE | [qwen3_5.md](https://whitecircle.com/halo/docs/agent-docs/models/qwen3_5.md#cp-wrapper) |
| `BailingMoeV2Attention` / `…FlashAttention2` / `…SdpaAttention` (Ling 2.0) | `BailingMoeV2UlyssesAttention` | fused `query_key_value`, `dense` output projection, Q/K norm before partial RoPE | [bailing.md](https://whitecircle.com/halo/docs/agent-docs/models/bailing.md#cp-wrapper) |
| `Cohere2MoeAttention` | `Cohere2MoeUlyssesAttention` | interleaved fp32 rotary on sliding layers only; full-attention layers are NoPE | [cohere2-moe.md](https://whitecircle.com/halo/docs/agent-docs/models/cohere2-moe.md) |

Every other attention class raises `CPConfigError`, and a model carrying state along the sequence
axis (linear attention, sequence-axis convolutions, window pooling) is refused by class name even
where its attention is wrapped — Qwen3.5/3.6 on every released checkpoint. Per family, with the reason
on its page: [Compatibility matrix](https://whitecircle.com/halo/docs/agent-docs/models.md#compatibility-matrix).

## CP with EP

EP is orthogonal to DP; CP reduces it. So `data_parallel_size = world_size / cp_size` whether EP is
on or off. CP does its all-to-all in attention and EP its DeepEP dispatch/combine in the MoE
block — different points in the forward pass, no conflict.

**Orthogonal mode** (`cp_size == world_size`): one CP group over all ranks, DP=1. **Partial mode**:
`world_size / cp_size` CP groups, each on a different batch.

CP always requires gradient sync when `world_size > 1`, even at DP=1 — each rank holds partial
gradients from its chunk. `DistributedTrainerMixin` applies FSDP2 across all ranks; for EP+CP the
same wrapper runs with EP params in `ignored_params`.

EP+CP needs node-local EP filling the NVLink domain: on 8 GPUs ep8/cp2 → DP4, ep8/cp4 → DP2,
ep8/cp8 → DP1 ([Topology rules](https://whitecircle.com/halo/docs/agent-docs/parallelism.md#topology-rules)).

**Pure CP on a MoE still gets EP layers.** `load_cp_model` builds an `ep_size == 1` EP
config whenever `needs_ep_layers` holds and the model declares experts, so the
MoE blocks run the [grouped-GEMM](https://whitecircle.com/halo/docs/agent-docs/optimization/grouped-gemm.md) expert path rather than the stock
per-expert loop — `needs_ep_layers` forces Liger's routed-expert swiglu/geglu off either way, so
without it the model would pay that cost for no speedup. Dense models under CP get no EP config.

## Usage

```bash
# CP-only (any supported model, long sequences)
torchrun --nproc_per_node=4 scripts/training/sft.py \
    examples/sft/qwen3/qwen3-4b-ultrachat.yaml --context_parallel_size=4 --packing=false

# EP+CP (MoE, node-local EP with ep_group_size equal to the NVLink domain)
torchrun --nproc_per_node=8 scripts/training/sft.py \
    examples/sft/gptoss/gptoss-20b-multinode-ep.yaml \
    --expert_parallel_size=8 --ep_scope=node --use_grouped_gemm=true --context_parallel_size=2 --packing=false
```

Both example configs set `packing: true` (refused under CP), and the gpt-oss one sets
`ep_scope: global` (refused under CP), so the commands override them. The gpt-oss one also pins the per-expert loop for
its ep16, so its command turns grouped GEMM back on.

Programmatic: `parallelism_config=ParallelismConfig(ep_size=8, cp_size=8)`.

### Checkpoint saving

`save_cp_checkpoint` serves **dense** CP runs only. An MoE carries EP layers (grouped GEMM is
on by default), and the saver ladder checks for EP layers first, so EP+CP and even `ep_size=1`
MoE take the EP save — which strips the `.original_attention.` prefix itself.

`trainer.save_model(output_dir)` gathers on every rank and the FS-aware save rank writes.
`_find_cp_wrapper()` finds the wrapper on the compile-unwrapped model, directly or under PEFT. The
wrapper's `state_dict()` remaps `.original_attention.` keys to standard paths and drops the
duplicate dense `.mlp.{gate,up,down}_proj` keys only on layers carrying routed experts — genuinely
dense layers keep their dense MLP weights.

On resume CP is a **Path B** mode: the trainer skips the checkpoint weight-reload (the CP wrapper
changes the module tree). The training scripts repoint `model_name_or_path` at the checkpoint so
`load_distributed_model` loads the trained weights at construction; a model not constructed from
the checkpoint raises rather than silently continuing on its current weights.

Configured FP32 masters are restored before the CP wrapper is installed
([Load precision](https://whitecircle.com/halo/docs/agent-docs/models.md#load-precision)).

Trainer state is restored, and so are LoRA adapters: the saved CP-normalized keys are remapped back
onto the live wrapped names, and a wholesale key miss raises.

`load_best_model_at_end` is refused at construction for CP full fine-tunes (base weights only load
at construction, so the best checkpoint cannot be reloaded in place). See
[Checkpoints & Resume](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md).

## Optimizations

Native GQA holds K and V at their own head count (no `repeat_kv`), and the `[B, S, H, D]` layout drops
the transposes. The all-to-alls run in native BFloat16, with Q, K and V in one buffer.

`torch.compile` traces the exchange without a graph break, though the reentrant checkpointing CP
forces still breaks the graph; its throughput effect under CP is not measured. Use Liger (default on).

## Limitations

**Trainers.** CP is declare-to-enable (`SUPPORTS_CP`, default `False`). Offline GRPO
under CP requires full fine-tuning: it rejects PEFT and native expert LoRA at any `kl_beta`, and an
explicit `ref_model`, since it scores the run-start reference through the distributed policy
(`reject_offline_cp_options`). A trainer that does not declare CP raises at construction, and its entry script
rejects `--context_parallel_size > 1` earlier through
`parallelism_config_from_args(..., trainer_cls=...)`, which reads the same flag. Full matrix:
[Trainer Compatibility](https://whitecircle.com/halo/docs/agent-docs/reference/trainer-architecture.md#trainer-compatibility). Nothing
inspects a trainer's loss for CP-safety, so a new trainer must verify its own objective before
declaring the flag.

**Models.** Only the wrappers in the [table above](#supported-model-architectures); anything else
raises `CPConfigError`. Linear-attention, sequence-axis convolution and window-pooling blocks
are rejected by class name even when the surrounding attention is wrapped.

**Generation.** `UlyssesModel.generate()` raises: each rank holds one sequence chunk and the
Ulysses attention has no KV-cache path. Generate from a saved checkpoint without CP.

**Axis combinations.** CP composes with EP only ([allowlist](https://whitecircle.com/halo/docs/agent-docs/parallelism.md#supported-combinations)). EP
alone OOMs at long sequences (it shards experts, not activations) — that is the case CP exists for.

**Knobs.** Everything below raises unless the verdict says otherwise.

| Knob | Under CP | Gate |
|---|---|---|
| `packing`, `padding_free` | rejected — the Ulysses path runs a dense causal kernel with no per-document boundaries | `halo/data/collators/factory.py`; `reject_cp_packed_collator` re-checks a hand-built collator |
| left-padded batches | rejected on **every** batch, on the host at collation (not cached — a prompt-left-padding collator pads only its ragged batches) | `LeftPaddingGuard` in `context_parallel/sequence_chunks.py`, applied by `DataParallelLoaderMixin._loader_collate_fn` |
| `attn_implementation` | must resolve to FA2/FA3/FA4; `flex_attention` is auto-switched with a warning, `eager`/`sdpa` rejected — except for a wrapper declaring `REQUIRES_FLASH_ATTN_LABEL = False` (Bailing), which runs on any label | `SUPPORTED_ATTN_IMPLEMENTATIONS`, `reject_unsupported_ulysses_model` |
| `init_from_scratch` | rejected — no sharded random init | `model_loading/parallel/load_rules.py` |
| `reset_sinks: false` on GPT-OSS | rejected at model load — the CP attention kernels drop the sink column, so live sinks would misnormalize the softmax in every layer | `reject_unsupported_model`, re-checked in `GptOssUlyssesAttention` |
| `label_smoothing_factor > 0`, `loss_type: dft` | rejected — the Trainer pops `labels` and pairs full labels with this rank's chunk logits | `reject_cp_trainer_args` |
| `compute_metrics`, `preprocess_logits_for_metrics` | rejected whenever set, not gated on `eval_strategy` (a direct `evaluate()`/`predict()` reaches the same misaligned chunk-logits path) — eval under CP is loss-only | same |
| SFT's metrics (`mean_token_accuracy`, `entropy`, `aux_loss`, `num_attended_tokens_seen`) | scored on the rank's chunk against the full labels, accumulated locally per micro-batch and reduced once per log, as in every mode ([SFT loss and metrics](https://whitecircle.com/halo/docs/agent-docs/reference/trainer-architecture.md#sft-loss-and-metrics)); under the [fused head loss](#fused-head-loss), accuracy from the chunk's `token_accuracy` weighed by its supervised targets, and no entropy | `halo/trainers/sft.py` (`_record_loss_metrics`) |
| multimodal inputs (`pixel_values`) | rejected — text-only | `context_parallel/ulysses_model.py` |
| `fsdp_shard_ep1_experts: false` | rejected — the CP path shards `ep1` experts unconditionally, so the flag would be a silent no-op | `_reject_fsdp_settings` |
| `accelerate launch` | rejected — CP requires `torchrun` | `reject_unsupported_run` and `reject_custom_parallelism_under_accelerate` |
| `gradient_checkpointing` | supported; `use_reentrant` is forced to `True` (warned when the config sets `false`; [rule](https://whitecircle.com/halo/docs/agent-docs/optimization/flash-attention.md#usage)) | `DistributedTrainerMixin._apply_training_args_overrides` |
| `use_liger_kernel` | supported; the model's own `cross_entropy` and `fused_linear_cross_entropy` patches are off (an explicit `cross_entropy` is warned), and a resolved `fused_linear_cross_entropy` runs in the CP wrapper ([fused head loss](#fused-head-loss)) | `halo/model_loading/liger/dispatch.py`, `context_parallel/ulysses_model.py` |
| `use_peft` / LoRA | supported for SFT and SMPO — CP leaves attention unsharded, so adapters stay replicated; offline GRPO rejects adapters | `reject_offline_cp_options` (offline GRPO) |
| QLoRA (`load_in_4bit`) | supported for SFT and SMPO on a **dense** model — CP keeps the standard loader and preserves `Params4bit`. On an MoE the grouped-GEMM loader takes over and rejects a quantized base (`use_grouped_gemm` is on by default); offline GRPO rejects adapters | `model_loading/parallel/load_rules.py`, `reject_offline_cp_options` |
| `use_hsdp`, `fsdp_reshard_after_forward` | supported — CP is one of HSDP's two accepted paths (pure DP, CP) and one of ZeRO-3's EP-free shapes | `_reject_hsdp`, `_reject_fsdp_settings` |
| `torch_compile` | not gated; the exchange traces without a graph break, throughput effect not measured | — |
| `save_sharded_ep` | rejected on a CP run — per-rank shards carry `.original_attention.` keys the merge script cannot remap | `reject_unmergeable_ep_sharded_save` |

## Adding a new model

1. Subclass `UlyssesAttention` in `families.py`. The default `_project_qkv` reads plain
   `q_proj`/`k_proj`/`v_proj` into `[B, S, H, D]`, the default `_apply_rotary` is rotate-half
   (`apply_rope`) and the output projection is `OUTPUT_PROJECTION = "o_proj"`; override only what the
   family does differently (GptOss split-halves rotary, Cohere2 interleaved fp32, Bailing's fused
   projection and `dense`).

    For an MLA family subclass `MLAUlyssesAttention` instead: it owns the geometry, the rope-half
    `_apply_rotary` and the V-padding `_attend`.

2. Declare the HF attention class name in the wrapper's `HF_MODULE_NAMES`. `ULYSSES_ATTENTION_MAP`
   is read off the subclass tree, so defining the class is the whole registration.
3. Test:

```bash
torchrun --nproc_per_node=2 \
    tests/gpu/parallelism/cp/test_cp_correctness.py
# EP+CP, if MoE (2-GPU minimal repro: EP=CP=world=2):
torchrun --nproc_per_node=2 \
    tests/gpu/parallelism/combined/test_ep_cp_correctness.py
```

Further CP tests live in `tests/gpu/parallelism/cp/`. Add the family to `FAMILIES` in
`tests/common/equivalence/cp_equivalence.py` and set `ulysses_cp=True` on its `TinyFamily`:
`tests/cpu/conventions/test_tiny_family_roster.py` fails on a wrapper no roster family builds.

Background: [DeepSpeed-Ulysses](https://arxiv.org/abs/2309.14509).
