# Context parallelism

Context parallelism splits one long sequence across GPUs, so each GPU holds only a chunk of the tokens. It makes long-context training fit in memory but does not make it faster.

## How it works

Activation memory grows with sequence length. Under context parallelism (CP) each GPU holds one chunk of the sequence and runs the per-token parts of every layer on that chunk alone.

Attention is the exception, because every token has to see the others. In the Ulysses scheme, which Halo uses, each GPU trades its sequence chunk for a slice of the attention heads over the whole sequence, computes attention, and trades back. The result is exact, not an approximation.

## When to use it

Use it when one sequence at the [context length](/glossary/context-length) you need does not fit on a GPU, even at batch size 1. On shorter rows it only adds communication.

## In Halo

Set `--context_parallel_size=N`. The head counts and the sequence length must divide by `cp_size`, and the collator pads the sequence to make that true.

- Each attention layer costs three all-to-alls.
- CP works for supervised fine-tuning, [SMPO](/glossary/smpo) and offline GRPO. Offline GRPO under CP is full fine-tuning only. Online and environment GRPO do not support it.
- It can pair with [expert parallelism](/glossary/expert-parallelism), not with [tensor parallelism](/glossary/tensor-parallelism).

In Halo's measurements on B300 GPUs, CP holds memory at 25 to 42 GiB from 16k to 64k tokens and buys no speed ([Performance](/halo/docs/performance)). The layout rules are in [Parallelism](/halo/docs/parallelism).

