Context length

Context length is the largest number of tokens a model can read at once, prompt and answer together. In training, the sequence length you choose sets how much memory each example needs.

How it works

A transformer attends over every token in its window at once. The context length, also called the context window, is the size of that window. It is set by how the model was built and trained.

Everything counts against it: the system prompt, the conversation so far, retrieved documents, tool results and the answer being written. All of it is counted in tokens, not words.

In training

The training length is a setting of its own, at or below what the model supports. Longer sequences cost more: activation memory grows with every token, and attention compute grows with the square of the length. A long sequence is a common reason for a CUDA out of memory error.

In Halo

max_length is the single sequence-length knob, and null resolves to the model's context window. In supervised fine-tuning a row longer than max_length is dropped, not truncated.

For long sequences, --context_parallel_size=N splits each sequence across GPUs. Context parallelism is a way to afford a long sequence, not a way to go faster. In Halo's rough numbers, GPT-OSS 20B at a 32k context trains on 8 B300 GPUs (ep8 + cp8, gradient checkpointing on) at 9,000 tokens per second per GPU. See Parallelism and Performance.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL