# Sequence packing

Sequence packing joins short training examples end to end into rows of one fixed length, so a batch holds real tokens and no padding. It speeds up training when examples are much shorter than the maximum length.

## How it works

Every row in a batch has the same length, so short examples are filled out with pad tokens. The loss ignores padding, but the GPU still computes on it. If the average example is a quarter of the row, about three quarters of the compute is wasted.

Packing puts several examples end to end in each row, up to the maximum [sequence length](/glossary/context-length). Two things keep it correct:

- **Boundaries.** A careful implementation tracks where each example starts, so examples in one row do not attend to each other. A simple one only joins them.
- **Fit.** Filling rows is a bin-packing problem. A common strategy is best-fit decreasing: from longest to shortest, each example goes into the fullest row that still has room.

## When to use it

Use it for text of short or mixed lengths, such as chat data. It gains nothing when rows already fill the maximum length.

A packed row holds several examples, so the same [batch size](/glossary/batch-size) covers more of them per step.

## In Halo

```yaml
max_length: 4096
packing: true
```

Packing needs an explicit `max_length`, which becomes both the pack size and the memory bound. It is the default choice for text [supervised fine-tuning](/glossary/supervised-fine-tuning). For a large corpus, pack once offline with `halo run prepare-dataset` and its `--pack-sequences` flag. See [Datasets](/halo/docs/data).

