Sequence packing

Sequence packing joins short training examples end to end into rows of one fixed length, so a batch holds real tokens and no padding. It speeds up training when examples are much shorter than the maximum length.

How it works

Every row in a batch has the same length, so short examples are filled out with pad tokens. The loss ignores padding, but the GPU still computes on it. If the average example is a quarter of the row, about three quarters of the compute is wasted.

Packing puts several examples end to end in each row, up to the maximum sequence length. Two things keep it correct:

  • Boundaries. A careful implementation tracks where each example starts, so examples in one row do not attend to each other. A simple one only joins them.
  • Fit. Filling rows is a bin-packing problem. A common strategy is best-fit decreasing: from longest to shortest, each example goes into the fullest row that still has room.

When to use it

Use it for text of short or mixed lengths, such as chat data. It gains nothing when rows already fill the maximum length.

A packed row holds several examples, so the same batch size covers more of them per step.

In Halo

YAML

max_length: 4096
packing: true

Packing needs an explicit max_length, which becomes both the pack size and the memory bound. It is the default choice for text supervised fine-tuning. For a large corpus, pack once offline with halo run prepare-dataset and its --pack-sequences flag. See Datasets.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL