Tokenization

Tokenization splits text into tokens, the sub-word pieces a language model reads and writes, and maps each one to an integer ID. Context limits, prices and training speed are all counted in tokens.

How it works

Each model has a tokenizer with a fixed vocabulary, from tens of thousands to a few hundred thousand tokens. Most are built with byte-pair encoding (BPE) or a close relative: start from single bytes or characters, then merge the most frequent neighboring pair into a new token, again and again, until the vocabulary is full.

Common words end up as one token. Rare words split into several. In English a token averages roughly three quarters of a word.

Special tokens

Some tokens carry structure, not text:

  • BOS and EOS mark the beginning and the end of a sequence. Generation stops when the model writes EOS, so a fine-tune that drops it produces a model that does not stop.
  • PAD fills short rows so that a batch is a rectangle. Padded positions are masked out of attention and of the loss.
  • Role markers come from the chat template.

Why token counts matter

A model reads at most its context length in tokens. Hosted models are priced per token. Training throughput is measured in tokens per second.

In Halo

halo run prepare-dataset tokenizes, optionally packs, and shards a corpus once, offline, so a run does not repeat that work at every start. See Datasets.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL