# Tokenization

Tokenization splits text into tokens, the sub-word pieces a language model reads and writes, and maps each one to an integer ID. Context limits, prices and training speed are all counted in tokens.

## How it works

Each model has a tokenizer with a fixed vocabulary, from tens of thousands to a few hundred thousand tokens. Most are built with byte-pair encoding (BPE) or a close relative: start from single bytes or characters, then merge the most frequent neighboring pair into a new token, again and again, until the vocabulary is full.

Common words end up as one token. Rare words split into several. In English a token averages roughly three quarters of a word.

## Special tokens

Some tokens carry structure, not text:

- **BOS and EOS** mark the beginning and the end of a sequence. Generation stops when the model writes EOS, so a fine-tune that drops it produces a model that does not stop.
- **PAD** fills short rows so that a batch is a rectangle. Padded positions are masked out of attention and of the loss.
- **Role markers** come from the [chat template](/glossary/chat-template).

## Why token counts matter

A model reads at most its [context length](/glossary/context-length) in tokens. Hosted models are priced per token. [Training throughput](/glossary/training-throughput) is measured in tokens per second.

## In Halo

`halo run prepare-dataset` tokenizes, optionally packs, and shards a corpus once, offline, so a run does not repeat that work at every start. See [Datasets](/halo/docs/data).

