Tokenization splits text into tokens, the sub-word pieces a language model reads and writes, and maps each one to an integer ID. Context limits, prices and training speed are all counted in tokens.
How it works
Each model has a tokenizer with a fixed vocabulary, from tens of thousands to a few hundred thousand tokens. Most are built with byte-pair encoding (BPE) or a close relative: start from single bytes or characters, then merge the most frequent neighboring pair into a new token, again and again, until the vocabulary is full.
Common words end up as one token. Rare words split into several. In English a token averages roughly three quarters of a word.
Special tokens
Some tokens carry structure, not text:
- BOS and EOS mark the beginning and the end of a sequence. Generation stops when the model writes EOS, so a fine-tune that drops it produces a model that does not stop.
- PAD fills short rows so that a batch is a rectangle. Padded positions are masked out of attention and of the loss.
- Role markers come from the chat template.
Why token counts matter
A model reads at most its context length in tokens. Hosted models are priced per token. Training throughput is measured in tokens per second.
In Halo
halo run prepare-dataset tokenizes, optionally packs, and shards a corpus once, offline, so a run does not repeat that work at every start. See Datasets.
