Pre-training is the first and largest stage of training a language model. Starting from random weights, the model learns to predict the next token over a very large corpus of raw text, with no task in mind.
How it works
The model reads raw text as tokens and, at every position, predicts the token that comes next. The loss is cross-entropy against the true next token. The text is its own label, so nobody has to annotate anything. This is called self-supervised learning.
Over a large enough corpus, that one objective makes the model pick up grammar, facts and patterns of reasoning. The result is a base model. It continues text well and does not yet follow requests reliably.
Pre-training vs fine-tuning
Most teams never pre-train. They take an open-weight model and fine-tune it. In between sits continued pre-training: more next-token training on raw text from one domain.
In Halo
Pre-training runs through the sft method on raw text. init_from_scratch: true builds random weights from the model's config instead of reading a checkpoint, and it runs on plain data parallelism (FSDP2) only. See pre-training and from-scratch.
