What is pre-training?

Pre-training is the first and largest stage of training a language model. Starting from random weights, the model learns to predict the next token over a very large corpus of raw text, with no task in mind.

How it works

The model reads raw text as tokens and, at every position, predicts the token that comes next. The loss is cross-entropy against the true next token. The text is its own label, so nobody has to annotate anything. This is called self-supervised learning.

Over a large enough corpus, that one objective makes the model pick up grammar, facts and patterns of reasoning. The result is a base model. It continues text well and does not yet follow requests reliably.

Pre-training vs fine-tuning

Pre-trainingFine-tuning
Starts fromrandom weightsa pretrained model
Dataraw text in bulkcurated examples, far fewer
Teacheslanguage and general knowledgea task, a format or a behavior

Most teams never pre-train. They take an open-weight model and fine-tune it. In between sits continued pre-training: more next-token training on raw text from one domain.

In Halo

Pre-training runs through the sft method on raw text. init_from_scratch: true builds random weights from the model's config instead of reading a checkpoint, and it runs on plain data parallelism (FSDP2) only. See pre-training and from-scratch.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL