# What is pre-training?

Pre-training is the first and largest stage of training a language model. Starting from random weights, the model learns to predict the next token over a very large corpus of raw text, with no task in mind.

## How it works

The model reads raw text as [tokens](/glossary/tokenization) and, at every position, predicts the token that comes next. The loss is cross-entropy against the true next token. The text is its own label, so nobody has to annotate anything. This is called self-supervised learning.

Over a large enough corpus, that one objective makes the model pick up grammar, facts and patterns of reasoning. The result is a base model. It continues text well and does not yet follow requests reliably.

## Pre-training vs fine-tuning

|  | Pre-training | Fine-tuning |
| --- | --- | --- |
| Starts from | random weights | a pretrained model |
| Data | raw text in bulk | curated examples, far fewer |
| Teaches | language and general knowledge | a task, a format or a behavior |

Most teams never pre-train. They take an [open-weight model](/glossary/open-weight-models) and [fine-tune](/glossary/what-is-fine-tuning) it. In between sits [continued pre-training](/glossary/continued-pre-training): more next-token training on raw text from one domain.

## In Halo

Pre-training runs through the `sft` method on raw text. `init_from_scratch: true` builds random weights from the model's config instead of reading a checkpoint, and it runs on plain data parallelism ([FSDP2](/glossary/fsdp)) only. See [pre-training and from-scratch](/halo/docs/training-methods/sft#pre-training-and-from-scratch).

