# Continued pre-training

Continued pre-training is more next-token training on raw, unlabeled text from one domain, applied to a model that is already pretrained. It is also called domain-adaptive pre-training.

## How it works

The objective is the one used in [pre-training](/glossary/pre-training): predict the next token. What changes is the starting point, a pretrained model, and the data: raw text from one domain, such as contracts, clinical notes or source code.

There are no prompts and no answers, so the loss covers every token of every document.

## When to use it

- You have a lot of raw domain text and few labeled examples.
- The domain's vocabulary and style are far from ordinary text.
- A task run comes next. [Supervised fine-tuning](/glossary/supervised-fine-tuning) (SFT) then starts from a model that already reads the domain.

Raw-text training can weaken how well a chat model follows instructions, a form of [catastrophic forgetting](/glossary/catastrophic-forgetting). Plan an instruction run after it. For documents that change, retrieval is the better tool: see [fine-tuning vs RAG](/glossary/fine-tuning-vs-rag).

## In Halo

Continued pre-training on raw text is the same trainer as SFT. Prepare the corpus once:

```bash
halo run prepare-dataset --mode text --text-field text --pack-sequences ...
```

It tokenizes documents with no chat template and appends an EOS (end-of-sequence) token per document. Then point `dataset` at the result. The config must state `train_on_completions_only: false` and the exact `max_length` the preparation used, or startup raises. See [pre-training and from-scratch](/halo/docs/training-methods/sft#pre-training-and-from-scratch).

