Continued pre-training

Continued pre-training is more next-token training on raw, unlabeled text from one domain, applied to a model that is already pretrained. It is also called domain-adaptive pre-training.

How it works

The objective is the one used in pre-training: predict the next token. What changes is the starting point, a pretrained model, and the data: raw text from one domain, such as contracts, clinical notes or source code.

There are no prompts and no answers, so the loss covers every token of every document.

When to use it

  • You have a lot of raw domain text and few labeled examples.
  • The domain's vocabulary and style are far from ordinary text.
  • A task run comes next. Supervised fine-tuning (SFT) then starts from a model that already reads the domain.

Raw-text training can weaken how well a chat model follows instructions, a form of catastrophic forgetting. Plan an instruction run after it. For documents that change, retrieval is the better tool: see fine-tuning vs RAG.

In Halo

Continued pre-training on raw text is the same trainer as SFT. Prepare the corpus once:

Bash

halo run prepare-dataset --mode text --text-field text --pack-sequences ...

It tokenizes documents with no chat template and appends an EOS (end-of-sequence) token per document. Then point dataset at the result. The config must state train_on_completions_only: false and the exact max_length the preparation used, or startup raises. See pre-training and from-scratch.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL