Synthetic data is training data written by a model instead of collected from people. It is used to start a dataset from a few examples, to copy a larger model's behavior into a smaller one, and to cover rare cases.
What it is used for
- Bootstrapping. From a handful of real examples, a strong model writes many more in the same form, enough for a first fine-tune.
- Distillation. A large model answers real prompts and a smaller one is tuned on its answers. See knowledge distillation.
- Rare cases. Edge cases, refusals and unusual formats that real logs hold too few of.
Risks
- Errors are copied. The tuned model learns the generator's mistakes and made-up facts along with its skills.
- Low diversity. A model asked for a thousand examples repeats topics, phrasings and structures. A model tuned on them can overfit to that narrow style.
How to check it
- Read a sample by hand before training.
- Remove duplicates and near-duplicates.
- Filter with a grader: rules, a verifier, or a second model that scores each row.
- Vary the generation prompts, each seeded with a different topic or real example.
- Keep the test set real. Evaluate on data people wrote.
In Halo
Halo ships tools for this, each run as halo run <tool>: openai-batched-generation for batched generation against a vLLM or OpenAI endpoint, dataset-deduplication to deduplicate generated or collected data, and rm-rejection-sampling for best-of-N with a reward model. See the halo CLI.
