# How to evaluate a fine-tuned model

To evaluate a fine-tuned model, test it on held-out examples it never trained on, compare it with the base model on the same prompts, measure the task itself, and check that it has not lost older skills.

## Steps

1. **Hold out a test set before training.** Set aside rows the model never sees, and check that none of them has a near-duplicate in the training set.
2. **Score the base model first.** Run the untuned model on the same test set with the same prompts and settings. A score means little without that baseline.
3. **Measure the task, not only the loss.** Held-out loss shows [overfitting](/glossary/overfitting), but users never see a loss. Use the task's own metric: exact match for extraction, accuracy for classification, tests passed for code, a rubric or a grader model for open answers.
4. **Check older skills.** Run a general test set that the fine-tune was not about. A drop there is [catastrophic forgetting](/glossary/catastrophic-forgetting).
5. **Read samples.** Put base and tuned answers side by side and read a few dozen. Metrics miss repetition, odd formatting and answers that do not stop.
6. **Test as you will serve.** Use the [chat template](/glossary/chat-template), system prompt and sampling settings you will run in production.
7. **Repeat for every checkpoint.** Keep the test set fixed so that runs stay comparable, and pick the [checkpoint](/glossary/model-checkpoint) with the best score, not the last one.

## With Halo

`test_size` carves an evaluation split out of the dataset, and `eval_strategy` with `eval_steps` sets when evaluation runs. `generate_eval_examples` adds a table of sample generations at each evaluation (on by default, off for supervised fine-tuning).

By default checkpoints are written in standard Hugging Face form and load with `from_pretrained`, so you can run your own test set against any of them. See [Monitoring](/halo/docs/monitoring#what-gets-logged).

