To evaluate a fine-tuned model, test it on held-out examples it never trained on, compare it with the base model on the same prompts, measure the task itself, and check that it has not lost older skills.
Steps
- Hold out a test set before training. Set aside rows the model never sees, and check that none of them has a near-duplicate in the training set.
- Score the base model first. Run the untuned model on the same test set with the same prompts and settings. A score means little without that baseline.
- Measure the task, not only the loss. Held-out loss shows overfitting, but users never see a loss. Use the task's own metric: exact match for extraction, accuracy for classification, tests passed for code, a rubric or a grader model for open answers.
- Check older skills. Run a general test set that the fine-tune was not about. A drop there is catastrophic forgetting.
- Read samples. Put base and tuned answers side by side and read a few dozen. Metrics miss repetition, odd formatting and answers that do not stop.
- Test as you will serve. Use the chat template, system prompt and sampling settings you will run in production.
- Repeat for every checkpoint. Keep the test set fixed so that runs stay comparable, and pick the checkpoint with the best score, not the last one.
With Halo
test_size carves an evaluation split out of the dataset, and eval_strategy with eval_steps sets when evaluation runs. generate_eval_examples adds a table of sample generations at each evaluation (on by default, off for supervised fine-tuning).
By default checkpoints are written in standard Hugging Face form and load with from_pretrained, so you can run your own test set against any of them. See Monitoring.
