How to evaluate a fine-tuned model

To evaluate a fine-tuned model, test it on held-out examples it never trained on, compare it with the base model on the same prompts, measure the task itself, and check that it has not lost older skills.

Steps

  1. Hold out a test set before training. Set aside rows the model never sees, and check that none of them has a near-duplicate in the training set.
  2. Score the base model first. Run the untuned model on the same test set with the same prompts and settings. A score means little without that baseline.
  3. Measure the task, not only the loss. Held-out loss shows overfitting, but users never see a loss. Use the task's own metric: exact match for extraction, accuracy for classification, tests passed for code, a rubric or a grader model for open answers.
  4. Check older skills. Run a general test set that the fine-tune was not about. A drop there is catastrophic forgetting.
  5. Read samples. Put base and tuned answers side by side and read a few dozen. Metrics miss repetition, odd formatting and answers that do not stop.
  6. Test as you will serve. Use the chat template, system prompt and sampling settings you will run in production.
  7. Repeat for every checkpoint. Keep the test set fixed so that runs stay comparable, and pick the checkpoint with the best score, not the last one.

With Halo

test_size carves an evaluation split out of the dataset, and eval_strategy with eval_steps sets when evaluation runs. generate_eval_examples adds a table of sample generations at each evaluation (on by default, off for supervised fine-tuning).

By default checkpoints are written in standard Hugging Face form and load with from_pretrained, so you can run your own test set against any of them. See Monitoring.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL