Overfitting is when a model memorizes its training examples instead of learning the pattern behind them. Its loss on the training data keeps falling while its loss on held-out data rises.
How to spot it
Hold out some rows and never train on them. Then compare two curves:
- Training loss falls for as long as the model keeps fitting its examples.
- Held-out loss falls with it at first, then flattens and turns up.
Where the curves part, learning has stopped and memorizing has begun.
An overfit language model repeats training answers almost word for word. It does well on prompts close to the training set and worse on new phrasing. A small dataset and many passes over it are the usual cause.
How to fix it
- Train less. Run fewer epochs, or keep the checkpoint with the lowest held-out loss.
- Add data. More examples, and more varied ones. See how much data you need.
- Train fewer weights. With LoRA, lower the rank.
- Keep measuring. Evaluate every checkpoint on the held-out rows.
In Halo
test_size carves an evaluation split out of the dataset, and eval_strategy with eval_steps sets when evaluation runs:
YAML
test_size: 0.01
eval_strategy: steps
eval_steps: 500Halo logs loss, learning rate and gradient norm at every logging_steps, and report_to sends them to Weights & Biases, ClearML or TensorBoard. See Monitoring.
