Learning rate

Learning rate is the size of the step an optimizer takes each time it updates a model's weights. Too high a rate makes training diverge or erases what the model knew, and too low a rate leaves the model barely changed.

How it works

Each training step computes a gradient, which says how to change every weight to lower the loss on the current batch. The optimizer moves the weights that way, and the learning rate scales how far.

The value in a config is the peak. Most runs climb to it and then decay. See warmup and schedules.

How to set it

Full fine-tuning moves pretrained weights, so it needs a small rate: on the order of 1e-5, and lower for larger models. LoRA trains new, small matrices and wants 5 to 10 times the full fine-tuning rate.

Sweep a few values on a short run and read the outputs, not only the loss:

  • Too high. The loss spikes, bounces, or turns to NaN (not a number). A quieter failure: the loss looks fine while the model loses pretrained ability (catastrophic forgetting).
  • Too low. The loss barely falls, and the tuned model behaves like the base model.

In Halo

YAML

learning_rate: 2.0e-05

That is the 4B full fine-tuning example. A large model wants a rate near 2.5e-6 to 5e-6, and the LoRA example uses 1.0e-04. To sweep without copying files, override it at launch:

Bash

halo launch sft my-run.yaml -n 8 --learning_rate=1e-5

See Writing a config.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL