Learning rate warmup and schedules

Learning rate warmup raises the rate from near zero to its peak over the first steps of training. A schedule then lowers it, usually along a straight line or a cosine curve, until the run ends.

How it works

The learning rate in a config is a peak, not a constant. A typical run has two phases:

  1. Warmup. The rate climbs in a straight line from zero to the peak, over a set number of steps or the first few percent of the run.
  2. Decay. The rate falls from the peak toward zero at the last step.
ScheduleShape of the decay
LinearA straight line down
CosineHalf a cosine wave: slow, then fast, then slow
ConstantNone: the rate stays at the peak

Why warm up

At the first step the optimizer has no history. Adam-style optimizers scale each weight's update by running averages of its past gradients, and the first averages are noisy. A full-size step on them can push pretrained weights far off. Small early steps let the averages settle.

Decay does the opposite job at the end: smaller steps let the model settle instead of bouncing around a good solution.

In Halo

YAML

lr_scheduler_type: cosine
warmup_steps: 32

warmup_steps is an exact step count at 1 or above and a fraction of the run below it (0.03). There is no warmup_ratio. See the supervised fine-tuning guide.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL