Learning rate warmup raises the rate from near zero to its peak over the first steps of training. A schedule then lowers it, usually along a straight line or a cosine curve, until the run ends.
How it works
The learning rate in a config is a peak, not a constant. A typical run has two phases:
- Warmup. The rate climbs in a straight line from zero to the peak, over a set number of steps or the first few percent of the run.
- Decay. The rate falls from the peak toward zero at the last step.
Why warm up
At the first step the optimizer has no history. Adam-style optimizers scale each weight's update by running averages of its past gradients, and the first averages are noisy. A full-size step on them can push pretrained weights far off. Small early steps let the averages settle.
Decay does the opposite job at the end: smaller steps let the model settle instead of bouncing around a good solution.
In Halo
YAML
lr_scheduler_type: cosine
warmup_steps: 32warmup_steps is an exact step count at 1 or above and a fraction of the run below it (0.03). There is no warmup_ratio. See the supervised fine-tuning guide.
