# Learning rate warmup and schedules

Learning rate warmup raises the rate from near zero to its peak over the first steps of training. A schedule then lowers it, usually along a straight line or a cosine curve, until the run ends.

## How it works

The [learning rate](/glossary/learning-rate) in a config is a peak, not a constant. A typical run has two phases:

1. **Warmup.** The rate climbs in a straight line from zero to the peak, over a set number of [steps](/glossary/epochs-and-steps) or the first few percent of the run.
2. **Decay.** The rate falls from the peak toward zero at the last step.

| Schedule | Shape of the decay |
| --- | --- |
| Linear | A straight line down |
| Cosine | Half a cosine wave: slow, then fast, then slow |
| Constant | None: the rate stays at the peak |

## Why warm up

At the first step the optimizer has no history. Adam-style optimizers scale each weight's update by running averages of its past gradients, and the first averages are noisy. A full-size step on them can push pretrained weights far off. Small early steps let the averages settle.

Decay does the opposite job at the end: smaller steps let the model settle instead of bouncing around a good solution.

## In Halo

```yaml
lr_scheduler_type: cosine
warmup_steps: 32
```

`warmup_steps` is an exact step count at 1 or above and a fraction of the run below it (`0.03`). There is no `warmup_ratio`. See the [supervised fine-tuning guide](/halo/docs/training-methods/sft#config).

