# Gradient checkpointing

Gradient checkpointing cuts training memory by storing only some activations in the forward pass and recomputing the rest in the backward pass. It trades extra compute for a lower memory peak.

## How it works

Backpropagation needs every layer's activations, the intermediate results of the forward pass. Normally they all stay in memory until the backward pass has used them. With long sequences they are often the largest part of a run's memory.

With gradient checkpointing, the forward pass keeps activations only at chosen points, usually the input of each transformer block. The backward pass runs each block forward again to rebuild what it needs, then frees it. Memory holds the stored inputs and one block's activations at a time. The price is roughly one more forward pass.

The gradients do not change, only speed and memory. Despite the name, it is unrelated to saving a [model checkpoint](/glossary/model-checkpoint) to disk.

## When to use it

Turn it on when a run does not fit in memory and you want to keep the [batch size](/glossary/batch-size) and the sequence length. Turn it off when the activations fit and you want the speed back.

## In Halo

In Halo's docs, `gradient_checkpointing: true` is usually the first lever when a run does not fit. With `gradient_checkpointing: false`, GPT-OSS at `ep8` ([expert parallelism](/glossary/expert-parallelism) of 8) and 4k tokens gave 29% more throughput at about 1.8 times the peak memory. See [Performance](/halo/docs/performance#levers-that-help).

