# Model checkpoint

Model checkpoints are snapshots of a model's weights saved to disk during or after training. A training checkpoint also holds the optimizer state and the step count, so a stopped run can resume where it left off.

## What is in it

- **Weights.** In Hugging Face form, a folder with `config.json`, the tokenizer files and the weights as `.safetensors` files. Inference needs nothing more.
- **Training state.** The optimizer's state, the position in the learning-rate schedule and the step count. A run needs these to resume exactly. AdamW keeps two extra values for each trained weight, so in full fine-tuning this part is larger than the weights.

A [LoRA](/glossary/lora) run saves only its small adapter.

## Why save more than one

- **Recovery.** A crash loses only the steps since the last save.
- **Selection.** The last checkpoint is not always the best. [Evaluate](/glossary/how-to-evaluate-a-fine-tuned-model) several and keep the best one.

The name is unrelated to [gradient checkpointing](/glossary/gradient-checkpointing), which saves memory during training.

## In Halo

A run writes standard Hugging Face checkpoints to `output_dir` on the cadence set by `save_strategy` and `save_steps`:

```yaml
output_dir: checkpoints/sft-qwen3-4b-ultrachat
save_strategy: steps
save_steps: 500
```

They load with `from_pretrained`. The exceptions are LoRA adapters and the opt-in sharded save, which needs `halo run merge-ep-shards` first.

To resume, point `resume_from_checkpoint` at a checkpoint directory, or set it to `true` for the latest one in `output_dir`. If the run saved with `save_only_model: true`, the resume is a warm restart: weights and schedule restored, optimizer moments fresh. See [Checkpoints](/halo/docs/checkpoints).

