Model checkpoints are snapshots of a model's weights saved to disk during or after training. A training checkpoint also holds the optimizer state and the step count, so a stopped run can resume where it left off.
What is in it
- Weights. In Hugging Face form, a folder with
config.json, the tokenizer files and the weights as.safetensorsfiles. Inference needs nothing more. - Training state. The optimizer's state, the position in the learning-rate schedule and the step count. A run needs these to resume exactly. AdamW keeps two extra values for each trained weight, so in full fine-tuning this part is larger than the weights.
A LoRA run saves only its small adapter.
Why save more than one
- Recovery. A crash loses only the steps since the last save.
- Selection. The last checkpoint is not always the best. Evaluate several and keep the best one.
The name is unrelated to gradient checkpointing, which saves memory during training.
In Halo
A run writes standard Hugging Face checkpoints to output_dir on the cadence set by save_strategy and save_steps:
YAML
output_dir: checkpoints/sft-qwen3-4b-ultrachat
save_strategy: steps
save_steps: 500They load with from_pretrained. The exceptions are LoRA adapters and the opt-in sharded save, which needs halo run merge-ep-shards first.
To resume, point resume_from_checkpoint at a checkpoint directory, or set it to true for the latest one in output_dir. If the run saved with save_only_model: true, the resume is a warm restart: weights and schedule restored, optimizer moments fresh. See Checkpoints.
