Model checkpoint

Model checkpoints are snapshots of a model's weights saved to disk during or after training. A training checkpoint also holds the optimizer state and the step count, so a stopped run can resume where it left off.

What is in it

  • Weights. In Hugging Face form, a folder with config.json, the tokenizer files and the weights as .safetensors files. Inference needs nothing more.
  • Training state. The optimizer's state, the position in the learning-rate schedule and the step count. A run needs these to resume exactly. AdamW keeps two extra values for each trained weight, so in full fine-tuning this part is larger than the weights.

A LoRA run saves only its small adapter.

Why save more than one

  • Recovery. A crash loses only the steps since the last save.
  • Selection. The last checkpoint is not always the best. Evaluate several and keep the best one.

The name is unrelated to gradient checkpointing, which saves memory during training.

In Halo

A run writes standard Hugging Face checkpoints to output_dir on the cadence set by save_strategy and save_steps:

YAML

output_dir: checkpoints/sft-qwen3-4b-ultrachat
save_strategy: steps
save_steps: 500

They load with from_pretrained. The exceptions are LoRA adapters and the opt-in sharded save, which needs halo run merge-ep-shards first.

To resume, point resume_from_checkpoint at a checkpoint directory, or set it to true for the latest one in output_dir. If the run saved with save_only_model: true, the resume is a warm restart: weights and schedule restored, optimizer moments fresh. See Checkpoints.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL