Checkpoints: Save, Resume, Export, Upload
A run writes standard HuggingFace checkpoints to output_dir on the cadence set
by save_strategy / save_steps. Every mode (FSDP2, EP, TP, CP, PP and their
combinations) writes a checkpoint that from_pretrained loads. The exceptions
are the opt-in sharded EP save, a PP save on per-node storage, and LoRA
adapters.
Every save uses the tensor names of the checkpoint the run loaded, vision towers
included, and an EP save writes the family's own expert layout
(details ↗). A
save holds an MTP head only if the run
trained one ↗, under the
release's keys, or the family's remote class builds its tail (Ling-3.0-flash).
Drafting with method=mtp needs one: vLLM refuses a checkpoint without it.
EP expert LoRA cannot be merged afterwards: train it with
merge_expert_lora_on_save: true to save the merged model.
Sharded EP saves
save_sharded_ep: true skips the gather, which makes checkpointing a very large
model faster. Run halo run merge-ep-shards before the checkpoint can be used;
the merged directory carries every rank's optimizer shard, so it also resumes
exactly on the topology that wrote it.
It needs one EP group spanning all ranks, no CP, ETP or expert LoRA, a shared
filesystem across nodes and a supported family, all checked at startup
(full list ↗).
Resume
Set resume_from_checkpoint to a checkpoint directory, or to true for the
newest complete one in output_dir, and launch the same training script again.
The training script restores the weights, so resume through it, not through a
hand-built model.
A save is safe to interrupt: save_total_limit never deletes a checkpoint
before its successor is complete. LoRA runs restore the adapters onto the base,
and MoE router-balancing state is restored automatically. Checkpoints from Halo
1.1.0 resume on this release; merge a sharded EP save first.
load_best_model_at_end is refused on a full fine-tune under CP, on an MoE with
EP layers, and under TP with several replicas
(full mechanics ↗).
Per-node PP saves
On per-node storage (DIST_OUTPUT_SHARED_FILESYSTEM=0) a PP save leaves each
node only its own stages' parts and its own ranks' optimizer state. Each node
resumes from its own directory while the topology and node ranks stay the same.
To load, serve or upload the result, copy every node's directory onto one host
and merge them:
Bash
halo run merge-pp-stages --input_dir /copies/node0/checkpoint-500 \
/copies/node1/checkpoint-500 --output_dir /models/mergedThe merge refuses a missing part or optimizer shard, and the merged directory also resumes on the same topology (Per-node export ↗).
Post-processing tools
All run as halo run <tool> <flags>; halo run <tool> -- --help lists the
flags.
The tools write to a new directory (only reset-sinks takes --in_place) and
reject a sharded EP save until merge-ep-shards has run. Only merge-ep-shards
and merge-pp-stages carry the optimizer shards. Flags and input guards:
Scripts reference ↗.
Serving the result
A gathered checkpoint loads with from_pretrained. vLLM and SGLang serve it
only where the pinned engine reads that family's layout:
- No serving path: Mistral4 and Ring on either engine (neither registers their architecture), and Zaya on SGLang, whose loader reads a layout the export does not carry.
- Untested: Zaya on vLLM's generic transformers backend, Ling 3.0, Inkling,
and GLM-5 Next and DeepSeek-V4 after
unfuse-moe-experts. The exports use the hub's names, but no engine has served one. - Every other family serves its normal save as is, except a Cohere2 MoE
(Command A+) EP save on SGLang: rewrite it with
unfuse-moe-expertsfirst. Atext_only_modelexport of Nemotron 3.5 Super VL serves afterreattach-vision-tower. - What has served on 1.2.0: one-step RL exports of Qwen3.6-35B-A3B and Gemma 4, an MTP head fit of Qwen3.6-35B-A3B and untrained exports of Nemotron 3.5 Super VL (reattached) and Lightning on both engines, and an MTP head fit of GLM-4.7-Flash on vLLM, each on short prompts.
- Fused experts on a per-expert-only loader are refused or dropped without
an error. Rewrite such a checkpoint with
unfuse-moe-expertsfirst; the result is the hub's own layout (per-engine loaders ↗). - Sharded EP saves need
merge-ep-shardsfirst, and per-node PP savesmerge-pp-stages. - LoRA runs serve as base plus adapter, or merged.
Serving during training, for the RL methods, is Rollout Servers.
Uploading to the HuggingFace Hub
Upload a final model yourself. A gathered checkpoint is a plain HF model directory, so the standard Hub CLI works:
Bash
hf auth login # once, or set HF_TOKEN
hf upload my-org/my-model checkpoints/sft-qwen3-4b-ultrachat/checkpoint-1000Upload the checkpoint directory (or a converted copy), not the whole
output_dir, and leave out its training state, or it goes public with the
weights:
trainer_state.jsonandscheduler.pt;optimizer*andrng_state*, unless the run setsave_only_model: true;router_balancing_biases.pt(bias balancing);reference_logps.pt(DPO/KTOprecompute_ref_log_probs, or offline GRPO full fine-tuning atkl_beta > 0);prefetch_pending-*.pt(async GRPO with prefetch): drawn but untrained prompts, answers included;resume_adapter/andresume_adapter.json(merge_expert_lora_on_saveor embedding LoRA).
For a LoRA run, upload the adapter directory, or merge it with
merge-peft-adapters for a standalone model. Load an uploaded adapter with
PeftModel.from_pretrained.
The inherited push_to_hub / hub_strategy upload is not tested end to end. It
pushes log/run.log and, for online or async GRPO, the completions/ prompts
along with the model files, and hub_strategy: checkpoint or all_checkpoints
also uploads training state
(details ↗).
Model card
Every checkpoint Halo writes holds a README.md model card tagged halo, so the
upload lists under that Hub tag. Before you publish, fill in its body and replace
a local-path base_model with the Hub id.
Shard layouts, merge flags and the full resume mechanics: Checkpoints ↗ · Model Merging ↗.
Source on GitHubUpdated 18 November 2026