Model Merging
Combine several fine-tuned checkpoints of the same architecture into one model in weight space — no extra training, no inference-time ensembling. scripts/after_training/merge_models.py is the self-contained merger (no mergekit dependency) and supports four methods:
All checkpoints must share the architecture: every model has to provide each parameter key at the same shape, or the merge raises — as does a model carrying a key the others do not.
Floating-point tensors are merged in float32 and cast to --dtype (default bfloat16), except router-balancing tensors, which keep their source dtype (a bf16 cast quantizes away the fp32 sign steps the balancing update writes). Non-float tensors (routing tables, expert-count buffers) are copied through unchanged, and a disagreement between models raises rather than inventing an average.
Usage
Bash
# Weighted linear average (weights are normalized by their sum)
python scripts/after_training/merge_models.py --method linear \
--models /ckpt/run-a:0.3 /ckpt/run-b:0.7 --output_dir <large-volume>/merged-linear
# SLERP halfway between two checkpoints
python scripts/after_training/merge_models.py --method slerp --t 0.5 \
--models /ckpt/run-a /ckpt/run-b --output_dir <large-volume>/merged-slerp
# TIES onto a base
python scripts/after_training/merge_models.py --method ties --density 0.6 --lambda 1.0 \
--base_model /ckpt/base --models /ckpt/run-a:0.5 /ckpt/run-b:0.5 --output_dir <large-volume>/merged-tiesEach --models entry is a local directory, optionally suffixed :weight (split on the last colon, so absolute paths are safe); the default weight is 1.0. Weights are read off disk, so a Hub id there raises FileNotFoundError; --tokenizer_source is a local directory too (a Hub id is refused, so the local --trust_remote_code default covers every source — download the repo first).
The Applies to column is enforced, not advisory: a knob passed explicitly for a method that does not consume it is rejected before any I/O, so a refused merge leaves no output directory behind. Defaults resolve after that gate, so naming a knob at its documented default still counts as passing it. A value outside the knob's range, or a non-finite :weight, is rejected at the same point: a --density outside (0, 1] would keep every delta, and a --t outside [0, 1] would extrapolate past both models.
Memory and output
Merging runs on CPU and streams the inputs one tensor at a time — each key is read from every model, merged, then released — so peak host memory is set by the costliest tensor (the largest, in practice), never the merged model: every input's copy of it as stored, the method's fp32 working copies of it, and the writer's staged output parts: the shard it is filling plus one per write thread (--max_shard_size each, HALO_CHECKPOINT_WRITE_THREADS, default 0). linear, task_arithmetic and slerp hold 3, 4 and 5 working copies whatever the model count; ties holds 5.25 per model plus 5 (each delta, their stacked copy, the sign-masked copy, and the agreement mask with its int64 cast), so a TIES merge of four bf16 models onto a bf16 base, whose largest tensor has 1B elements, peaks near 120 GB. The tool warns before merging when this estimate exceeds the host's available RAM. Peak disk is the merged artifact, one input's size.
The output is a standard HF checkpoint: sharded safetensors plus an index above --max_shard_size (5 GB by default), or a single model.safetensors with no index when it fits.
Every non-weight file in --tokenizer_source is copied: config, tokenizer, the multimodal processor files, and any remote-code modules the config's auto_map names. Two are rewritten after the copy: the saved config's dtype is set to the merged dtype, and the README.md model card gains the halo Hub tag, or is created holding it (Hub model card).
The resume state is the exception: trainer_state.json, scheduler.pt, router_balancing_biases.pt, reference_logps.pt, rng_state_*, prefetch_pending-* and a merged checkpoint's resume_adapter/ with its resume_adapter.json describe one training run, which an N-way merge has none of, so they are dropped. The merged model therefore ships no balancing sidecar — a family whose balancing bias lives in a checkpoint slot keeps it through the merged weights, while a bias_update_transient run's bias is gone (see Checkpoints).
Every input must carry the merge's reference key set (the base's under task_arithmetic/ties, else the first model's), nothing beyond it, and each key at one shape; a mismatch is refused from the safetensors headers before any output is written. Two kinds of base key no fine-tune stores are legitimate, both tensors the shipped config's model never reads from a checkpoint, and the merge leaves them out as every fine-tune export does:
- a Hub base's stored tied
lm_head.weight, when every fine-tune stores the embedding and the shipped config ties the pair at both levels (a multimodal wrapper's owntie_word_embeddingsand its text sub-config's). The load rebuilds the head from the merged embedding; - keys the shipped config's model class drops at load (
_keys_to_ignore_on_load_unexpected): themtp.*head of the Qwen3.5/3.6 Hub checkpoints, GLM-4.7-Flash'smodel.layers.47.*MTP layer, Nemotron-H'smtp.*orlanguage_model.mtp.*, which its composite's text class names. A text class's other drops stay refused, as its composite reads them: Qwen3.5'smodel.visual.*.
The merge keeps its inputs' key layout, so the result loads with from_pretrained and serves wherever its inputs would (serving rules). A --tokenizer_source that ships no tokenizer files raises once the weights are written, since no from_pretrained-based consumer could build a tokenizer from the output — re-point it at a directory that carries one, or pass --allow_missing_tokenizer to accept the tokenizer-less artifact.
slerp falls back to linear interpolation when the two tensors are near-(anti-)colinear (|cosine| > 0.9995), where spherical interpolation is numerically unstable and ≈ equal anyway.
Other merges
Weight-space merging is distinct from the reassembly/adapter merges:
- EP sharded checkpoints → reassemble with
scripts/after_training/merge_ep_shards.py. See Checkpoints & Resume. - LoRA / PEFT adapters → fold into the base with
scripts/after_training/merge_peft_adapters.py. A directory carrying native EP expert-LoRA (peft_type: EXPERT_LORAorLORA_WITH_EP_EXPERT_LORA, or*.experts.<attr>.lora_{A,B}keys), whichmerge_and_unloadwould silently drop, takes the streaming merge instead (as does--streamingfor any adapter too large to load);merge_expert_lora_on_save: truefolds them at save time. A base with fewer embedding rows than the adapter's tokenizer has ids grows to it first (PEFT → Merging adapters). See PEFT (LoRA).
Source on GitHubUpdated 18 November 2026