Model Merging

Combine several fine-tuned checkpoints of the same architecture into one model in weight space — no extra training, no inference-time ensembling. scripts/after_training/merge_models.py is the self-contained merger (no mergekit dependency) and supports four methods:

MethodWhat it doesModelsNeeds --base_model
linearWeighted average of the weights (always divided by the weight sum, so per-model :weight values are relative)≥2No
slerpSpherical interpolation between two checkpointsexactly 2No
task_arithmeticbase + Σ wᵢ·(modelᵢ − base) — sum task vectors onto a base≥2Yes
tiesTrim each task vector to its top-density magnitudes, elect a per-parameter sign, average sign-agreeing deltas, add λ× onto the base (arXiv:2306.01708)≥2Yes

All checkpoints must share the architecture: every model has to provide each parameter key at the same shape, or the merge raises — as does a model carrying a key the others do not.

Floating-point tensors are merged in float32 and cast to --dtype (default bfloat16), except router-balancing tensors, which keep their source dtype (a bf16 cast quantizes away the fp32 sign steps the balancing update writes). Non-float tensors (routing tables, expert-count buffers) are copied through unchanged, and a disagreement between models raises rather than inventing an average.

Usage

Bash

# Weighted linear average (weights are normalized by their sum)
python scripts/after_training/merge_models.py --method linear \
    --models /ckpt/run-a:0.3 /ckpt/run-b:0.7 --output_dir <large-volume>/merged-linear

# SLERP halfway between two checkpoints
python scripts/after_training/merge_models.py --method slerp --t 0.5 \
    --models /ckpt/run-a /ckpt/run-b --output_dir <large-volume>/merged-slerp

# TIES onto a base
python scripts/after_training/merge_models.py --method ties --density 0.6 --lambda 1.0 \
    --base_model /ckpt/base --models /ckpt/run-a:0.5 /ckpt/run-b:0.5 --output_dir <large-volume>/merged-ties

Each --models entry is a local directory, optionally suffixed :weight (split on the last colon, so absolute paths are safe); the default weight is 1.0. Weights are read off disk, so a Hub id there raises FileNotFoundError; --tokenizer_source is a local directory too (a Hub id is refused, so the local --trust_remote_code default covers every source — download the repo first).

FlagDefaultApplies toMeaning
--models—allLocal model dirs, each optionally path:weight (the suffix is rejected under slerp, which reads no weights)
--output_dir—allWhere to write the merged HF checkpoint. Must be a new directory — it is refused if it matches any source it reads from (--models, --base_model, --tokenizer_source), since the write deletes the model*.safetensors it does not own
--methodlinearalllinear / slerp / task_arithmetic / ties
--base_model—task_arithmetic, tiesModel the task vectors are relative to (required)
--dtypebfloat16allOutput dtype (float16 / bfloat16 / float32; short aliases fp16/bf16/fp32 accepted)
--density0.6tiesFraction of each task vector's deltas kept, in (0, 1]
--lambda1.0tiesScale of the merged delta onto the base; any finite value
--t0.5slerpInterpolation factor, limited to [0, 1]
--tokenizer_sourcebase, else first modelallLocal directory to copy config + tokenizer/processor from
--max_shard_size5GBallPer-file cap for the output safetensors shards
--allow_missing_tokenizeroffallAccept a merged checkpoint whose --tokenizer_source ships no tokenizer files (refused by default)
--trust_remote_codeonallTrust remote code when re-reading the merged config (--no-trust_remote_code to disable); needed for Bailing/Ling
--quietoffallSuppress per-key progress logging

The Applies to column is enforced, not advisory: a knob passed explicitly for a method that does not consume it is rejected before any I/O, so a refused merge leaves no output directory behind. Defaults resolve after that gate, so naming a knob at its documented default still counts as passing it. A value outside the knob's range, or a non-finite :weight, is rejected at the same point: a --density outside (0, 1] would keep every delta, and a --t outside [0, 1] would extrapolate past both models.

Memory and output

Merging runs on CPU and streams the inputs one tensor at a time — each key is read from every model, merged, then released — so peak host memory is set by the costliest tensor (the largest, in practice), never the merged model: every input's copy of it as stored, the method's fp32 working copies of it, and the writer's staged output parts: the shard it is filling plus one per write thread (--max_shard_size each, HALO_CHECKPOINT_WRITE_THREADS, default 0). linear, task_arithmetic and slerp hold 3, 4 and 5 working copies whatever the model count; ties holds 5.25 per model plus 5 (each delta, their stacked copy, the sign-masked copy, and the agreement mask with its int64 cast), so a TIES merge of four bf16 models onto a bf16 base, whose largest tensor has 1B elements, peaks near 120 GB. The tool warns before merging when this estimate exceeds the host's available RAM. Peak disk is the merged artifact, one input's size.

The output is a standard HF checkpoint: sharded safetensors plus an index above --max_shard_size (5 GB by default), or a single model.safetensors with no index when it fits.

Every non-weight file in --tokenizer_source is copied: config, tokenizer, the multimodal processor files, and any remote-code modules the config's auto_map names. Two are rewritten after the copy: the saved config's dtype is set to the merged dtype, and the README.md model card gains the halo Hub tag, or is created holding it (Hub model card).

The resume state is the exception: trainer_state.json, scheduler.pt, router_balancing_biases.pt, reference_logps.pt, rng_state_*, prefetch_pending-* and a merged checkpoint's resume_adapter/ with its resume_adapter.json describe one training run, which an N-way merge has none of, so they are dropped. The merged model therefore ships no balancing sidecar — a family whose balancing bias lives in a checkpoint slot keeps it through the merged weights, while a bias_update_transient run's bias is gone (see Checkpoints).

Every input must carry the merge's reference key set (the base's under task_arithmetic/ties, else the first model's), nothing beyond it, and each key at one shape; a mismatch is refused from the safetensors headers before any output is written. Two kinds of base key no fine-tune stores are legitimate, both tensors the shipped config's model never reads from a checkpoint, and the merge leaves them out as every fine-tune export does:

  • a Hub base's stored tied lm_head.weight, when every fine-tune stores the embedding and the shipped config ties the pair at both levels (a multimodal wrapper's own tie_word_embeddings and its text sub-config's). The load rebuilds the head from the merged embedding;
  • keys the shipped config's model class drops at load (_keys_to_ignore_on_load_unexpected): the mtp.* head of the Qwen3.5/3.6 Hub checkpoints, GLM-4.7-Flash's model.layers.47.* MTP layer, Nemotron-H's mtp.* or language_model.mtp.*, which its composite's text class names. A text class's other drops stay refused, as its composite reads them: Qwen3.5's model.visual.*.

The merge keeps its inputs' key layout, so the result loads with from_pretrained and serves wherever its inputs would (serving rules). A --tokenizer_source that ships no tokenizer files raises once the weights are written, since no from_pretrained-based consumer could build a tokenizer from the output — re-point it at a directory that carries one, or pass --allow_missing_tokenizer to accept the tokenizer-less artifact.

slerp falls back to linear interpolation when the two tensors are near-(anti-)colinear (|cosine| > 0.9995), where spherical interpolation is numerically unstable and ≈ equal anyway.

Other merges

Weight-space merging is distinct from the reassembly/adapter merges:

  • EP sharded checkpoints → reassemble with scripts/after_training/merge_ep_shards.py. See Checkpoints & Resume.
  • LoRA / PEFT adapters → fold into the base with scripts/after_training/merge_peft_adapters.py. A directory carrying native EP expert-LoRA (peft_type: EXPERT_LORA or LORA_WITH_EP_EXPERT_LORA, or *.experts.<attr>.lora_{A,B} keys), which merge_and_unload would silently drop, takes the streaming merge instead (as does --streaming for any adapter too large to load); merge_expert_lora_on_save: true folds them at save time. A base with fewer embedding rows than the adapter's tokenizer has ids grows to it first (PEFT → Merging adapters). See PEFT (LoRA).

Source on GitHubUpdated 18 November 2026