To merge a LoRA adapter, load the base model, load the adapter on top of it, fold the adapter's weights into the base and save the result. The output is one ordinary checkpoint that needs no adapter at inference.
Why merge
A LoRA run saves only the adapter. Merging adds (alpha / r) · B · A to each weight the adapter sits beside. The result is a standalone model:
- One checkpoint to ship and serve, in the base model's format.
- No adapter math at inference, so no added latency.
Do not merge if you serve several adapters on one base and swap them per request. Merging also gives up the small file: the output is as large as the base.
Steps
With the Hugging Face PEFT library:
- Load the exact base model the adapter was trained on. For a QLoRA adapter, load it in 16-bit, not 4-bit.
- Load the adapter on top with
PeftModel.from_pretrained. - Call
merge_and_unload(). It returns a plain Transformers model. - Save it with
save_pretrained, and save the tokenizer beside it. - Compare a few outputs with the unmerged model.
Python
from peft import PeftModel
from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained("base-model")
model = PeftModel.from_pretrained(base, "path/to/adapter")
model.merge_and_unload().save_pretrained("merged-model")With Halo
halo run merge-peft-adapters folds a LoRA adapter into its base as one checkpoint. Give it an output path different from its input: the tools refuse to run in place. Like every Halo tool, it answers -- --help with its flags. It refuses EP (expert-parallel) expert-LoRA adapters. Train those with merge_expert_lora_on_save: true to save the merged model instead. See post-processing tools.
