# What is PEFT?

PEFT (parameter-efficient fine-tuning) is a family of methods that train a small number of weights, usually added ones, while the pretrained model stays frozen. PEFT is also the name of the Hugging Face library that implements them.

## The main methods

Each of these freezes the pretrained weights and adds new ones in a different place:

| Method | What is trained |
| --- | --- |
| [LoRA](/glossary/lora) (low-rank adaptation) | low-rank matrices added beside existing weight matrices |
| Adapters | small bottleneck layers inserted inside each transformer block |
| Prompt tuning | a few learned vectors prepended to the input embeddings |

LoRA is the one most used with LLMs. [QLoRA](/glossary/qlora) is LoRA on a base model loaded in 4-bit.

## Why use it

- **Memory.** No gradients or optimizer state for the frozen weights.
- **Storage.** The output is a small file, not a copy of the model.
- **Swapping.** One base model serves many tasks by changing the small part.

The cost is capacity: see [LoRA vs full fine-tuning](/glossary/lora-vs-full-fine-tuning).

## The library

`peft` is the Hugging Face library that implements these methods. `get_peft_model` wraps a model, `save_pretrained` writes only the adapter, and `merge_and_unload` [folds a LoRA adapter into the base](/glossary/how-to-merge-a-lora-adapter).

## In Halo

Halo's supported runtime lists PEFT 0.18.x. Set `use_peft: true` and the `lora_*` fields to train a LoRA adapter instead of the full model. LoRA is rejected under tensor parallelism. See [LoRA and QLoRA](/halo/docs/training-methods/sft#lora-and-qlora) in the Halo docs.

