LoRA target modules

LoRA target modules are the layers of a model that get a LoRA adapter. They are listed by module name, such as q_proj and v_proj for attention, and every layer left off the list stays frozen with no adapter.

Which layers there are

A transformer block has two groups of linear layers, and LoRA can wrap any of them:

GroupCommon namesRole
Attentionq_proj, k_proj, v_proj, o_projquery, key, value and output projections
MLP (feed-forward)gate_proj, up_proj, down_projgate, up and down projections

Names differ between model families, so read them from the model before you write the list.

How to choose

  • q_proj and v_proj only. The smallest adapter. The original LoRA paper adapted these two.
  • Attention and MLP. More trained parameters at the same rank, and often closer to full fine-tuning.

More targets mean a larger adapter and more memory. To hold the size fixed, trade targets against rank. In the Hugging Face PEFT library, all-linear selects every linear layer except the output layer.

In Halo

The key is lora_target_modules. The Qwen3 4B LoRA example targets attention and MLP:

YAML

lora_target_modules: [q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj]

On mixture-of-experts (MoE) models, all-linear misses the fused expert tensors, so name the expert projections explicitly. LoRA is rejected under tensor parallelism. See LoRA and QLoRA in the Halo docs.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL