PEFT (LoRA) Fine-Tuning

LoRA trains small rank-decomposed adapter matrices while the base model stays frozen, eliminating optimizer states for frozen parameters.

It runs under DDP/FSDP, EP, CP and pure ETP; TP, EP+TP and PP reject it, and QLoRA runs only under DDP/FSDP and CP-on-dense (full matrix and reasons: Parallelism compatibility). Under EP, LoRA targets both attention (via PEFT) and the MoE experts (via native grouped adapters).

Offline GRPO under CP is full fine-tuning only and rejects every adapter shape; SFT and SMPO keep their CP adapter support. Most other configurations take it. The embedding trainer accepts LoRA and DoRA, not QLoRA (Embedding — PEFT / LoRA).

Supported methods

Uses the PEFT library (pinned 0.18.1). LoRA is the only method with tested parallelism support; other PEFT methods (AdaLoRA, Prompt Tuning, (IA)3) reach the trainer through a custom peft_config but are untested with it. An AdaLoRA or (IA)3 layer is refused at construction wherever an out-of-place fold runs (the weight-sync trainers, embedding LoRA, merge_expert_lora_on_save): the fold covers plain LoRA and DoRA only (Online RL).

MethodEnableNotes
LoRAuse_peft: trueDefault low-rank adaptation
QLoRALoRA + BitsAndBytes 4-bit/8-bit baseLoRA on a quantized base
DoRAuse_dora: trueWeight-decomposed (magnitude + direction). Attention only — rejected alongside expert projections, which have no grouped DoRA
rsLoRAuse_rslora: truealpha/sqrt(r) scaling, for high ranks. Applies to attention and native expert adapters

Configuration

Fields come from TRL's ModelConfig and are converted to a LoraConfig via get_peft_config():

YAML

use_peft: true
lora_r: 64                    # Rank (TRL default 16)
lora_alpha: 128               # Scaling factor (alpha/r applied to adapter output)
lora_dropout: 0.05
lora_task_type: CAUSAL_LM     # CAUSAL_LM | SEQ_CLS | FEATURE_EXTRACTION
lora_target_modules: [q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj]

lora_target_modules is tri-state: unset (None) falls to PEFT's own per-architecture table, which has no entry for any MoE or multimodal family here (PEFT raises Please specify target_modules) and adapts only q_proj/v_proj on the dense ones it lists (Qwen3, Llama, Mistral), unscanned by the exclusion below — so name the modules; a populated list selects those modules. Expert-LoRA-only means listing only expert projections — the peel then empties the list, which is what disables attention PEFT. Writing lora_target_modules: [] yourself names nothing at all and is rejected: with no adapter to create, the run would full-finetune at the LoRA learning rate.

The spelling decides how a target matches. A list names modules at any length: [o_proj] adapts every module whose path is o_proj or ends in .o_proj (whole trailing segments, so [proj] does not reach o_proj). A bare YAML string is PEFT's own spelling, all-linear or a regex fullmatched against each full module path ('.*\.0\.self_attn\.o_proj'), so a bare o_proj matches nothing. The command line always gives a list (--lora_target_modules=o_proj names the module), so a regex goes in the YAML; a list entry spelling a regex (\ * + ? ( ) [ ] { } | ^ $) is refused at parse time, use_peft: false included, since PEFT compares list entries to paths verbatim and it would match nothing, unnoticed beside an entry that does match. ModelArguments (halo/configs/script_arguments.py) keeps a one-entry list a list, where TRL's ModelConfig collapses it to its bare entry; build_peft_config turns [all-linear] into PEFT's bare-string sentinel, and a lone expert projection ([experts], or a bare experts) still goes to the expert peel.

Targets PEFT cannot adapt

A stock LoRA adapter decomposes the matched module's own weight and adds the delta to that module's output, which leaves two module shapes out of its reach:

ShapeExampleWithout the exclusion
A wrapper holding its weight in a child moduleGemma 4's Gemma4ClippableLinear, the vision and audio towers' projectionsinjection raises Target module ... is not supported
A layer subclass computing something other than its base layer's affine mapDeepSeek-V4's grouped o_a_proj, whose block-diagonal output is one group widethe delta has the wrong width and the first forward raises

Both are reachable by an ordinary target list — target names are matched by suffix over the whole module tree, so a multimodal tower is matched by the language model's q_proj…o_proj spelling — and by all-linear, which reaches every nn.Linear subclass a family defines.

build_peft_config (halo/model_loading/parallel/peft_setup.py) scans the live model with PEFT's own matcher and puts those modules in the LoraConfig's exclude_modules by full path (PEFT also honors an entry as a dotted suffix), so an identically named leaf elsewhere keeps its adapter. A warning names the count and one example path per reason. If it leaves no adaptable target at all, the run is rejected rather than trained with nothing — or, when native expert adapters are live, with the attention half missing. The verdict is agreed across ranks: a rank whose probe faulted would otherwise adapt a different parameter set and surface only as a hang.

The two shapes are told apart structurally: a wrapper owns no parameter of its own, and a subclass overriding nn.Linear.forward is measured once per class and geometry against F.linear's output shape — the low-precision compute drop-in and a quantized base change only how the map is computed and keep their adapters. A module of a type PEFT has no adapter layer for at all (Inkling's rel_logits_proj) is matched by no recipe's names and still raises PEFT's own Target module ... is not supported when named. The exclusion is saved into adapter_config.json, so reloading the adapter onto the same base loaded through the same class reproduces the same injection; resume rebuilds it from the live model instead and never reads that file. The merge tools pick that class off the adapter's keys (Merging adapters). An exclusion found on the context-parallel wrapped tree is written in the plain module spelling, as the adapter's tensor keys already are, so a plain reload and the merge tool honor it; the live config keeps the wrapper's spelling for the wrapped model.

A tower whose projections are plain nn.Linear (CLIP/SigLIP/Pixtral-style) is untouched by any of this and keeps its adapters; on text-only data those adapters receive no gradient and stay at their zero initialization, costing optimizer state and nothing else. On an image row they train under either checkpointing mode, at the cost of recomputing the tower in the backward (Tower blocks).

One consequence for a mistyped target: a name that matches a container rather than a leaf (mlp on Qwen3) drops out of the injection with that warning, so the run trains the targets that did match. Read the excluded count — it is the tell.

Hyperparameters

  • Rank (lora_r): 8–32 for 1B–3B, 32–96 for 7B–14B, 64–384 for 27B+. With a tuned learning rate, performance is largely rank-independent; rank matters most when the LR is suboptimal.
  • Alpha: alpha = rank (conservative) or 2 * rank (aggressive). For rank ≥ 64, prefer use_rslora.
  • Target modules: all-linear on a dense model or on stock experts (use_grouped_gemm: false). On EP layers it runs on GPT-OSS, Qwen3 MoE, Gemma 4, Inkling and LFM-2, without reaching the fused expert tensors. Every other EP family refuses it, as it would put PEFT layers inside the EP layer. There, name attention and expert projections explicitly (see MoE models).
  • Learning rate: higher than the same run's full fine-tune. The shipped SFT recipes start at 1e-4, and the code-contests recipes run 3e-6 against 2e-7.
  • Dropout: honored on SFT. The preference, reward and offline-GRPO trainers run TRL's disable_dropout_in_model after the adapter wrap (disable_dropout defaults to True), zeroing PEFT's lora_dropout along with the model's own for reference-vs-policy determinism. A warning fires when a configured dropout was zeroed; disable_dropout: false keeps it. Leaving lora_dropout out keeps TRL's 0.05 default, so lora_dropout: 0.0 is the setting that states no dropout.
TrainerFull-FT bandLoRA start
SFT2e-6 – 2e-51e-4
SMPO / DPO5e-7 – 5e-65e-5
GRPO1e-6 – 5e-63e-6 (every shipped LoRA async GRPO recipe under examples/grpo/environmental/)

The effective-batch formula is unchanged by LoRA — see SFT.

Ready configs: examples/sft/qwen3/qwen3-4b-ultrachat-lora.yaml (and its -qlora counterpart, which adds load_in_4bit: true and use_bnb_nested_quant: true over TRL's default bnb_4bit_quant_type: nf4).

MoE models — expert targets and full-trained modules

MoE experts are fused nn.Parameter tensors (GptOssExperts.gate_up_proj / down_proj, and the equivalents on other families), not nn.Linear, so PEFT's all-linear cannot reach them; on gpt-oss it resolves to the attention q/k/v/o_proj only. Name the expert projections explicitly instead. split_expert_lora_targets (halo/model_loading/parallel/peft_setup.py) peels those entries out of the list and routes them to native grouped-LoRA built inside each EP layer (_init_expert_lora in halo/distributed/expert_parallel/moe_layer/expert_lora.py); the remaining attention names stay on stock PEFT.

lora_target_modules entryRequests
gate_proj / up_proj / down_projthat logical projection
gate_proj_gmm / up_proj_gmmthe same, in the grouped-GEMM storage spelling
gate_up_projthe fused gate+up projection
experts / mlp.expertsall three expert projections

Requests are logical; what gets built follows the family's storage. One adapter is created per stored 3-D expert tensor that any requested projection touches.

On a family that stores the fused gate_up_proj — every family except Qwen3, Bailing, Nemotron-H and grouped-GEMM GptOss (below) — asking for gate_proj alone adapts the whole [E, H, 2M] tensor: gate and up share one rank-r subspace. On the separate-storage layouts it adapts gate alone, with its own rank r. Nemotron-H's experts store up_proj and down_proj only, so a gate-only request builds no adapter and raises (reject_unrealized_expert_lora).

lora_r must be a multiple of 8. The grouped GEMM reads the adapter's rank dimension as a stride that spans a multiple of 16 bytes (GROUPED_MM_STRIDE_ALIGNMENT_BYTES), and the adapter GEMMs run at the activation dtype, bf16 at the narrowest, whatever the experts are stored in; the EP layer refuses any other rank as it builds the adapters, where the first expert matmul would otherwise fail with strides should be multiple of 16 bytes.

GptOss picks its layout at runtime: with grouped-GEMM (SM90+, the default) it stores gate_proj_gmm/up_proj_gmm separately, without it a fused interleaved tensor. The same YAML therefore yields a different adapter, and resume across that switch is refused.

The peel is name-based, not module-based. gate_proj/up_proj/down_proj also spell the plain nn.Linear MLPs of a hybrid MoE — dense prefix layers (first_k_dense_replace) and shared experts — and those are adapted by neither half once the name is peeled. A warning names what moved.

Shared-expert LoRA has no supported route under EP: named generically it is peeled; named explicitly (shared_experts.down_proj) PEFT wraps a Linear inside the EP layer and reject_peft_lora_in_ep_layers rejects it.

Do not reach for TRL's lora_target_parameters on an MoE model — it is rejected on any model carrying EP layers (ep_size > 1, or the default use_grouped_gemm), whatever it targets, so an attention-only run is rejected too. It is PEFT's own mechanism for fused nn.Parameter experts, and it wraps the module owning the parameter, which sits above the EP layer and so escapes both EP validators.

Weights PEFT can't LoRA are full-trained (a trainable copy, not a low-rank adapter) via lora_modules_to_save: embed_tokens (an nn.Embedding), lm_head (excluded by all-linear), and the MoE router / gate. An expert-only run builds no PeftModel, so lora_modules_to_save raises there rather than leaving those modules silently frozen — add an attention target, or drop the field. unfreeze_layers_patterns / freeze_layers_patterns (halo/model_loading/parallel/freeze_patterns.py) are refused on any adapter run (attention LoRA or native expert LoRA): it freezes every base parameter and trains only the adapters, so a pattern would select nothing. Use lora_modules_to_save for a fully trained module.

Router training under EP

The router/gate is absorbed into the EP module for every family but Gemma4, whose router is a sibling module the wrapper never owns. EP modules are FSDP-ignored, so their gradients are DP-averaged by a hook the EP layer registers at construction on the original router param, one that never sees the trainable copy modules_to_save creates.

EPMoELayerBase.reattach_dp_average_grad_hooks re-attaches it to the live copy automatically, so router in lora_modules_to_save stays DP-consistent in every EP mode; the ep1 FSDP-sharded path (fsdp_shard_ep1_experts) and the multi-node deferred sweep already cover it.

modules_to_save on any other EP-internal submodule is unsupported: reject_unsynced_trainable_params raises naming any trainable param left inside an EP module with no gradient sync — it would otherwise silently drift across DP ranks. Its sibling reject_peft_lora_in_ep_layers (halo/trainers/parallelism/adapter_rules.py) rejects stock-PEFT LoRA layers placed inside EP modules.

Example: gpt-oss LoRA under EP

YAML

lora_target_modules:            # attention → PEFT; expert projections → native grouped-LoRA
- q_proj
- k_proj
- v_proj
- o_proj
- gate_up_proj                  # fused gate+up expert projection
- down_proj
lora_modules_to_save:           # full-trained (PEFT can't LoRA these)
- embed_tokens
- lm_head
- router                        #   DP-sync hook re-attached automatically under EP

examples/grpo/environmental/gptoss/vllm/gptoss-20b-code-contests-lora-ep4.yaml is a full Environmental-GRPO version, with the router left frozen.

Adapter dtype under FSDP: get_peft_model upcasts adapters (and modules_to_save copies) to fp32, but FSDP2 requires a uniform dtype per shard group. _cast_peft_params_to_compute_dtype casts trainable fp32 PEFT params to the compute dtype before wrapping (to fp32 instead under fp32_non_ep_params). EP params are excluded by identity, before the dtype test: downcasting a deliberately-fp32 router or expert would negate fp32_router / fp32_experts. They are FSDP-ignored unless FSDP2 manages the ep1 experts (fsdp_shard_ep1_experts at ep_group_size == 1); there an fp32 router copy takes a shard group of its own (Data Parallelism).

Parallelism compatibility

ModeLoRAQLoRATarget restrictionNotes
DDP / FSDPYesYesNoneStandard path; QLoRA on a MoE needs use_grouped_gemm: false (QLoRA)
HSDPYesNoNonePure DP or CP only, so expert adapters train at ep_size: 1; every replica starts from the first one's adapters (HSDP). QLoRA rejected: use_hsdp is refused under QLoRA, which skips FSDP2 (QLoRA)
EPYesNoAttention + expertsAttention via PEFT; expert FFNs via native grouped adapters. QLoRA rejected: EP loaders materialize plain de-quantized weights, losing Params4bit
CPYesYes*NoneCP shards the sequence, not the weights — adapters stay replicated; standard from_pretrained preserves Params4bit. *QLoRA only on a dense model: an MoE takes the grouped-GEMM loader, which rejects a quantized base
TPNoNo—Rejected — adapters are plain tensors outside the TP graph
EP+CPYesNoAttention + expertsBoth active
EP+TPNoNo—Both adapter kinds rejected: attention LoRA as under TP, native expert LoRA by the gate's model_has_expert_lora arm
ETPYesNoAttention onlyExpert adapters rejected at config time by ParallelismConfig (expert_tp_size > 1 gives the replicated adapter half a partial, never-synced gradient)
EP+ETPYesNoAttention onlyExpert adapters rejected at config time, as under ETP
PPNoNo—Rejected, as a peft_config or an already PEFT-wrapped model — the adapter save and resume are not stage-aware (PP rejected combinations)

EP. The expert names above route to native grouped LoRA — grouped [E_local, K, r]/[E_local, r, N] adapters stored alongside each expert weight, applied in the grouped-GEMM compute and gradient-synced across the EP group like the experts themselves. lora_B is zero-initialized, so the initial delta is zero. The frozen base experts stay bf16.

The default save writes a standalone adapter; merge_expert_lora_on_save: true folds the delta into the base for a servable HF checkpoint instead, keeping the unmerged adapter beside it for resume (Merging adapters).

TP / EP+TP. For a colwise-sharded base, lora_B becomes a per-rank output shard while lora_A stays replicated. Nothing broadcasts the replicated matrix (it diverges from init) and nothing distinguishes the sharded one from a replica (the TP replicated-grad sync averages and corrupts it), so the trained adapter is rank-inconsistent and will not reload onto a non-TP model. Use FSDP/DP, CP, or pure ETP instead.

The gate is reject_lora_under_tp (halo/trainers/parallelism/adapter_rules.py) and it refuses both adapter kinds: a PeftModel or any PEFT tuner layer injected in place (read off the tuner layers, so a backbone's own lora_* weights stay base weights), and — checked first, via model_has_expert_lora — the native grouped expert adapters, which are no tuner layer and which the TP replicated-grad sweep skips by param identity. expert_tp_size > 1 rejects expert LoRA earlier still, at config time in ParallelismConfig, before the checkpoint downloads.

Measured cost

Both tables measured on the Blackwell image, Halo 1.1.0 or earlier.

Dense — 1× B300 (SM103), DistributedSFTTrainer, AdamWBF16, Liger, FA4, Qwen3-8B, seq 16384, BS=1, GC, 10 steps / 3 warmup (full fine-tuning at this shape: Liger → Benchmarks):

ConfigTrainabletokens/s/GPUPeak memory
LoRA r=64, attn only61M (0.7%)18,89734.3 GiB
LoRA r=64, all linear175M (2.1%)16,20335.0 GiB
QLoRA r=64, all linear (NF4 base)175M (2.1%)15,41025.3 GiB

Attention-only runs 18% faster than full fine-tuning (15,966) at about half its memory; all-linear is ~14% slower than attention-only, since adapter matmuls run on every MLP layer. Throughput is rank-invariant within a variant (attn-only ~18.8k, all-linear ~16.0k across r=16/64/128): the frozen base forward/backward dominates the step, and only memory grows with rank.

QLoRA saves ~10 GiB more than bf16 all-linear LoRA at ~5% lower throughput. It is the path onto a smaller GPU, since plain LoRA needs ~34 GiB at this shape even at the minimum rank.

MoE — same setup on 8× B300, gpt-oss-20b (32 experts, top_k=4) at EP=2, seq 4096, r=64 (full fine-tuning at this shape: Throughput Benchmarks → EP-only):

ConfigTrainabletokens/s/GPUPeak memory
LoRA r=64, attn only (PEFT)32M (0.28%)16,16629.6 GiB
LoRA r=64, experts only (grouped)425M (3.60%)17,00231.9 GiB
LoRA r=64, attn + experts457M (3.86%)14,31632.0 GiB

LoRA under EP is far leaner than full fine-tuning. The frozen base carries no gradients or optimizer state, so every variant peaks at 30–32 GiB against full fine-tuning's 78.9 GiB at this shape (batch 1), and every variant runs faster than its 12,432 tok/s/GPU. Experts-only is 5% faster than attention-only, since the grouped expert adapters fold into the grouped-GEMM compute; attn + experts is slower than either alone.

On qwen3-30b-a3b (128 experts) at EP=2, experts-only r=64 is 9.39% trainable at 9,894 tok/s/GPU and 46.7 GiB. At batch 1 tok/s/GPU varies ±10% run-to-run.

Reference model handling

Non-CP offline GRPO and Async GRPO with Environments disable the adapter (disable_adapter()) to compute reference log-probs from the frozen base instead of a second model copy. Under EP the mixin patches that context (install_ep_aware_disable_adapter) so it reverts the native EP expert adapters too, giving a true frozen-base reference for both adapter halves; the patch also covers TRL's use_adapter(None) for online GRPO / DPO / KTO.

The pinned PEFT 0.18.1 clears requires_grad on the adapters a module holds when disable_adapter() enters and restores it on the ones it holds at exit. A reference pass that is the first forward after an FSDP2 reshard (online GRPO at beta > 0 with no old-logps recompute) enters on the sharded params and exits on the unsharded copies, which would freeze every sharded adapter silently. install_fsdp2_safe_disable_adapter wraps the context for every PEFT model: when the registered trainable params changed across it, it reshards before PEFT's exit, so the restore lands on the sharded params (upstream PEFT drops the freeze in huggingface/peft#3839, which 0.18.1 predates).

An explicit reference model (DPO, KTO, self-distillation's KL reference_model) is never parallelized: under EP or TP it is a whole dense replica on every rank, warned about rather than refused. LoRA with the adapter disabled, or precomputed reference log-probs, avoids it. An expert-only LoRA run builds no PeftModel, so nothing switches its adapters off: DPO and KTO need precompute_ref_log_probs there, and offline GRPO loads a frozen base ref_model. Per trainer: DPO → Reference model, Offline GRPO → Reference model. SMPO is reference-free.

On online / async GRPO, where no adapter wraps the model — a full fine-tune, or an expert-only LoRA run, which builds no PeftModel — beta != 0 holds a frozen reference: the script loads the base the way the policy loads and the trainer hands it to TRL in place of the fp32 copy TRL would build (Online GRPO). It is an unparallelized replica per rank, refused before it loads when it exceeds half the free GPU memory or 80% of the node's available host RAM across its ranks. Add an attention LoRA target (it wraps the model, and the disabled adapter is the reference) or set beta: 0 to hold none.

Online RL — rollout-server weight sync

Online and Async GRPO with Environments generate rollouts from a separate rollout server — vLLM or SGLang (Rollout Servers) — which loads no adapter unless rollout_adapter_sync serves the policy as one on SGLang (Adapter sync).

Otherwise each NCCL weight sync forwards every base weight with its adapter folded in, under base-model param names (PEFT prefixes stripped, lora_* params skipped). Without the fold, the server would generate from the un-adapted base.

The fold is out of place, one tensor at a time as the sync sends it (lora_folded in halo/models/peft_adapters.py, lora_folded_data in halo/distributed/lora_fold.py): the value PEFT's merge_adapter would write on the same placement, bit for bit, on Linear, Embedding, Conv and target_parameters layers, with lora_bias and DoRA. A Conv adapter on an FSDP2 shard folds on whole tensors (a DTensor convolution would run on each rank's slice of the adapter rank) and reshards the result, which holds the whole base and adapter tensors on every rank, so it suits the small convolutions of a patch embedding. A fold on FSDP2 shards can differ from a single-device merge by rounding, since the delta's contraction is summed across shards. The frozen base is never written: PEFT's in-place merge followed by its unmerge would not give it back, since in bf16 (w + d) - d misses w by a rounding step wherever the two roundings do not cancel. The fold holds one tensor's temporaries at a time, so it adds at most one local shard of the largest adapted tensor (a full copy on one device) to the sync's peak. A PEFT layer the fold does not cover (LoRA on nn.MultiheadAttention, trainable tokens, quantized LoRA, a variant other than DoRA, a grouped conv) is refused when the trainer is built, and so is a PEFT adapter on a tensor the sync's dense push does not send (an EP layer's expert weights, which take native expert LoRA), since only that push folds PEFT adapters. Each sync checks again: the layers before it streams, the targets as its dense push ends.

Both trainers share this path (gather_and_send_weights in halo/trainers/grpo/rollout/weight_push.py), which also folds in the EP / FSDP2 / TP gathers; the fold is a collective on all ranks under FSDP2.

Embedding models

The embedding trainer injects LoRA into its backbone in place, on data parallelism only, and on a MoE whose family has an EP layer only at use_grouped_gemm: false (its stock experts): Embedding — PEFT / LoRA.

Quantized training (QLoRA)

QLoRA combines LoRA with 4-bit/8-bit base quantization under DDP/FSDP (a MoE base whose family has an EP layer class also needs use_grouped_gemm: false), and under CP on a dense model. CP keeps the standard from_pretrained loader, which preserves Params4bit. GPU rows train dense Qwen3-0.6B, and gpt-oss-20b with stock experts through a save and reload. AdamWBF16 optimizes only the bf16 adapter params; the frozen quantized base is skipped.

load_distributed_model raises for EP, TP, and the grouped-GEMM MoE loader — pure ETP included, since the gate reads ep_group_size = ep_size × expert_tp_size. Those loaders materialize plain de-quantized weights, so Params4bit are lost and PEFT's 4-bit adapter dispatch fails.

The 4-bit compute dtype follows the run's own precision (bf16/fp16 on the training config), not TRL's ModelConfig.dtype — whose "float32" default nothing else here reads, and which would otherwise dequantize and compute every 4-bit matmul in fp32.

QLoRA is SFT/offline territory, but offline GRPO rejects it under CP. The online and async GRPO trainers reject a quantized base at construction (reject_unsyncable_model). The NCCL weight sync forwards raw parameter storage under base-weight names, so a bnb-packed 4-bit base would ship non-floating-point tensors that corrupt the served policy. Plain LoRA is the supported RL adapter path — see Online GRPO.

Under torchrun data parallelism the 4-bit base cannot be FSDP2-sharded (fully_shard rejects the non-float Params4bit), so the trainer skips FSDP2 and all-reduces the (small) adapter gradients across ranks, with the frozen base replicated per rank.

Nothing wraps the model, so the FSDP2 sharding knobs cannot take effect: a non-default use_hsdp, fsdp_reshard_after_forward, fsdp_reshard_after_backward or fsdp_defer_grad_sync is rejected at trainer construction rather than silently ignored, naming the offending flags. QLoRA under CP takes the same gate. Under accelerate launch with the DDP config (multigpu_dp_config.yaml), the other QLoRA launcher, accelerate owns the wrap and those knobs only warn; its FSDP2 configs cannot flatten the 4-bit tensors.

Pass a BitsAndBytesConfig through load_distributed_model(..., quantization_config=bnb_config), or pass a pre-loaded quantized model plus peft_config to the trainer. For 8-bit optimizer states without a 4-bit base, set optim: adamw_8bit.

FusedLinearCrossEntropy for long sequences

Liger FLCE fuses lm_head with the loss instead of materializing the batch × seq × vocab logits. Qwen3-8B QLoRA r=32 all-linear at 32k (BS=1, GC, Liger, FA4, on one B300, Halo 1.1.0 or earlier) peaks at 43.5 GiB with plain CE and 20.9 GiB with FLCE, so by peak it fits a 24 GB GPU only with FLCE. FLCE is SFT-only and disables entropy logging.

Checkpoint saving

save_model() saves only adapter weights (~MB). Which saver writes them, from which rank and with which collective under each mode: Checkpoints. Two layouts are specific to LoRA:

  • An expert-only run has no PEFT model at all, so save_expert_lora_adapters writes the adapter with a synthesized config: peft_type: EXPERT_LORA, the base model path, and every field the delta depends on (r, lora_alpha, lora_dropout, use_rslora, expert_projections).
  • A run that grew its vocabulary (added_special_tokens past the checkpoint's padded rows) saves the grown embedding and an untied head with the adapters, offline included, and resumes them; accelerate-managed FSDP and LoRA on one layer of an untied embedding pair are refused for such a run (Checkpoints). An expert-only run has no adapter to carry them: its resume grows the frozen rows again, and merge_expert_lora_on_save exports them whole. The merge grows the base to match (Merging adapters).
  • Under CP the adapter keys drop the wrapper's path segments (.original_attention., extra .model.) so they load onto a non-CP model. The module paths in the written adapter_config.json (exclude_modules, and target_modules once all-linear made PEFT rewrite it to full paths) are respelled the same way, while the live config keeps the wrapper's spelling.

Reloading elsewhere. Attention adapters carry standard PEFT keys and load onto a stock PeftModel.from_pretrained. Native expert adapters do not: grouped [E, K, r] tensors have no PEFT representation, and PEFT drops unrecognized adapter keys without a warning (load_adapter filters missing_keys and never reads unexpected_keys).

Every directory carrying them is therefore labeled with a peft_type PEFT does not know, so an external load raises on the label rather than returning a half-adapted model: EXPERT_LORA for an expert-only save, LORA_WITH_EP_EXPERT_LORA for a mixed attention+expert one.

The mixed label keeps every LoraConfig field beside it, plus an ep_expert_lora block recording the expert half's r/lora_alpha/lora_dropout/use_rslora/expert_projections; resume reads the tensors directly and the file only for its scaling check (below).

The in-memory merge and convert paths check those markers (reject_expert_lora_adapter) and fall through to a tensor-key scan when the config is absent or carries a stock peft_type, so an unmarked directory holding .experts.<attr>.lora_{A,B} keys is refused too; merge_peft_adapters.py routes such an adapter to the streaming merge.

Resume: EP/CP rebuild the base with zero-initialized adapters at init, so trained adapters are restored from the checkpoint's adapter_model.safetensors (a merged checkpoint's resume_adapter/; not from the base reload) by restore_adapters (halo/checkpoint/parallel/adapters.py), which load_resume_weights calls; a single-process or DDP run's adapter-only checkpoint takes it too, keeping the adapters at their live dtype where PEFT's load_adapter would recast bf16 adapters to fp32. Before any copy, every rank matches the file to the tensors a save of the live model writes, key for key at the saved shape (PEFT's own selection for the attention adapters, modules_to_save and their buffers; the gathered layout for the expert adapters), and the verdict is joined: a missing, unexpected or reshaped tensor raises on every rank with nothing written. That covers resuming expert adapters into a run that does not build them (EP off, use_grouped_gemm: false, or the expert projections dropped from lora_target_modules), a run that builds them from a file without them, and a changed lora_r. A changed LoRA scaling raises too: resume compares the scaling inputs the saved adapter_config.json records (lora_alpha, use_rslora, use_dora, alpha_pattern, and the expert half's lora_alpha and use_rslora) with the live run's, since names and shapes would still match and the delta would silently rescale. A missing adapter_config.json raises too.

Under CP remap_cp_adapter_keys_to_live maps the saved, normalized keys back onto the live wrapper spelling, read off the same PEFT selection the save made, so every adapter-name spelling PEFT writes (lora_A.default.weight, modules_to_save.default.weight, lora_embedding_A.default, a DoRA magnitude) maps back; a key it does not produce fails the check above. See Checkpoints.

Merging adapters

Bash

python scripts/after_training/merge_peft_adapters.py \
    --adapter_dir path/to/adapter/checkpoint \
    --output_dir path/to/merged/model

Add --task classification --num_labels N for classification models. Handles causal LM, sequence classification, multimodal/VLM, and trust_remote_code architectures: --trust_remote_code defaults on for a local base and off for a Hub base (Scripts). Use --device_map auto for large models.

The base loads through the class the adapter's tensor keys address: an adapter from a text_only_model: true run (keys under model.layers) loads the text-only *ForCausalLM, not the multimodal wrapper, and keys naming no module of either class are refused — applied to the wrong class, PEFT matches nothing, warns, and the tool would write the bare base as a finished model. --fp32_norms writes every normalization leaf in fp32, the layout convert_to_bf16.py writes.

A run whose added_special_tokens outgrew the checkpoint's embedding trained on rows no base holds. The merge (and convert_to_bf16.py --check_inference's adapter load) grows the base to the adapter directory's tokenizer under the run's own rule before PEFT loads the adapter, which carries those rows (Checkpoint saving): only when that tokenizer has more ids than the base has rows, and the base's own tokenizer fits them (a checkpoint whose tokenizer already outruns its rows loads unchanged). The added rows start as NaN, so none is served from uninitialized memory: an adapter saved before adapter saves carried them leaves some unset (the input embedding's, or an untied head's), and those are rebuilt with a warning as the mean of the checkpoint's rows. The run drew them by transformers' mean resizing and an adapter run keeps them frozen, so the rebuilt rows match the run's to within that draw (apply_adapter, halo/checkpoint/tools/adapter_merge.py).

The merge re-applies the run's training sidecars to the merged model and copies both files into the output: the GptOss sinks policy recorded in training_provenance.json, and router_balancing_biases.pt into the family's native routing-bias slots, so the merged model routes on the trained bias.

The in-memory merge refuses native EP expert-LoRA adapters (peft_type of EXPERT_LORA or LORA_WITH_EP_EXPERT_LORA, or tensor keys ending at .experts.<attr>.lora_{A,B}), because PeftModel.merge_and_unload silently drops the grouped expert deltas; convert_to_bf16.py --peft shares that refusal. A stock PEFT adapter on a per-expert nn.Linear MoE keys as .experts.<i>.<proj>.lora_A.default.weight and merges normally; only the grouped namespace is refused.

merge_peft_adapters.py folds such an adapter after the fact through its streaming merge, automatically for expert-only and mixed adapters. merge_expert_lora_on_save: true produces the merged servable checkpoint at training time instead, for expert-only and mixed runs alike. Each merged training checkpoint also keeps the unmerged adapter in resume_adapter/, which resume restores onto the base, so the run continues from its trained adapters rather than the fold (Merge-on-save checkpoints).

It needs native grouped expert adapters to exist: lora_target_modules must name at least one expert projection, or reject_unmergeable_expert_lora_save raises.

Both halves are folded: expert deltas inside each layer's gather_expert_state_dict, PEFT (attention) deltas out of place into each tensor as it is written, with the weight sync's fold (above). The frozen base is never written, so training continues unchanged. The fold happens in the gathered EP save under mixin-managed FSDP2 (torchrun); it is rejected under accelerate-managed FSDP and with save_sharded_ep: true.

Streaming merge

merge_peft_adapters.py --streaming (automatic for an adapter holding native expert LoRA, whose in-memory merge is refused) loads no model: it reads each tensor of the base's safetensors shards, folds the adapter into the ones it targets and writes the result through the streaming shard writer, so host memory is the adapter, one tensor, the fp32 expert deltas of the layer being folded and the writer's parts: about one expert layer's fp32 delta times a small factor (the preflight budgets 3x), which at 397B scale is tens of GB per layer. Linear LoRA (rslora, rank and alpha patterns, a LoRA bias) and DoRA fold through PEFT's own layer, so each tensor equals the in-memory merge; native expert LoRA folds scaling * bmm(A.float(), B.float()), the weight sync's and the merge-on-save's arithmetic, through the family's ExpertLayout, on any projection subset. modules_to_save tensors replace the base's. Tensors keep the base's stored dtype and key spelling, so --dtype is ignored (logged) and --device_map, --fp32_norms and --task classification are refused. The tool resolves every key the adapter rewrites against the base before the first write and raises, with no output, when one is absent (a text-only class's spelling, a fused or renamed layout, a tied head), when the base's experts are not in the family's export layout, when a stored tensor was resized (a grown vocabulary), on embedding LoRA, and on a training_provenance.json recording neutralized GptOss sinks; merge those in memory. The base's config.json and tokenizer files are copied, the adapter's tokenizer replaces them, and the adapter config lands in original_adapter_config/.

Adapters must train. Trainer construction runs verify_adapters_trainable (halo/model_loading/parallel/peft_setup.py) on the PEFT wrap before any mode wraps the model, so the check covers every mode alike. It raises when a tuner layer holds no trainable parameter for an adapter meant to train: an active adapter whose config is not inference_mode, counting every tensor in the layer's adapter_layer_names (lora_A/lora_B, lora_embedding_*, lora_magnitude_vector, other tuners' own). It reads live requires_grad and holds no collective; every rank builds the same wrap and raises alike. PEFT 0.21 and later re-enable adapters by matching named_parameters() names against named_modules() prefixes, so UlyssesModel (its names omit the original_attention level) would leave every CP adapter frozen; the pin peft<0.19 keeps that path unreachable, and a bump needs a release that fixes peft#4181.

Tests and benchmarks

Bash

# LoRA SFT under one shape (ep / ep_cp / etp / cp / fsdp, --qlora for a 4-bit base), the save
# reloaded on a fresh base; LoRA under TP is refused at construction (tests/cpu/trainers/test_expert_lora_tp_gate.py),
# adapters that left the wrap frozen likewise (tests/cpu/peft/test_adapters_trainable_check.py)
torchrun --nproc_per_node=2 tests/gpu/trainers/lora/test_lora_sft_modes.py --model gpt_oss_20b --mode ep --save-reload
torchrun --nproc_per_node=2 tests/gpu/trainers/lora/test_lora_sft_modes.py --model qwen3_0_6b --mode cp --save-reload --resume
# merge_expert_lora_on_save checkpoint: servable, and resumed exactly against an uninterrupted run
torchrun --nproc_per_node=2 tests/gpu/trainers/lora/test_lora_merged_save_resume.py \
    --family gpt_oss --adapters mixed
# weight syncs leave the frozen base bit-identical (no server), on every family the sync serves
# under each mode (fsdp dense; ep1 / ep2 / etp2 MoE)
torchrun --nproc_per_node=2 tests/gpu/trainers/lora/test_lora_weight_sync_exact.py \
    --family qwen3_moe --mode ep2 --adapters mixed

# Adapters on an on-policy run, asserted on the SERVED policy (needs a live vLLM server):
# attention LoRA under EP and pure ETP, native grouped expert LoRA under EP
torchrun --nproc_per_node=2 tests/gpu/trainers/grpo/test_online_grpo_vllm_e2e.py \
    --trainer online --mode expert_lora_ep2
# attention LoRA under FSDP2 DP, and the TP refusal at trainer construction
torchrun --nproc_per_node=2 tests/gpu/trainers/grpo/test_online_grpo_vllm_e2e.py \
    --trainer online --mode lora_tp2_rejected

# Dense LoRA/QLoRA benchmark
torchrun --nproc_per_node=1 tests/gpu/profiling/benchmark_sft_dense.py \
    --model qwen3-8b --seq 16384 --lora_r 64 --lora_all_linear --qlora
# Expert-targeting LoRA under EP
torchrun --nproc_per_node=8 tests/gpu/profiling/benchmark_sft_ep.py \
    --ep 2 --seq 4096 --lora_r 64 --lora_experts

Omit --lora_all_linear for attention-only; on the EP benchmark it means attention + experts, while --lora_experts is the native grouped adapters alone.

Source on GitHubUpdated 18 November 2026