# PEFT (LoRA) Fine-Tuning

LoRA trains small rank-decomposed adapter matrices while the base model stays frozen, eliminating optimizer
states for frozen parameters.

It runs under DDP/FSDP, EP, CP and pure ETP; **TP**, **EP+TP** and **PP** reject it, and QLoRA runs only
under DDP/FSDP and CP-on-dense (full matrix and reasons: [Parallelism compatibility](#parallelism-compatibility)).
Under EP, LoRA targets both attention (via PEFT) and the MoE experts (via native grouped adapters).

Offline GRPO under CP is full fine-tuning only and rejects every adapter shape; SFT and SMPO keep
their CP adapter support. Most other configurations take it. The embedding trainer
accepts LoRA and DoRA, not QLoRA ([Embedding — PEFT / LoRA](https://whitecircle.com/halo/docs/agent-docs/training-methods/embedding.md#peft--lora)).

## Supported methods

Uses the [PEFT library](https://github.com/huggingface/peft) (pinned 0.18.1). LoRA is the only method with tested
parallelism support; other PEFT methods (AdaLoRA, Prompt Tuning, (IA)3) reach the trainer through a custom
`peft_config` but are untested with it. An AdaLoRA or (IA)3 layer is refused at construction wherever an
out-of-place fold runs (the weight-sync trainers, embedding LoRA, `merge_expert_lora_on_save`): the fold
covers plain LoRA and DoRA only ([Online RL](#online-rl--rollout-server-weight-sync)).

| Method | Enable | Notes |
|---|---|---|
| LoRA | `use_peft: true` | Default low-rank adaptation |
| QLoRA | LoRA + BitsAndBytes 4-bit/8-bit base | LoRA on a quantized base |
| DoRA | `use_dora: true` | Weight-decomposed (magnitude + direction). Attention only — **rejected** alongside expert projections, which have no grouped DoRA |
| rsLoRA | `use_rslora: true` | `alpha/sqrt(r)` scaling, for high ranks. Applies to attention *and* native expert adapters |

## Configuration

Fields come from TRL's `ModelConfig` and are converted to a `LoraConfig` via `get_peft_config()`:

```yaml
use_peft: true
lora_r: 64                    # Rank (TRL default 16)
lora_alpha: 128               # Scaling factor (alpha/r applied to adapter output)
lora_dropout: 0.05
lora_task_type: CAUSAL_LM     # CAUSAL_LM | SEQ_CLS | FEATURE_EXTRACTION
lora_target_modules: [q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj]
```

`lora_target_modules` is tri-state: unset (`None`) falls to PEFT's own per-architecture table, which
has no entry for any MoE or multimodal family here (PEFT raises `Please specify target_modules`) and
adapts only `q_proj`/`v_proj` on the dense ones it lists (Qwen3, Llama, Mistral), unscanned by the
exclusion below — so name the modules; a populated list selects those modules. Expert-LoRA-only means listing **only** expert projections — the peel then empties
the list, which is what disables attention PEFT. Writing `lora_target_modules: []` yourself names nothing
at all and is rejected: with no adapter to create, the run would full-finetune at the LoRA learning rate.

The spelling decides how a target matches. A list names modules at any length: `[o_proj]` adapts every
module whose path is `o_proj` or ends in `.o_proj` (whole trailing segments, so `[proj]` does not reach
`o_proj`). A bare YAML string is PEFT's own spelling, `all-linear` or a regex fullmatched against each full
module path (`'.*\.0\.self_attn\.o_proj'`), so a bare `o_proj` matches nothing. The command line always gives
a list (`--lora_target_modules=o_proj` names the module), so a regex goes in the YAML; a list entry spelling a
regex (`\ * + ? ( ) [ ] { } | ^ $`) is refused at parse time, `use_peft: false` included, since PEFT compares
list entries to paths verbatim and it would match nothing, unnoticed beside an entry that does match.
`ModelArguments` (`halo/configs/script_arguments.py`) keeps a one-entry list a list, where TRL's `ModelConfig`
collapses it to its bare entry; `build_peft_config` turns `[all-linear]` into PEFT's bare-string sentinel, and
a lone expert projection (`[experts]`, or a bare `experts`) still goes to the expert peel.

### Targets PEFT cannot adapt

A stock LoRA adapter decomposes the matched module's own weight and adds the delta to that module's
output, which leaves two module shapes out of its reach:

| Shape | Example | Without the exclusion |
|---|---|---|
| A wrapper holding its weight in a child module | Gemma 4's `Gemma4ClippableLinear`, the vision and audio towers' projections | injection raises `Target module ... is not supported` |
| A layer subclass computing something other than its base layer's affine map | DeepSeek-V4's grouped `o_a_proj`, whose block-diagonal output is one group wide | the delta has the wrong width and the first forward raises |

Both are reachable by an ordinary target list — target names are matched by suffix over the whole
module tree, so a multimodal tower is matched by the language model's `q_proj`…`o_proj` spelling — and
by `all-linear`, which reaches every `nn.Linear` subclass a family defines.

`build_peft_config` (`halo/model_loading/parallel/peft_setup.py`) scans the live model with PEFT's own
matcher and puts those modules in the `LoraConfig`'s `exclude_modules` by full path (PEFT also honors
an entry as a dotted suffix), so an identically named leaf elsewhere keeps its adapter. A warning names
the count and one example path per reason. If it leaves no adaptable target at all, the run is rejected
rather than trained with nothing — or, when native expert adapters are live, with the attention half
missing. The verdict is agreed across ranks: a rank whose probe faulted would otherwise adapt a
different parameter set and surface only as a hang.

The two shapes are told apart structurally: a wrapper owns no parameter of its own, and a subclass
overriding `nn.Linear.forward` is measured once per class and geometry against `F.linear`'s output
shape — the low-precision compute drop-in and a quantized base change only *how* the map is computed
and keep their adapters. A module of a type PEFT has no adapter layer for at all (Inkling's
`rel_logits_proj`) is matched by no recipe's names and still raises PEFT's own
`Target module ... is not supported` when named. The exclusion is saved into `adapter_config.json`,
so reloading the adapter onto the same base loaded through the same class reproduces the same
injection; resume rebuilds it from the live model instead and never reads that file. The merge tools
pick that class off the adapter's keys ([Merging adapters](#merging-adapters)). An exclusion found on
the context-parallel wrapped tree is written in the plain module spelling, as the adapter's tensor keys
already are, so a plain reload and the merge tool honor it; the live config keeps the wrapper's spelling
for the wrapped model.

A tower whose projections are plain `nn.Linear` (CLIP/SigLIP/Pixtral-style) is untouched by any of
this and keeps its adapters; on text-only data those adapters receive no gradient and stay at their
zero initialization, costing optimizer state and nothing else. On an image row they train under
either checkpointing mode, at the cost of recomputing the tower in the backward
([Tower blocks](https://whitecircle.com/halo/docs/agent-docs/parallelism/data-parallelism.md#tower-blocks-under-reentrant-checkpointing)).

One consequence for a mistyped target: a name that matches a container rather than a leaf (`mlp` on
Qwen3) drops out of the injection with that warning, so the run trains the targets that did match. Read the excluded count — it is the tell.

### Hyperparameters

- **Rank (`lora_r`):** 8–32 for 1B–3B, 32–96 for 7B–14B, 64–384 for 27B+. With a tuned learning rate,
  performance is largely rank-independent; rank matters most when the LR is suboptimal.
- **Alpha:** `alpha = rank` (conservative) or `2 * rank` (aggressive). For rank ≥ 64, prefer `use_rslora`.
- **Target modules:** all-linear on a dense model or on stock experts (`use_grouped_gemm: false`). On EP layers it
  runs on GPT-OSS, Qwen3 MoE, Gemma 4, Inkling and LFM-2, without reaching the fused expert tensors. Every other EP
  family refuses it, as it would put PEFT layers inside the EP layer. There, name attention and expert projections
  explicitly (see [MoE models](#moe-models--expert-targets-and-full-trained-modules)).
- **Learning rate:** higher than the same run's full fine-tune. The shipped SFT recipes start at `1e-4`, and the
  code-contests recipes run `3e-6` against `2e-7`.
- **Dropout:** honored on SFT. The preference, reward and offline-GRPO trainers run TRL's
  `disable_dropout_in_model` after the adapter wrap (`disable_dropout` defaults to `True`), zeroing PEFT's
  `lora_dropout` along with the model's own for reference-vs-policy determinism. A warning fires when a
  configured dropout was zeroed; `disable_dropout: false` keeps it. Leaving `lora_dropout` out keeps TRL's
  0.05 default, so `lora_dropout: 0.0` is the setting that states no dropout.

| Trainer | Full-FT band | LoRA start |
|---|---|---|
| SFT | `2e-6` – `2e-5` | `1e-4` |
| SMPO / DPO | `5e-7` – `5e-6` | `5e-5` |
| GRPO | `1e-6` – `5e-6` | `3e-6` (every shipped LoRA async GRPO recipe under `examples/grpo/environmental/`) |

The effective-batch formula is unchanged by LoRA — see
[SFT](https://whitecircle.com/halo/docs/agent-docs/training-methods/sft.md#learning-rate-and-global-batch-size).

Ready configs: `examples/sft/qwen3/qwen3-4b-ultrachat-lora.yaml` (and its `-qlora` counterpart, which adds
`load_in_4bit: true` and `use_bnb_nested_quant: true` over TRL's default `bnb_4bit_quant_type: nf4`).

## MoE models — expert targets and full-trained modules

MoE experts are fused `nn.Parameter` tensors (`GptOssExperts.gate_up_proj` / `down_proj`, and the
equivalents on other families), not `nn.Linear`, so PEFT's `all-linear` cannot reach them; on gpt-oss it
resolves to the attention `q/k/v/o_proj` only. Name the expert projections explicitly instead.
`split_expert_lora_targets` (`halo/model_loading/parallel/peft_setup.py`) peels those entries out of the list and
routes them to native grouped-LoRA built inside each EP layer (`_init_expert_lora` in
`halo/distributed/expert_parallel/moe_layer/expert_lora.py`); the remaining attention names stay on stock PEFT.

| `lora_target_modules` entry | Requests |
|---|---|
| `gate_proj` / `up_proj` / `down_proj` | that logical projection |
| `gate_proj_gmm` / `up_proj_gmm` | the same, in the grouped-GEMM storage spelling |
| `gate_up_proj` | the fused gate+up projection |
| `experts` / `mlp.experts` | all three expert projections |

Requests are **logical**; what gets built follows the family's storage. One adapter is created per stored
3-D expert tensor that any requested projection touches.

On a family that stores the fused `gate_up_proj` — every family except Qwen3, Bailing, Nemotron-H and
grouped-GEMM GptOss (below) — asking for `gate_proj` alone adapts the whole `[E, H, 2M]` tensor: gate and up
share one rank-`r` subspace. On the separate-storage layouts it adapts gate alone, with its own rank `r`.
Nemotron-H's experts store `up_proj` and `down_proj` only, so a gate-only request builds no adapter and raises
(`reject_unrealized_expert_lora`).

`lora_r` must be a multiple of 8. The grouped GEMM reads the adapter's rank dimension as a stride that
spans a multiple of 16 bytes (`GROUPED_MM_STRIDE_ALIGNMENT_BYTES`), and the adapter GEMMs run at the
activation dtype, bf16 at the narrowest, whatever the experts are stored in; the EP layer refuses any
other rank as it builds the adapters, where the first expert matmul would otherwise fail with
`strides should be multiple of 16 bytes`.

GptOss picks its layout at runtime: with grouped-GEMM (SM90+, the default) it stores
`gate_proj_gmm`/`up_proj_gmm` separately, without it a fused interleaved tensor. The same YAML therefore
yields a different adapter, and resume across that switch is refused.

The peel is **name-based, not module-based**. `gate_proj`/`up_proj`/`down_proj` also spell the plain
`nn.Linear` MLPs of a hybrid MoE — dense prefix layers (`first_k_dense_replace`) and shared experts — and
those are adapted by neither half once the name is peeled. A warning names what moved.

Shared-expert LoRA has no supported route under EP: named generically it is peeled; named explicitly
(`shared_experts.down_proj`) PEFT wraps a Linear inside the EP layer and `reject_peft_lora_in_ep_layers`
rejects it.

Do not reach for TRL's `lora_target_parameters` on an MoE model — it is **rejected** on any model carrying
EP layers (`ep_size > 1`, or the default `use_grouped_gemm`), whatever it targets, so an attention-only run
is rejected too. It is PEFT's own mechanism for fused `nn.Parameter` experts, and it wraps the module
*owning* the parameter, which sits above the EP layer and so escapes both EP validators.

Weights PEFT can't LoRA are **full-trained** (a trainable copy, not a low-rank adapter) via
`lora_modules_to_save`: `embed_tokens` (an `nn.Embedding`), `lm_head` (excluded by `all-linear`), and the
MoE `router` / `gate`. An **expert-only** run builds no `PeftModel`, so `lora_modules_to_save` raises there
rather than leaving those modules silently frozen — add an attention target, or drop the field.
`unfreeze_layers_patterns` / `freeze_layers_patterns` (`halo/model_loading/parallel/freeze_patterns.py`) are
refused on any adapter run (attention LoRA or native expert LoRA): it freezes every base parameter and trains
only the adapters, so a pattern would select nothing. Use `lora_modules_to_save` for a fully trained module.

### Router training under EP

The router/gate is absorbed into the EP module for every family but Gemma4, whose router is a sibling module
the wrapper never owns. EP modules are FSDP-ignored, so their gradients are DP-averaged by a hook the EP
layer registers at construction on the *original* router param, one that never sees the trainable copy
`modules_to_save` creates.

`EPMoELayerBase.reattach_dp_average_grad_hooks` re-attaches it to the live copy automatically, so `router` in
`lora_modules_to_save` stays DP-consistent in every EP mode; the ep1 FSDP-sharded path
(`fsdp_shard_ep1_experts`) and the multi-node deferred sweep already cover it.

`modules_to_save` on any **other** EP-internal submodule is unsupported:
`reject_unsynced_trainable_params` raises naming any trainable param left inside an EP module with
no gradient sync — it would otherwise silently drift across DP ranks. Its sibling
`reject_peft_lora_in_ep_layers` (`halo/trainers/parallelism/adapter_rules.py`) rejects stock-PEFT LoRA layers placed
inside EP modules.

### Example: gpt-oss LoRA under EP

```yaml
lora_target_modules:            # attention → PEFT; expert projections → native grouped-LoRA
- q_proj
- k_proj
- v_proj
- o_proj
- gate_up_proj                  # fused gate+up expert projection
- down_proj
lora_modules_to_save:           # full-trained (PEFT can't LoRA these)
- embed_tokens
- lm_head
- router                        #   DP-sync hook re-attached automatically under EP
```

`examples/grpo/environmental/gptoss/vllm/gptoss-20b-code-contests-lora-ep4.yaml` is a full
Environmental-GRPO version, with the router left frozen.

**Adapter dtype under FSDP:** `get_peft_model` upcasts adapters (and `modules_to_save` copies) to fp32, but
FSDP2 requires a uniform dtype per shard group. `_cast_peft_params_to_compute_dtype` casts trainable fp32
PEFT params to the compute dtype before wrapping (to fp32 instead under `fp32_non_ep_params`). EP params are
excluded by identity, before the dtype test: downcasting a deliberately-fp32 router or expert would negate
`fp32_router` / `fp32_experts`. They are FSDP-ignored unless FSDP2 manages the ep1 experts
(`fsdp_shard_ep1_experts` at `ep_group_size == 1`); there an fp32 router copy takes a shard group of its
own ([Data Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/data-parallelism.md#fsdp2-strategy-by-mode)).

## Parallelism compatibility

| Mode | LoRA | QLoRA | Target restriction | Notes |
|---|---|---|---|---|
| DDP / FSDP | Yes | Yes | None | Standard path; QLoRA on a MoE needs `use_grouped_gemm: false` ([QLoRA](#quantized-training-qlora)) |
| HSDP | Yes | No | None | Pure DP or CP only, so expert adapters train at `ep_size: 1`; every replica starts from the first one's adapters ([HSDP](https://whitecircle.com/halo/docs/agent-docs/parallelism/data-parallelism.md#hsdp-hybrid-sharded-data-parallel)). QLoRA rejected: `use_hsdp` is refused under QLoRA, which skips FSDP2 ([QLoRA](#quantized-training-qlora)) |
| EP | Yes | No | Attention + experts | Attention via PEFT; expert FFNs via native grouped adapters. QLoRA rejected: EP loaders materialize plain de-quantized weights, losing `Params4bit` |
| CP | Yes | Yes* | None | CP shards the sequence, not the weights — adapters stay replicated; standard `from_pretrained` preserves `Params4bit`. *QLoRA only on a **dense** model: an MoE takes the grouped-GEMM loader, which rejects a quantized base |
| TP | **No** | No | — | Rejected — adapters are plain tensors outside the TP graph |
| EP+CP | Yes | No | Attention + experts | Both active |
| EP+TP | **No** | No | — | Both adapter kinds rejected: attention LoRA as under TP, native expert LoRA by the gate's `model_has_expert_lora` arm |
| ETP | Yes | No | Attention only | Expert adapters rejected at config time by `ParallelismConfig` (`expert_tp_size > 1` gives the replicated adapter half a partial, never-synced gradient) |
| EP+ETP | Yes | No | Attention only | Expert adapters rejected at config time, as under ETP |
| PP | **No** | No | — | Rejected, as a `peft_config` or an already PEFT-wrapped model — the adapter save and resume are not stage-aware ([PP rejected combinations](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md#rejected-combinations)) |

**EP.** The expert names [above](#moe-models--expert-targets-and-full-trained-modules) route to **native grouped
LoRA** — grouped `[E_local, K, r]`/`[E_local, r, N]` adapters stored alongside each expert weight, applied in
the grouped-GEMM compute and gradient-synced across the EP group like the experts themselves. `lora_B` is
zero-initialized, so the initial delta is zero. The frozen base experts stay bf16.

The default save writes a standalone adapter; `merge_expert_lora_on_save: true` folds the delta into the
base for a servable HF checkpoint instead, keeping the unmerged adapter beside it for resume
([Merging adapters](#merging-adapters)).

**TP / EP+TP.** For a colwise-sharded base, `lora_B` becomes a per-rank output shard while `lora_A` stays
replicated. Nothing broadcasts the replicated matrix (it diverges from init) and nothing distinguishes the
sharded one from a replica (the TP replicated-grad sync averages and corrupts it), so the trained adapter is
rank-inconsistent and will not reload onto a non-TP model. Use FSDP/DP, CP, or pure ETP instead.

The gate is `reject_lora_under_tp` (`halo/trainers/parallelism/adapter_rules.py`) and it refuses both
adapter kinds: a `PeftModel` or any PEFT tuner layer injected in place (read off the tuner layers, so a
backbone's own `lora_*` weights stay base weights), and — checked first, via `model_has_expert_lora` — the native
grouped expert adapters, which are no tuner layer and which the TP replicated-grad sweep skips by param
identity. `expert_tp_size > 1` rejects expert LoRA earlier still, at config time in `ParallelismConfig`,
before the checkpoint downloads.

## Measured cost

Both tables measured on the Blackwell image, Halo 1.1.0 or earlier.

Dense — 1× B300 (SM103), `DistributedSFTTrainer`, AdamWBF16, Liger, FA4, Qwen3-8B, seq 16384, BS=1, GC,
10 steps / 3 warmup (full fine-tuning at this shape: [Liger → Benchmarks](https://whitecircle.com/halo/docs/agent-docs/optimization/liger-kernels.md#benchmarks)):

| Config | Trainable | tokens/s/GPU | Peak memory |
|---|---|---|---|
| LoRA r=64, attn only | 61M (0.7%) | 18,897 | 34.3 GiB |
| LoRA r=64, all linear | 175M (2.1%) | 16,203 | 35.0 GiB |
| QLoRA r=64, all linear (NF4 base) | 175M (2.1%) | 15,410 | 25.3 GiB |

Attention-only runs 18% faster than full fine-tuning (15,966) at about half its memory; all-linear is ~14%
slower than attention-only, since adapter matmuls run on every MLP layer. Throughput is **rank-invariant**
within a variant (attn-only ~18.8k, all-linear ~16.0k across r=16/64/128): the frozen base forward/backward
dominates the step, and only memory grows with rank.

QLoRA saves ~10 GiB more than bf16 all-linear LoRA at ~5% lower throughput. It is the path onto a smaller
GPU, since plain LoRA needs ~34 GiB at this shape even at the minimum rank.

MoE — same setup on 8× B300, `gpt-oss-20b` (32 experts, top_k=4) at EP=2, seq 4096, r=64 (full
fine-tuning at this shape: [Throughput Benchmarks → EP-only](https://whitecircle.com/halo/docs/agent-docs/optimization/throughput-benchmarks.md#ep-only-batch-scaling)):

| Config | Trainable | tokens/s/GPU | Peak memory |
|---|---|---|---|
| LoRA r=64, attn only (PEFT) | 32M (0.28%) | 16,166 | 29.6 GiB |
| LoRA r=64, experts only (grouped) | 425M (3.60%) | 17,002 | 31.9 GiB |
| LoRA r=64, attn + experts | 457M (3.86%) | 14,316 | 32.0 GiB |

**LoRA under EP is far leaner than full fine-tuning.** The frozen base carries no gradients or optimizer
state, so every variant peaks at 30–32 GiB against full fine-tuning's 78.9 GiB at this shape (batch 1), and
every variant runs faster than its 12,432 tok/s/GPU. Experts-only is 5% faster than attention-only, since
the grouped expert adapters fold into the grouped-GEMM compute; attn + experts is slower than either alone.

On `qwen3-30b-a3b` (128 experts) at EP=2, experts-only r=64 is 9.39% trainable at 9,894 tok/s/GPU and
46.7 GiB. At batch 1 tok/s/GPU varies ±10% run-to-run.

## Reference model handling

Non-CP offline GRPO and Async GRPO with Environments disable the adapter (`disable_adapter()`) to compute reference log-probs
from the frozen base instead of a second model copy. Under EP the mixin patches that context
(`install_ep_aware_disable_adapter`) so it reverts the native EP expert adapters too, giving a true frozen-base
reference for both adapter halves; the patch also covers TRL's `use_adapter(None)` for online GRPO / DPO / KTO.

The pinned PEFT 0.18.1 clears `requires_grad` on the adapters a module holds when `disable_adapter()`
enters and restores it on the ones it holds at exit. A reference pass that is the first forward after an
FSDP2 reshard (online GRPO at `beta > 0` with no old-logps recompute) enters on the sharded params and exits
on the unsharded copies, which would freeze every sharded adapter silently. `install_fsdp2_safe_disable_adapter`
wraps the context for every PEFT model: when the registered trainable params changed across it, it reshards
before PEFT's exit, so the restore lands on the sharded params (upstream PEFT drops the freeze in
huggingface/peft#3839, which 0.18.1 predates).

An explicit reference model (DPO, KTO, self-distillation's KL `reference_model`) is never parallelized:
under EP or TP it is a whole dense replica on every rank, warned about rather than refused. LoRA with the
adapter disabled, or precomputed reference log-probs, avoids it. An expert-only LoRA run builds no
`PeftModel`, so nothing switches its adapters off: DPO and KTO need `precompute_ref_log_probs` there, and
offline GRPO loads a frozen base `ref_model`. Per trainer: [DPO → Reference model](https://whitecircle.com/halo/docs/agent-docs/training-methods/preference/dpo.md#reference-model),
[Offline GRPO → Reference model](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/offline.md#reference-model). SMPO is reference-free.

On online / async GRPO, where no adapter wraps the model — a full fine-tune, or an expert-only LoRA
run, which builds no `PeftModel` — `beta != 0` holds a frozen reference: the script loads the base the
way the policy loads and the trainer hands it to TRL in place of the fp32 copy TRL would build
([Online GRPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/online.md#grpo-objective-for-verifiable-rewards)). It is an
unparallelized replica per rank, refused before it loads when it exceeds half the free GPU memory or 80% of the node's
available host RAM across its ranks. Add an attention LoRA target (it wraps the
model, and the disabled adapter is the reference) or set `beta: 0` to hold none.

## Online RL — rollout-server weight sync

Online and Async GRPO with Environments generate rollouts from a separate rollout server — vLLM or SGLang
([Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#weight-sync)) — which loads **no adapter** unless
`rollout_adapter_sync` serves the policy as one on SGLang ([Adapter sync](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#adapter-sync)).

Otherwise each NCCL weight sync forwards every base weight with its adapter folded in, under base-model param names
(PEFT prefixes stripped, `lora_*` params skipped). Without the fold, the server would generate from the
un-adapted base.

The fold is out of place, one tensor at a time as the sync sends it (`lora_folded` in
`halo/models/peft_adapters.py`, `lora_folded_data` in `halo/distributed/lora_fold.py`): the value PEFT's `merge_adapter` would write on the same placement, bit
for bit, on Linear, Embedding, Conv and `target_parameters` layers, with `lora_bias` and DoRA. A Conv
adapter on an FSDP2 shard folds on whole tensors (a DTensor convolution would run on each rank's slice of the
adapter rank) and reshards the result, which holds the whole base and adapter tensors on every rank, so it suits
the small convolutions of a patch embedding. A fold
on FSDP2 shards can differ from a single-device merge by rounding, since the delta's contraction is
summed across shards. The frozen base is never written: PEFT's in-place merge followed by its unmerge
would not give it back, since in bf16 `(w + d) - d` misses `w` by a rounding step wherever the two
roundings do not cancel. The fold holds one tensor's temporaries at a time, so it adds at most one
local shard of the largest adapted tensor (a full copy on one device) to the sync's peak. A PEFT layer
the fold does not cover (LoRA on `nn.MultiheadAttention`, trainable tokens, quantized LoRA, a variant
other than DoRA, a grouped conv) is refused when the trainer is built, and so is a PEFT adapter on a
tensor the sync's dense push does not send (an EP layer's expert weights, which take native expert LoRA),
since only that push folds PEFT adapters. Each sync checks again: the layers before it streams, the
targets as its dense push ends.

Both trainers share this path (`gather_and_send_weights` in `halo/trainers/grpo/rollout/weight_push.py`),
which also folds in the EP / FSDP2 / TP gathers; the fold is a collective on all ranks under FSDP2.

## Embedding models

The embedding trainer injects LoRA into its backbone in place, on data parallelism only, and on a MoE whose family has an EP layer only at `use_grouped_gemm: false` (its stock experts): [Embedding — PEFT / LoRA](https://whitecircle.com/halo/docs/agent-docs/training-methods/embedding.md#peft--lora).

## Quantized training (QLoRA)

QLoRA combines LoRA with 4-bit/8-bit base quantization under DDP/FSDP (a MoE base whose family has an EP layer
class also needs `use_grouped_gemm: false`), and under CP on a dense model. CP keeps the standard `from_pretrained` loader,
which preserves `Params4bit`. GPU rows train dense Qwen3-0.6B, and gpt-oss-20b with stock experts through a save and
reload. AdamWBF16 optimizes only the bf16 adapter params; the frozen quantized base is skipped.

`load_distributed_model` raises for EP, TP, and the grouped-GEMM MoE loader — pure ETP included, since
the gate reads `ep_group_size = ep_size × expert_tp_size`. Those loaders materialize plain de-quantized
weights, so `Params4bit` are lost and PEFT's 4-bit adapter dispatch fails.

The 4-bit **compute** dtype follows the run's own precision (`bf16`/`fp16` on the training config), not
TRL's `ModelConfig.dtype` — whose `"float32"` default nothing else here reads, and which would otherwise
dequantize and compute every 4-bit matmul in fp32.

QLoRA is SFT/offline territory, but offline GRPO rejects it under CP. The online and async GRPO trainers reject a quantized base at
construction (`reject_unsyncable_model`). The NCCL weight sync forwards raw parameter storage under
base-weight names, so a bnb-packed 4-bit base would ship non-floating-point tensors that corrupt the served
policy. Plain LoRA is the supported
RL adapter path — see [Online GRPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/online.md#lora).

Under `torchrun` data parallelism the 4-bit base cannot be FSDP2-sharded (`fully_shard` rejects the non-float
`Params4bit`), so the trainer skips FSDP2 and all-reduces the (small) adapter gradients across ranks, with
the frozen base replicated per rank.

Nothing wraps the model, so the FSDP2 sharding knobs cannot take effect: a non-default `use_hsdp`,
`fsdp_reshard_after_forward`, `fsdp_reshard_after_backward` or `fsdp_defer_grad_sync` is rejected at trainer
construction rather than silently ignored, naming the offending flags. QLoRA under CP takes the same gate. Under `accelerate launch` with
the DDP config (`multigpu_dp_config.yaml`), the other QLoRA launcher, accelerate owns the wrap and those knobs only warn; its FSDP2
configs cannot flatten the 4-bit tensors.

Pass a `BitsAndBytesConfig` through `load_distributed_model(..., quantization_config=bnb_config)`, or pass a
pre-loaded quantized model plus `peft_config` to the trainer. For 8-bit optimizer states without a 4-bit
base, set `optim: adamw_8bit`.

### FusedLinearCrossEntropy for long sequences

[Liger FLCE](https://whitecircle.com/halo/docs/agent-docs/optimization/liger-kernels.md) fuses `lm_head` with the loss instead of materializing the
`batch × seq × vocab` logits. Qwen3-8B QLoRA r=32 all-linear at 32k (BS=1, GC, Liger, FA4, on one B300, Halo 1.1.0 or earlier) peaks at
**43.5 GiB with plain CE and 20.9 GiB with FLCE**, so by peak it fits a 24 GB GPU only with FLCE. FLCE is
SFT-only and disables entropy logging.

## Checkpoint saving

`save_model()` saves only adapter weights (~MB). Which saver writes them, from which rank and with which
collective under each mode: [Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#peft-lora-saving). Two layouts are
specific to LoRA:

- An **expert-only** run has no PEFT model at all, so `save_expert_lora_adapters` writes the adapter with a
  synthesized config: `peft_type: EXPERT_LORA`, the base model path, and every field the delta depends on
  (`r`, `lora_alpha`, `lora_dropout`, `use_rslora`, `expert_projections`).
- A run that **grew its vocabulary** (`added_special_tokens` past the checkpoint's padded rows) saves the grown
  embedding and an untied head with the adapters, offline included, and resumes them; accelerate-managed FSDP and
  LoRA on one layer of an untied embedding pair are refused for such a run
  ([Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#peft-lora-saving)). An expert-only run has no adapter to carry them:
  its resume grows the frozen rows again, and `merge_expert_lora_on_save` exports them whole. The merge grows the
  base to match ([Merging adapters](#merging-adapters)).
- Under **CP** the adapter keys drop the wrapper's path segments (`.original_attention.`, extra `.model.`) so
  they load onto a non-CP model. The module paths in the written `adapter_config.json` (`exclude_modules`,
  and `target_modules` once `all-linear` made PEFT rewrite it to full paths) are respelled the same way,
  while the live config keeps the wrapper's spelling.

**Reloading elsewhere.** Attention adapters carry standard PEFT keys and load onto a stock
`PeftModel.from_pretrained`. Native expert adapters do not: grouped `[E, K, r]` tensors have no PEFT
representation, and PEFT drops unrecognized adapter keys without a warning (`load_adapter` filters
`missing_keys` and never reads `unexpected_keys`).

Every directory carrying them is therefore labeled with a `peft_type` PEFT does not know, so an external load
raises on the label rather than returning a half-adapted model: `EXPERT_LORA` for an expert-only save,
`LORA_WITH_EP_EXPERT_LORA` for a mixed attention+expert one.

The mixed label keeps every `LoraConfig` field beside it, plus an `ep_expert_lora` block recording the expert
half's `r`/`lora_alpha`/`lora_dropout`/`use_rslora`/`expert_projections`; resume reads the tensors directly
and the file only for its scaling check (below).

The in-memory merge and convert paths check those markers (`reject_expert_lora_adapter`) and fall through
to a tensor-key scan when the config is absent or carries a stock `peft_type`, so an unmarked directory
holding `.experts.<attr>.lora_{A,B}` keys is refused too; `merge_peft_adapters.py` routes such an adapter to the
[streaming merge](#streaming-merge).

**Resume:** EP/CP rebuild the base with zero-initialized adapters at init, so trained adapters are restored
from the checkpoint's `adapter_model.safetensors` (a merged checkpoint's `resume_adapter/`; not from the base reload) by
`restore_adapters` (`halo/checkpoint/parallel/adapters.py`), which `load_resume_weights` calls; a single-process
or DDP run's adapter-only checkpoint takes it too, keeping the adapters at their live dtype where PEFT's
`load_adapter` would recast bf16 adapters to fp32. Before any copy, every rank matches the file to the tensors a
save of the live model writes, key for key at the saved shape (PEFT's own selection for the attention adapters,
`modules_to_save` and their buffers; the gathered layout for the expert adapters), and the verdict is joined: a
missing, unexpected or reshaped tensor raises on every rank with nothing written. That covers resuming expert
adapters into a run that does not build them (EP off, `use_grouped_gemm: false`, or the expert projections
dropped from `lora_target_modules`), a run that builds them from a file without them, and a changed `lora_r`.
A changed LoRA scaling raises too: resume compares the scaling inputs the saved `adapter_config.json` records
(`lora_alpha`, `use_rslora`, `use_dora`, `alpha_pattern`, and the expert half's `lora_alpha` and
`use_rslora`) with the live run's, since names and shapes would still match and the delta would silently
rescale. A missing `adapter_config.json` raises too.

Under CP `remap_cp_adapter_keys_to_live` maps the saved, normalized keys back onto the live wrapper
spelling, read off the same PEFT selection the save made, so every adapter-name spelling PEFT writes
(`lora_A.default.weight`, `modules_to_save.default.weight`, `lora_embedding_A.default`, a DoRA magnitude) maps
back; a key it does not produce fails the check above. See
[Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#resume-by-parallelism-mode).

## Merging adapters

```bash
python scripts/after_training/merge_peft_adapters.py \
    --adapter_dir path/to/adapter/checkpoint \
    --output_dir path/to/merged/model
```

Add `--task classification --num_labels N` for classification models. Handles causal LM, sequence
classification, multimodal/VLM, and trust_remote_code architectures: `--trust_remote_code` defaults on for
a local base and off for a Hub base ([Scripts](https://whitecircle.com/halo/docs/agent-docs/reference/scripts-reference.md#post-training-scripts)). Use
`--device_map auto` for large models.

The base loads through the class the adapter's tensor keys address: an adapter from a `text_only_model: true`
run (keys under `model.layers`) loads the text-only `*ForCausalLM`, not the multimodal wrapper, and keys
naming no module of either class are refused — applied to the wrong class, PEFT matches nothing, warns, and
the tool would write the bare base as a finished model.
`--fp32_norms` writes every normalization leaf in fp32, the layout `convert_to_bf16.py` writes.

A run whose `added_special_tokens` outgrew the checkpoint's embedding trained on rows no base holds. The merge (and
`convert_to_bf16.py --check_inference`'s adapter load) grows the base to the adapter directory's tokenizer under the
run's own rule before PEFT loads the adapter, which carries those rows ([Checkpoint saving](#checkpoint-saving)): only
when that tokenizer has more ids than the base has rows, and the base's own tokenizer fits them (a checkpoint whose
tokenizer already outruns its rows loads unchanged). The added rows start as NaN, so none is served from
uninitialized memory: an adapter saved before adapter saves carried them leaves some unset (the input embedding's,
or an untied head's), and those are rebuilt with a warning as the mean of the checkpoint's rows. The run drew them by
transformers' mean resizing and an adapter run keeps them frozen, so the rebuilt rows match the run's to within that
draw (`apply_adapter`, `halo/checkpoint/tools/adapter_merge.py`).

The merge re-applies the run's training sidecars to the merged model and copies both files into the output:
the GptOss sinks policy recorded in `training_provenance.json`, and `router_balancing_biases.pt` into the
family's native routing-bias slots, so the merged model routes on the trained bias.

The in-memory merge **refuses native EP expert-LoRA adapters** (`peft_type` of `EXPERT_LORA` or
`LORA_WITH_EP_EXPERT_LORA`, or tensor keys ending at `.experts.<attr>.lora_{A,B}`), because
`PeftModel.merge_and_unload` silently drops the grouped expert deltas; `convert_to_bf16.py --peft` shares that
refusal. A stock PEFT adapter on a per-expert `nn.Linear` MoE keys as `.experts.<i>.<proj>.lora_A.default.weight`
and merges normally; only the grouped namespace is refused.

`merge_peft_adapters.py` folds such an adapter after the fact through its [streaming merge](#streaming-merge),
automatically for expert-only and mixed adapters. `merge_expert_lora_on_save: true` produces the merged servable
checkpoint at training time instead, for expert-only **and** mixed runs alike. Each merged training checkpoint also keeps the unmerged adapter in `resume_adapter/`, which resume restores onto the
base, so the run continues from its trained adapters rather than the fold
([Merge-on-save checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#merge-on-save-checkpoints)).

It needs native grouped expert adapters to exist: `lora_target_modules` must name at least one expert
projection, or `reject_unmergeable_expert_lora_save` raises.

Both halves are folded: expert deltas inside each layer's `gather_expert_state_dict`, PEFT (attention)
deltas out of place into each tensor as it is written, with the weight sync's fold
([above](#online-rl--rollout-server-weight-sync)). The frozen base is never written, so training continues
unchanged. The fold happens
in the gathered EP save under mixin-managed FSDP2 (torchrun); it is rejected under accelerate-managed FSDP
and with `save_sharded_ep: true`.


### Streaming merge

`merge_peft_adapters.py --streaming` (automatic for an adapter holding native expert LoRA, whose in-memory merge is
refused) loads no model: it reads each tensor of the base's safetensors shards, folds the adapter into the ones it
targets and writes the result through the streaming shard writer, so host memory is the adapter, one tensor, the
fp32 expert deltas of the layer being folded and the writer's parts: about one expert layer's fp32 delta times a small
factor (the preflight budgets 3x), which at 397B scale is tens of GB per layer. Linear LoRA (rslora,
rank and alpha patterns, a LoRA bias) and DoRA fold through PEFT's own layer, so each tensor equals the in-memory
merge; native expert LoRA folds `scaling * bmm(A.float(), B.float())`, the weight sync's and the merge-on-save's
arithmetic, through the family's `ExpertLayout`, on any projection subset. `modules_to_save` tensors replace
the base's. Tensors keep the base's stored dtype and key spelling, so `--dtype` is ignored (logged) and `--device_map`,
`--fp32_norms` and `--task classification` are refused. The tool resolves every key the adapter rewrites against the base before the
first write and raises, with no output, when one is absent (a text-only class's spelling, a fused or renamed layout,
a tied head), when the base's experts are not in the family's export layout, when a stored tensor was resized (a
grown vocabulary), on embedding LoRA, and on a `training_provenance.json` recording neutralized GptOss sinks; merge
those in memory. The base's `config.json` and tokenizer files are copied, the adapter's tokenizer replaces them, and
the adapter config lands in `original_adapter_config/`.

**Adapters must train.** Trainer construction runs `verify_adapters_trainable` (`halo/model_loading/parallel/peft_setup.py`)
on the PEFT wrap before any mode wraps the model, so the check covers every mode alike. It raises when a tuner layer
holds no trainable parameter for an adapter meant to train: an active adapter whose config is not `inference_mode`,
counting every tensor in the layer's `adapter_layer_names` (`lora_A`/`lora_B`, `lora_embedding_*`,
`lora_magnitude_vector`, other tuners' own). It reads live `requires_grad` and holds no collective; every rank builds
the same wrap and raises alike. PEFT 0.21 and later re-enable adapters by matching `named_parameters()` names against
`named_modules()` prefixes, so `UlyssesModel` (its names omit the `original_attention` level) would leave every CP
adapter frozen; the pin `peft<0.19` keeps that path unreachable, and a bump needs a release that fixes
[peft#4181](https://github.com/huggingface/peft/issues/4181).

## Tests and benchmarks

```bash
# LoRA SFT under one shape (ep / ep_cp / etp / cp / fsdp, --qlora for a 4-bit base), the save
# reloaded on a fresh base; LoRA under TP is refused at construction (tests/cpu/trainers/test_expert_lora_tp_gate.py),
# adapters that left the wrap frozen likewise (tests/cpu/peft/test_adapters_trainable_check.py)
torchrun --nproc_per_node=2 tests/gpu/trainers/lora/test_lora_sft_modes.py --model gpt_oss_20b --mode ep --save-reload
torchrun --nproc_per_node=2 tests/gpu/trainers/lora/test_lora_sft_modes.py --model qwen3_0_6b --mode cp --save-reload --resume
# merge_expert_lora_on_save checkpoint: servable, and resumed exactly against an uninterrupted run
torchrun --nproc_per_node=2 tests/gpu/trainers/lora/test_lora_merged_save_resume.py \
    --family gpt_oss --adapters mixed
# weight syncs leave the frozen base bit-identical (no server), on every family the sync serves
# under each mode (fsdp dense; ep1 / ep2 / etp2 MoE)
torchrun --nproc_per_node=2 tests/gpu/trainers/lora/test_lora_weight_sync_exact.py \
    --family qwen3_moe --mode ep2 --adapters mixed

# Adapters on an on-policy run, asserted on the SERVED policy (needs a live vLLM server):
# attention LoRA under EP and pure ETP, native grouped expert LoRA under EP
torchrun --nproc_per_node=2 tests/gpu/trainers/grpo/test_online_grpo_vllm_e2e.py \
    --trainer online --mode expert_lora_ep2
# attention LoRA under FSDP2 DP, and the TP refusal at trainer construction
torchrun --nproc_per_node=2 tests/gpu/trainers/grpo/test_online_grpo_vllm_e2e.py \
    --trainer online --mode lora_tp2_rejected

# Dense LoRA/QLoRA benchmark
torchrun --nproc_per_node=1 tests/gpu/profiling/benchmark_sft_dense.py \
    --model qwen3-8b --seq 16384 --lora_r 64 --lora_all_linear --qlora
# Expert-targeting LoRA under EP
torchrun --nproc_per_node=8 tests/gpu/profiling/benchmark_sft_ep.py \
    --ep 2 --seq 4096 --lora_r 64 --lora_experts
```

Omit `--lora_all_linear` for attention-only; on the EP benchmark it means attention + experts, while
`--lora_experts` is the native grouped adapters alone.

## Related pages

- [Configuration Reference](https://whitecircle.com/halo/docs/agent-docs/reference/configuration-reference.md) — full `lora_*` / `use_peft` parameters
- [Expert Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/expert-parallelism.md) · [Tensor Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/tensor-parallelism.md)
- [BF16 Optimizer](https://whitecircle.com/halo/docs/agent-docs/optimization/bf16-optimizer.md) — composes with LoRA
- [LoRA](https://arxiv.org/abs/2106.09685) · [QLoRA](https://arxiv.org/abs/2305.14314) · [DoRA](https://arxiv.org/abs/2402.09353)
