LFM-2 MoE
Liquid AI's Lfm2MoeForCausalLM — a hybrid MoE (interleaved short-convolution and full-attention layers) with sigmoid routing and an optional non-trainable expert_bias buffer for top-k selection correction. EP, ETP, and selective TP are supported; CP is blocked by the sequence-axis short-conv layers.
Per-axis support: Compatibility matrix. Pipeline parallelism refuses the checkpoints at the tie gate, before the registry check: they tie their head (tie_word_embeddings is the config-class default) and ship no lm_head tensor.
EP layer
EPLfm2MoELayer (halo/distributed/expert_parallel/families/lfm2.py) replaces Lfm2MoeSparseMoeBlock.
- Routing: sigmoid scores, plus optional
expert_biasfor top-k selection only; the selected weights come from the unbiased scores, normalized to sum 1 whennorm_topk_prob(default), then scaled byrouted_scaling_factor.expert_biasis a buffer (not trainable), so it affects selection but doesn't appear in gradients and needs no grad-sync hook. - Storage: fused
gate_up_projanddown_projas 3D tensors. Compute via Grouped GEMM (fused-GLU path). - No shared experts (the EP layer extends
EPSharedExpertsMoELayerBasewithshared_experts=None, so the shared leg is skipped). - Gathered saves write the per-expert hub layout (
experts.{i}.w{1,3,2}.weight) via the spec'sFUSED_SPLIT_PER_EXPERTExpertLayout— the names both pinned engines' per-expert loaders read, so either serves the export and takes the RL weight sync. SGLang 0.5.22's LFM-2 loader also reads the fused pair (experts.gate_up_proj/experts.down_proj, out-features-major as trained). See Serving on vLLM / SGLang.
TP and ETP
Selective attention-only TP — Lfm2MoeAttention is in TP_SHARDABLE_ATTENTION_CLASSES (halo/distributed/tensor_parallel/parallelize_attention.py); Lfm2MoeSparseMoeBlock is skipped (handled by EP). Only the full-attention layers shard — the short-conv layers stay replicated, so the per-rank footprint falls by far less than 1/tp_size.
LFM-2 uses the fused contiguous-halves ETP layout: _init_fused_glu_params splits the [gate(M) | up(M)] halves and stores gate_proj / up_proj / down_proj separately when expert_tp_size > 1. The non-trainable expert_bias buffer is replicated across ETP ranks.
Packing
Documents stay isolated in both halves of the hybrid stack. The attention layers key on the packed
position_ids; for the Lfm2MoeShortConv layers the collators and SMPO's padding-free forward
additionally emit seq_idx for this family (segment_markers_for in halo/models/segment_markers.py
turns it on by model type) and both conv paths honor it, which zeroes the cross-document term exactly
rather than approximately.
Why CP isn't supported
LFM-2 is architecturally hybrid: Lfm2MoeDecoderLayer interleaves full_attention blocks with Lfm2MoeShortConv layers (an nn.Conv1d over the sequence axis via causal_conv1d_fn), selected per layer by config.layer_types.
The short-conv layers mix tokens along the sequence axis, so a Ulysses sequence split would sever the convolution receptive field across CP ranks. That is the same blocker as Qwen3.5/3.6's Qwen3_5MoeGatedDeltaNet. No Ulysses wrapper is provided for the hybrid stack, so CP is unavailable on released LFM-2 checkpoints.
Configs
The production SFT example is examples/sft/lfm2/lfm2.5-8b-a1b-ultrachat-ep2.yaml.
It trains LiquidAI/LFM2.5-8B-A1B on UltraChat with EP2 and gathered Hugging Face
checkpoints, on packed 16,384-token rows without gradient checkpointing. A row holds about 43 GB of
activations (LFM2-24B-A2B: about 70 GB). By computation, not measurement (this shape has not run on the full-size model), that fits beside the EP state
on B200/B300 but not on an 80 GB GPU, where --gradient_checkpointing=true brings it back. Each rank still sees 65,536 tokens
per optimizer step; the longer row spreads the per-micro-batch fixed cost over twice the tokens
(Throughput Benchmarks lists 35,517 tok/s/GPU at 16,384
against 31,262 at 8,192). The path is also exercised by tests/gpu/trainers/sft/test_sft_families.py --suite lfm2_moe
(LiquidAI/LFM2-24B-A2B, --mode fsdp and --mode ep). The LFM2.5 VL
self-distillation example, examples/distillation/lfm2/self-distill-lfm2.5-vl-1.6b.yaml,
uses a dense LiquidAI/LFM2.5-VL-1.6B checkpoint (text_config.model_type: lfm2,
no experts), so none of the MoE columns above apply to it.
Routing drift: sigmoid routing under SFT can collapse onto a handful of experts (same failure mode as Bailing). Two mitigations work — freeze the router weights with freeze_layers_patterns: ["*.feed_forward.gate.weight"], or set moe_balancing: bias_update (EP path), whose DeepSeek-V3 sign-updates land in the block's own expert_bias buffer, adopted whole so the trained bias exports with every checkpoint.
On a use_expert_bias: false checkpoint the slot is materialized at enable time (a zero buffer in the same sigmoid-score space, a semantic no-op at creation) and use_expert_bias flips in the exported config, so serving engines load and apply the trained bias there too.
moe_balancing: auto picks bias_update here. Lfm2MoeForCausalLM has no aux-loss path — no output_router_logits / router_aux_loss_coef config field, no router-logit output, no aux term in the loss — and a forward that declares no output_router_logits parameter is the signal auto keys on, so it takes the EP-layer bias wherever the EP layers apply (ep_size > 1 or use_grouped_gemm, the default under torchrun).
Without the EP layers nothing carries the bias and no aux term can reach the loss, so auto resolves to none with a warning; freeze the router there. See Performance & Balancing Flags and BalancingBiasCallback.
Source on GitHubUpdated 18 November 2026