LFM-2 MoE Cookbook

Liquid AI's LFM2.5-8B-A1B and LFM2-24B-A2B mix short-convolution layers with full-attention layers. A sigmoid router with a selection bias picks four experts per token, out of 32 on the 8B and 64 on the 24B.

Support

Modes: the LFM-2 MoE row of the Supported Matrix; online RL: Rollout engines.

  • Checkpoints: LiquidAI/LFM2.5-8B-A1B and LiquidAI/LFM2-24B-A2B.
  • GPUs: the 8B runs at EP2 on two GPUs, 16 experts per GPU. The 24B runs at EP4 on four GPUs or EP8 on eight.

Recipes

Run these in the training container.

Full fine-tune

LFM2.5-8B-A1B on two GPUs:

Bash

halo launch sft examples/sft/lfm2/lfm2.5-8b-a1b-ultrachat-ep2.yaml -n 2 \
  --output_dir=/data/checkpoints/lfm2.5-8b-a1b-sft

The recipe leaves report_to at its default, none. To log to Weights & Biases, copy it and add report_to: wandb; this field can't be overridden on the command line.

LFM2-24B-A2B on four GPUs:

Bash

halo launch sft examples/sft/lfm2/lfm2.5-8b-a1b-ultrachat-ep2.yaml -n 4 \
  --model_name_or_path=LiquidAI/LFM2-24B-A2B --expert_parallel_size=4 \
  --output_dir=/data/checkpoints/lfm2-24b-a2b-sft

On eight GPUs, change these to -n 8 and --expert_parallel_size=8. EP4 on eight GPUs is rejected at startup.

Other layouts

Add one of these to the two-GPU command:

  • EP2 + TP2, when attention memory is the limit: --tensor_parallel_size=2. TP shards only the full-attention layers (6 of 24 on the 8B), so per-GPU memory drops much less than half.
  • Pure ETP2, to shard every expert instead of placing whole experts: --expert_parallel_size=1 --expert_tensor_parallel_size=2

LoRA

Bash

halo launch sft examples/sft/lfm2/lfm2.5-8b-a1b-ultrachat-ep2.yaml -n 2 \
  --use_peft=true --learning_rate=1e-4 \
  --lora_target_modules=q_proj,k_proj,v_proj,out_proj \
  --output_dir=/data/checkpoints/lfm2.5-8b-a1b-lora

out_proj is also the output projection of every short-convolution block (18 of 24 layers on the 8B), so this adapter trains both kinds of layer.

GRPO

Read Before a GRPO run first. This setup serves the SFT checkpoint from vLLM on GPUs 0–1 and trains on GPUs 2–3. On the host:

Bash

VLLM_MODEL=/data/checkpoints/lfm2.5-8b-a1b-sft VLLM_CUDA_DEVICES=0,1 VLLM_TP=2 VLLM_ENABLE_R3=1 \
VLLM_TOOL_PARSER=lfm2 \
  docker compose -f docker-compose.vllm.yml up -d vllm-server

In the training container, copy the template, set dataset, environment_type and rewards, then launch:

Bash

cp examples/grpo/environmental/environmental-grpo-template.yaml lfm2-grpo.yaml
CUDA_VISIBLE_DEVICES=2,3 halo launch grpo/environmental lfm2-grpo.yaml -n 2 \
  --model_name_or_path=/data/checkpoints/lfm2.5-8b-a1b-sft \
  --routing_replay=rollout --beta=0.0 \
  --output_dir=/data/checkpoints/lfm2.5-8b-a1b-grpo

Add --expert_parallel_size=2 to shard the experts across the two trainer GPUs. SGLang 0.5.22 also serves and weight-syncs LFM-2: add --rollout_backend=sglang --rollout_server_url=http://localhost:30000 and start the server with SGLANG_ENABLE_R3=1.

Settings that matter

  • Router balancing. The recipe sets moe_balancing: bias_update, which auto also picks under EP. Sigmoid routing can collapse onto a few experts during SFT. The updates land in the router's own expert_bias, so the trained bias ships in every checkpoint and the served model routes on it. LFM-2 has no router auxiliary loss. Without EP or grouped GEMM nothing carries the bias, so freeze the router instead with freeze_layers_patterns: ["*.feed_forward.gate.weight"].
  • Gradient checkpointing. The recipe trains 16,384-token packed rows without it, a shape that has not run on the full-size model. The fit on B200 and B300 is computed, not measured. On 80 GB GPUs add --gradient_checkpointing=true.
  • Tool parser. vLLM needs VLLM_TOOL_PARSER=lfm2. The default hermes leaves LFM-2's tool calls as text, and the run stops by its second round.

Limits

  • No CP: the short-convolution layers can't split the sequence across ranks.
  • No pipeline parallelism: LFM-2 is not a registered pipeline family.

Export and serve

Gathered saves write the hub's per-expert layout (experts.{i}.w1/w3/w2), which both engines read. Smoke-test with AutoModelForCausalLM (snippet). Serve from the host with SGLang on port 30000:

Bash

SGLANG_MODEL="$HALO_SCRATCH/checkpoints/lfm2.5-8b-a1b-sft" \
  docker compose -f docker-compose.sglang.yml up -d

Reference

Source on GitHubUpdated 18 November 2026