How to fine-tune LFM2

To fine-tune LFM2 with Halo, train LiquidAI/LFM2.5-8B-A1B on two GPUs with expert parallelism of size 2 (EP2) and bias-update balancing. The same recipe covers LiquidAI/LFM2-24B-A2B on four GPUs with EP4.

Steps

  1. Start the training container. Set up the host as installation describes, then start the cookbook container. Run the commands below inside it.
  2. Pick the checkpoint. LFM2.5-8B-A1B has 32 routed experts and starts on two NVIDIA B300 GPUs. LFM2-24B-A2B has 64 and starts on four or eight. Both are mixture-of-experts models.
  3. Write the config. Create lfm2-sft.yaml for supervised fine-tuning of all weights. Among its keys:

    YAML

    model_name_or_path: LiquidAI/LFM2.5-8B-A1B
    moe_balancing: bias_update
    expert_parallel_size: 2
    attn_implementation: flash_attention_2
  4. Launch two processes. With expert parallelism of size 2 (EP2), each GPU owns 16 experts.

    Bash

    halo launch sft lfm2-sft.yaml -n 2
  5. Scale to LFM2-24B-A2B. Set model_name_or_path: LiquidAI/LFM2-24B-A2B and expert_parallel_size: 4, change output_dir, and launch with -n 4.

Common mistakes

  • Context parallelism. Keep context parallelism disabled for all released LFM2 checkpoints. The short-convolution layers operate across the sequence axis and cannot be split.
  • EP4 on eight GPUs. It is rejected at config time. Raise expert_parallel_size to 8, or use ep4 + etp2.
  • Looking for an auxiliary loss. LFM2 has no native router auxiliary loss. moe_balancing: bias_update corrects expert selection during training.

With Halo

Halo gathers the fused expert weights when it saves. Tensor parallelism shards only the full-attention layers, so the footprint per GPU falls by considerably less than 1/tp_size. The LFM2 cookbook has the full config, layouts with tensor and expert-tensor parallelism, an attention LoRA adapter, and reinforcement learning with GRPO.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL