How to fine-tune Mistral Small 4

To fine-tune Mistral Small 4 with Halo, convert mistralai/Mistral-Small-4-119B-2603 from FP8 to BF16 once, then train the converted checkpoint on eight GPUs with expert parallelism of size 8 (EP8).

Steps

  1. Start the training container. Set up the host as installation describes, then start the cookbook container. Run the commands below inside it.
  2. Download the checkpoint.

    Bash

    hf download mistralai/Mistral-Small-4-119B-2603 \
      --local-dir /data/models/mistral-small-4-119b-fp8
  3. Convert it to BF16 (bfloat16). The public checkpoint stores its expert weights in FP8, so convert it once before training with expert parallelism (EP).

    Bash

    python scripts/before_training/convert_mistral4_bf16.py \
      --model_id /data/models/mistral-small-4-119b-fp8 \
      --output_dir /data/models/mistral-small-4-119b-bf16
  4. Write the config. Create mistral4-sft.yaml for supervised fine-tuning of all weights. Among its keys:

    YAML

    model_name_or_path: /data/models/mistral-small-4-119b-bf16
    moe_balancing: bias_update_transient
    expert_parallel_size: 8
    attn_implementation: flash_attention_2
  5. Launch eight processes. The recipe starts with eight NVIDIA B300 GPUs. EP8 (expert_parallel_size: 8) places 16 routed experts on each.

    Bash

    halo launch sft mistral4-sft.yaml -n 8

Common mistakes

  • Too little disk. The source and destination checkpoints need about 500 GB in total.
  • Plain bias_update. It raises. The router has no exportable bias slot, so bias_update_transient balances routing in training only, and every exported checkpoint serves without the bias.
  • Serving and online reinforcement learning. Neither pinned engine (vLLM 0.26.0, SGLang 0.5.17) serves a Mistral 4 checkpoint that Halo exports. Run inference from Transformers. Online GRPO and async GRPO with environments are refused. Offline GRPO remains available.

With Halo

Mistral Small 4 is a mixture-of-experts model with 128 routed experts, four selected for each token, and one shared expert. Halo preserves its group-top-k router and shared expert. The Mistral 4 cookbook has the full config, layouts with context, tensor and expert-tensor parallelism, a LoRA adapter, and multimodal inference.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL