To fine-tune Mistral Small 4 with Halo, convert mistralai/Mistral-Small-4-119B-2603 from FP8 to BF16 once, then train the converted checkpoint on eight GPUs with expert parallelism of size 8 (EP8).
Steps
- Start the training container. Set up the host as installation describes, then start the cookbook container. Run the commands below inside it.
- Download the checkpoint.
Bash
hf download mistralai/Mistral-Small-4-119B-2603 \ --local-dir /data/models/mistral-small-4-119b-fp8 - Convert it to BF16 (bfloat16). The public checkpoint stores its expert weights in FP8, so convert it once before training with expert parallelism (EP).
Bash
python scripts/before_training/convert_mistral4_bf16.py \ --model_id /data/models/mistral-small-4-119b-fp8 \ --output_dir /data/models/mistral-small-4-119b-bf16 - Write the config. Create
mistral4-sft.yamlfor supervised fine-tuning of all weights. Among its keys:YAML
model_name_or_path: /data/models/mistral-small-4-119b-bf16 moe_balancing: bias_update_transient expert_parallel_size: 8 attn_implementation: flash_attention_2 - Launch eight processes. The recipe starts with eight NVIDIA B300 GPUs. EP8 (
expert_parallel_size: 8) places 16 routed experts on each.Bash
halo launch sft mistral4-sft.yaml -n 8
Common mistakes
- Too little disk. The source and destination checkpoints need about 500 GB in total.
- Plain
bias_update. It raises. The router has no exportable bias slot, sobias_update_transientbalances routing in training only, and every exported checkpoint serves without the bias. - Serving and online reinforcement learning. Neither pinned engine (vLLM 0.26.0, SGLang 0.5.17) serves a Mistral 4 checkpoint that Halo exports. Run inference from Transformers. Online GRPO and async GRPO with environments are refused. Offline GRPO remains available.
With Halo
Mistral Small 4 is a mixture-of-experts model with 128 routed experts, four selected for each token, and one shared expert. Halo preserves its group-top-k router and shared expert. The Mistral 4 cookbook has the full config, layouts with context, tensor and expert-tensor parallelism, a LoRA adapter, and multimodal inference.
