Mistral 4 MoE Cookbook
Mistral Small 4 119B has 128 routed experts plus a shared expert and picks four per token. It uses MLA attention and ships with a Pixtral vision encoder; the recipe trains on text and keeps the multimodal wrapper intact. The public checkpoint stores its language-model weights in FP8, so convert it to BF16 once before training.
Support
Modes: the Mistral4 MoE row of the Supported Matrix; online RL: Rollout engines.
- Checkpoint:
mistralai/Mistral-Small-4-119B-2603, converted to BF16. - GPUs: eight at EP8, 16 experts per GPU, at 32,000 tokens. This shape has not run on the full-size model, and its fit is computed, not measured.
- Disk: by arithmetic the BF16 copy takes about 240 GB (119B parameters at two bytes), beside the FP8 download.
Recipes
Run these in the training container.
Convert the checkpoint
Bash
halo run convert-mistral4-bf16 --model_id mistralai/Mistral-Small-4-119B-2603 \
--output_dir /data/models/mistral-small-4-119b-bf16The converter streams one shard at a time and writes a standard BF16 Hugging Face checkpoint.
Full fine-tune
Bash
HALO_EP_SHARED_OVERLAP=1 halo launch sft examples/sft/mistral4/mistral-small-4-119b-ultrachat-ep.yaml -n 8 \
--model_name_or_path=/data/models/mistral-small-4-119b-bf16 \
--output_dir=/data/checkpoints/mistral-small-4-sftOther layouts
Add one of these to the full fine-tune command:
- EP8 + CP2, for long sequences, which passes a tiny-model LoRA test only:
--context_parallel_size=2 --packing=false - EP8 + TP2, when attention memory is the limit:
--tensor_parallel_size=2. TP shards the MLA expansion and output projections and keeps the compression projections whole. - Pure ETP8, to shard every expert instead of placing whole experts:
--expert_parallel_size=1 --expert_tensor_parallel_size=8 - EP2 + ETP4, experimental:
--expert_parallel_size=2 --expert_tensor_parallel_size=4
LoRA
Bash
HALO_EP_SHARED_OVERLAP=1 halo launch sft examples/sft/mistral4/mistral-small-4-119b-ultrachat-ep.yaml -n 8 \
--model_name_or_path=/data/models/mistral-small-4-119b-bf16 \
--use_peft=true --learning_rate=1e-4 \
--lora_target_modules=q_a_proj,q_b_proj,kv_a_proj_with_mqa,kv_b_proj,o_proj \
--output_dir=/data/checkpoints/mistral-small-4-loraGRPO
Online and async GRPO are not available for this family (see Limits). Offline GRPO trains on pre-generated, scored completions and needs no server: see Offline GRPO.
Settings that matter
- Router balancing. The recipe sets
moe_balancing: bias_update_transient. The router has no bias slot, so plainbias_updateraises, and it has no auxiliary loss either. The transient bias balances training only: exported checkpoints serve without it, so near-tied expert picks can differ between training and serving. - Chat template. The recipe forces
jinja-templates/mistral4/mistral4-multiturn.jinja, becausemistral4-instruct.jinjaaccepts only single-turn rows and UltraChat is multi-turn. It marks completions withassistant_message_template: "[/INST]". - Shared expert. Launch with
HALO_EP_SHARED_OVERLAP=1, as the commands above do. The shared expert is as wide as a routed one, and the setting runs it on a side stream while the tokens travel to their experts, instead of after they return. - Attention. The recipe leaves
attn_implementationunset, so Halo picks FA4 on Blackwell and FA3 on Hopper. MLA's query/key and value widths are both 128 here, so FA4's general kernel runs it; the head_dim-256 kernel limits do not apply.
Limits
- No online RL. Neither vLLM 0.32.0 nor SGLang 0.5.22 has a
mistral4model class, so Halo refuses online and async GRPO when it builds the trainer. - Neither pinned engine serves a Halo export. vLLM serves the public repo only through its
Mistral-native
params.jsonlayout, which a Halo export does not have. vLLM's generic transformers backend (--model-impl transformers) is untested here. - LoRA is refused under TP, and attention TP can't combine with ETP.
- No pipeline parallelism: Mistral 4 is not a registered pipeline family.
Export and serve
The gathered save is a standard Hugging Face checkpoint with the vision tower intact. Run inference
with transformers: load it with AutoModelForImageTextToText and AutoProcessor, which take images
too (snippet). To serve the base model, point vLLM at the public
hub repo.
Reference
- Mistral 4 model notes ↗: routing, the CP and TP plans for MLA, the serving gap
- Offline GRPO
- Model card
Source on GitHubUpdated 18 November 2026