Mistral4 (text backbone of Mistral3 VLMs)
Mistral4ForCausalLM is the text side of Mistral3ForConditionalGeneration — the LM in mistralai/Mistral-Small-4-* and any mistral3 VLM. It is a DeepSeek-V3 clone: softmax + group-topk routing with an always-on shared expert, MLA (multi-head latent attention) with llama-4 logarithmic position scaling, and YARN RoPE.
Per-axis support: Compatibility matrix. The Mistral4 suite runs EP+ETP as ep_size=2, expert_tp_size=4; the legal sibling ep4+etp2 is not in it. EP+CP is a valid shape whose only GPU test on this model is a tiny-model LoRA row: tests/gpu/trainers/lora/test_lora_merged_save_resume.py --family mistral4 --cp-size 2 (ep2+cp2).
Coverage: tests/gpu/parallelism/test_moe_all_parallelism.py --family mistral4 (one node per single-node mode under pytest -m "gpu and mistral4"; the single-node EP+CP coverage lives in the script's Cohere2 MoE rows); tests/cpu/parallelism/test_mistral4_registries.py pins the EP/CP/TP registry claims.
EP layer
EPMistral4MoELayer (halo/distributed/expert_parallel/families/mistral4.py) replaces Mistral4MoE.
- Routing: the EP layer calls
Mistral4TopkRouterand keeps only its logits (the router also returns weights and indices, which balancing and routing replay have to recompute);route_tokens_to_expertsthen does softmax + group-topk inline, mirroring the router's own arithmetic. Noe_score_correction_bias(unlike GLM-4);norm_topk_probandrouted_scaling_factorare honored. - Load balancing: Mistral4 has no bias slot (a plain softmax router, no correction-bias tensor), so
moe_balancing: bias_updateraises and the explicitbias_update_transientis the trainer-only opt-in (balancing modes).Mistral4ForCausalLM.forwarddeclares nooutput_router_logitsparameter either, soaux_losscannot work andautoresolves tononewith a warning naming the transient opt-in. - Routed experts:
Mistral4Experts, fusedgate_up_proj [E, 2M, H](F.linear convention; matmul-internal form[E, H, 2M]) anddown_proj [E, H, M]. Same layout as Qwen3.5 / GLM-4, so the sharedEPMoELayerBasefused-GLU helpers cover compute (Grouped GEMM on SM90+). - Shared expert:
Mistral4MLP, replicated per rank, summed after DeepEP combine. Intermediate size =moe_intermediate_size * n_shared_experts, as wide as a routed expert, so the shipped config launches withHALO_EP_SHARED_OVERLAP=1, which computes it on a side stream during the dispatch all-to-all (DeepEP). - Checkpoints: weights stored in matmul convention internally; the export's
FUSEDexpert layout transposes back to F.linear convention at save sofrom_pretrainedreloads as-is. The publicmistralai/Mistral-Small-4-119B-*checkpoint ships every language-model weight in FP8 — dequantize once withscripts/before_training/convert_mistral4_bf16.py(streams shard-by-shard, handles scalar and per-expert_scale_invlayouts) before training. - Activation: the routed-expert SwiGLU runs through the fused Triton kernel (
halo/kernels/fused_glu.py) when the gate is a genuine SiLU, logged asglu_combine=silu.
Packing
Packed documents stay isolated on both paths, but flash needs a toolkit patch:
Mistral4Attention.forward takes position_ids as an explicit parameter and forwards only
**kwargs to the attention interface, so the varlen detection never sees them.
patch_mistral4_flash_packed_position_ids (halo/models/families/mistral4.py) re-injects them
and is applied at load whenever the resolved implementation is a flash one. The dense path builds the
packed mask itself and is clean to grouped-GEMM reduction noise.
CP wrapper
Mistral4Attention → Mistral4UlyssesAttention (halo/distributed/context_parallel/families.py), on the shared MLAUlyssesAttention.
Three MLA-specific bits the wrapper handles:
Mismatched head dims —
qk_head_dim(nope + rope) andv_head_dimcan differ. Flash-attn's native GQA wants one head_dim, so V is padded toqk_head_dimbefore the kernel and the output cropped back.Shared rope head —
kv_a_proj_with_mqaproduces one rotary K vector broadcast to all heads. The broadcast is materialized before the Ulysses all-to-all so the head-dim scatter has a contiguous tensor.Llama-4 position scale — transformers' own
get_llama_4_attn_scalemultiplies Q after RoPE, in_apply_rotaryon the local chunk before the all-to-all, read off theposition_idsthe decoder hands the attention (this chunk's slice of the sequence's positions).A
rope_parameterswithoutllama_4_scaling_beta/original_max_position_embeddingsis refused at construction rather than trained unscaled.
RoPE follows config.rope_interleave: rotate-half (apply_rotary_pos_emb) or interleaved pairs (apply_rotary_pos_emb_interleave).
TP
Selective TP via apply_tp_to_attention_only() — Mistral4Attention is in TP_SHARDABLE_ATTENTION_CLASSES (halo/distributed/tensor_parallel/parallelize_attention.py). MLA reuses GLM-4's compress/expand layout, so the plan applies verbatim:
- Sharded (colwise):
q_b_proj,kv_b_proj(head-dim expansions). - Replicated:
q_a_proj,kv_a_proj_with_mqa,q_a_layernorm,kv_a_layernorm— low-rank compressions where sharding produces degenerate per-rank shapes. - Sharded (rowwise):
o_proj.
MoE blocks (Mistral4MoE, Mistral4Experts, Mistral4MLP) are skipped — they belong to EP, not TP.
ETP
Mistral4's expert layout matches GLM-4 / LFM-2 (contiguous halves [gate(M) | up(M)] in matmul-convention gate_up_proj [E, H, 2M]), so ETP works automatically via EPMoELayerBase._init_fused_glu_params: with expert_tp_size > 1 the helper splits halves before sharding and stores gate_proj / up_proj / down_proj separately on each ETP rank (intermediate slice [rank*M/tp : (rank+1)*M/tp]).
On save, gate/up shards are all-gathered, concatenated back into contiguous halves, transposed to F.linear convention. The shared expert (Mistral4MLP) stays replicated. See Expert Tensor Parallelism → Compatibility.
Liger kernel
Liger upstream ships no mistral4 applier; the toolkit covers mistral4 from its own spec in halo/model_loading/liger/family_specs.py. A mistral3 checkpoint resolves through its text tower: this spec for a mistral4 tower, upstream's mistral applier for a Mistral Small 3.x one.
With use_liger_kernel: true it swaps Mistral4RMSNorm for Liger's fused RMSNorm, Mistral4MLP for the toolkit's fused SwiGLU (the shared expert's (config, intermediate_size=None) constructor is inherited, not restated), routes cross-entropy through Liger, and offers the fused loss on Mistral4ForCausalLM for text-only checkpoints (off by default).
mistral3 checkpoints load as Mistral3ForConditionalGeneration, whose own head runs, so the Liger dispatch forces the fused loss off for them and keeps CE (Liger Kernels).
RoPE stays unfused — Mistral4 combines YARN with the llama-4 logarithmic scale, which Liger's liger_rotary_pos_emb doesn't model. The spec declares rope=False, and the Liger dispatch honors an applier's own rope default over its generic default-on, so rope stays off unless the user explicitly pins rope: true in liger_kernel_config — which the applier then refuses with NotImplementedError.
Model loading
Both Mistral3ForConditionalGeneration and Mistral4ForCausalLM are registered upstream under AutoModelForImageTextToText, not AutoModelForCausalLM. The loaders resolve the class via resolve_auto_model_class() — any model_type in transformers' MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES loads through AutoModelForImageTextToText, everything else falls back to AutoModelForCausalLM. That keeps the *ForConditionalGeneration wrapper intact, so saved checkpoints carry the right architectures entry and never drop the vision tower.
Pass model_class= to override; the same resolver backs scripts/before_training/patch_vocab.py.
Inside the multimodal wrapper the text backbone is at model.model.language_model. The EP / CP / TP / FSDP plumbing checks that path first (then its .layers), falling back to model.model.layers for the standalone causal LM, so VLM checkpoints get the same selective TP / FSDP2 treatment.
Configs
Mistral-Small-4-119B (mistralai/Mistral-Small-4-119B-2603) trains under EP at 32K max length. The expert count (128 routed + 1 shared, top-4) divides evenly across 8 and 16 ranks, so EP=8 (16 experts/rank) and EP=16 are the natural fits. Example: examples/sft/mistral4/mistral-small-4-119b-ultrachat-ep.yaml, whose fit is computed, not measured: this shape has not run on the full-size model.
Serving
vLLM 0.32.0 knows nothing of this family: mistral4 appears in neither its architecture registry nor any model_type mapping. The composite Mistral3ForConditionalGeneration is registered, but it builds its text tower through init_vllm_registered_model off text_config, whose mistral4 spelling resolves to no class.
The public mistralai/Mistral-Small-4-* repo serves only because vLLM detects its Mistral-native params.json layout and routes the whole model to PixtralForConditionalGeneration; a toolkit export is plain HF-format and has no such path. vLLM's generic transformers backend (--model-impl transformers) is the one remaining route, neither pinned nor verified here.
RL weight sync is refused at construction on both engines, off each client's UNSERVABLE_MODEL_TYPES: the sync forwards hub-spelled tensors into the server's own loader, and neither vLLM 0.32.0 nor SGLang 0.5.22 registers a Mistral4 class.
A vLLM thinking budget is refused too, by the trainer and by an eval reading a training config: vLLM serves the family with its mistral_common tokenizer, whose [THINK]/[/THINK] markers encode as plain text while the model writes control tokens, so the budget never counts.
Online and async GRPO on Mistral4 raise at trainer build rather than sync into a server that cannot hold the model (Rollout Servers). SFT, preference and offline methods are unaffected; they never touch a rollout engine. Offline GRPO always scores through the per-row chunked path, since the query scale steps at a position a left pad would move (Row positions).
Source on GitHubUpdated 18 November 2026