MoE load balancing

MoE load balancing keeps tokens spread evenly across the experts of a mixture-of-experts (MoE) model during training, so that no expert is overloaded and none is starved of gradient.

Why it matters

In a mixture-of-experts model a router sends each token to a few experts. Nothing forces it to use them evenly. Left alone it can favor a handful: those are overloaded, and the rest see few tokens, get little gradient and stop improving. Under expert parallelism uneven routing also means uneven work per GPU.

How it works

  • Auxiliary loss. A loss term computed from the router's logits penalizes uneven use. It is added to the training loss with a small coefficient.
  • Bias update. Each expert gets a bias on its router score. After a step the bias is lowered for overloaded experts and raised for underused ones. No loss term is added.

In Halo

moe_balancing sets the mode. Which modes work depends on the model family:

ModeMeaning
aux_lossThe model's router auxiliary loss, as in the Qwen3 MoE and GPT-OSS recipes
bias_updateBias-update balancing, as for GLM-4.7-Flash, LFM2 and ZAYA1
bias_update_transientBias-update balancing for a router with no exportable bias slot. Exported checkpoints serve without the bias
noneNo balancing. Gemma 4 has no balancing path

Expert load is logged per layer (moe/load_cv, moe/dead_frac). The cookbooks show the mode per family.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL