# MoE load balancing

MoE load balancing keeps tokens spread evenly across the experts of a mixture-of-experts (MoE) model during training, so that no expert is overloaded and none is starved of gradient.

## Why it matters

In a [mixture-of-experts](/glossary/mixture-of-experts) model a router sends each token to a few experts. Nothing forces it to use them evenly. Left alone it can favor a handful: those are overloaded, and the rest see few tokens, get little gradient and stop improving. Under [expert parallelism](/glossary/expert-parallelism) uneven routing also means uneven work per GPU.

## How it works

- **Auxiliary loss.** A loss term computed from the router's logits penalizes uneven use. It is added to the training loss with a small coefficient.
- **Bias update.** Each expert gets a bias on its router score. After a step the bias is lowered for overloaded experts and raised for underused ones. No loss term is added.

## In Halo

`moe_balancing` sets the mode. Which modes work depends on the model family:

| Mode | Meaning |
| --- | --- |
| `aux_loss` | The model's router auxiliary loss, as in the [Qwen3 MoE](/glossary/how-to-fine-tune-qwen3) and [GPT-OSS](/glossary/how-to-fine-tune-gpt-oss) recipes |
| `bias_update` | Bias-update balancing, as for GLM-4.7-Flash, LFM2 and ZAYA1 |
| `bias_update_transient` | Bias-update balancing for a router with no exportable bias slot. Exported checkpoints serve without the bias |
| `none` | No balancing. Gemma 4 has no balancing path |

Expert load is logged per layer (`moe/load_cv`, `moe/dead_frac`). The [cookbooks](/halo/docs/cookbooks) show the mode per family.

