MoE load balancing keeps tokens spread evenly across the experts of a mixture-of-experts (MoE) model during training, so that no expert is overloaded and none is starved of gradient.
Why it matters
In a mixture-of-experts model a router sends each token to a few experts. Nothing forces it to use them evenly. Left alone it can favor a handful: those are overloaded, and the rest see few tokens, get little gradient and stop improving. Under expert parallelism uneven routing also means uneven work per GPU.
How it works
- Auxiliary loss. A loss term computed from the router's logits penalizes uneven use. It is added to the training loss with a small coefficient.
- Bias update. Each expert gets a bias on its router score. After a step the bias is lowered for overloaded experts and raised for underused ones. No loss term is added.
In Halo
moe_balancing sets the mode. Which modes work depends on the model family:
Expert load is logged per layer (moe/load_cv, moe/dead_frac). The cookbooks show the mode per family.
