# What is a mixture-of-experts (MoE) model?

Mixture-of-experts (MoE) models are transformers whose feed-forward blocks are split into many experts, with a router that sends each token to only a few. They hold many parameters but use a small share for each token.

## How it works

In a dense [transformer](/glossary/transformer) every token passes through the same feed-forward block (MLP, for multilayer perceptron) in each layer. An MoE model replaces that block with many parallel ones, the experts, and a router. For each token the router scores the experts and sends the token to the top few. Their outputs are combined into one.

The attention layers usually stay shared. Only the MLP part is split.

## Why it matters

Total and active parameters come apart. A model named 35B-A3B holds about 35 billion parameters and uses about 3 billion for any one token. Compute per token follows the active count, and memory follows the total.

That makes inference cheap for the model's size and training harder: every expert has to sit in GPU memory, and the router has to keep [tokens spread across experts](/glossary/moe-load-balancing).

## In Halo

Halo supports fifteen MoE families, and ten have a cookbook. [Expert parallelism](/glossary/expert-parallelism) gives each GPU a share of the experts and sends each token to the GPU that holds its expert:

```bash
halo launch sft examples/sft/qwen3_5/qwen3.5-35b-a3b-ultrachat-ep.yaml -n 8
```

The [supported models](/halo/docs/models) page lists the families.

