# Muon Optimizer (Newton-Schulz Orthogonalization)

Muon projects each gradient matrix onto the nearest orthogonal matrix via Newton-Schulz iteration and uses that as the update direction on matrix-shaped parameters. Fused Triton kernels do the momentum/nesterov and weight-decay/update math per parameter; the orthogonalization itself runs the Newton-Schulz kernel (quack/cutlass, or a pure-torch fallback). Owning file: `halo/optimizers/muon.py`.

## Usage

```yaml
optim: muon
learning_rate: 3e-4
weight_decay: 0.1
```

```bash
torchrun --nproc_per_node=8 \
    scripts/training/sft.py examples/sft/qwen3/qwen3-4b-ultrachat.yaml \
    --optim=muon --learning_rate=3e-4 --weight_decay=0.1
```

`optim: muon` routes through `muon_optimizer_from_args` (`halo/optimizers/muon.py`), which forwards `learning_rate`, `weight_decay`, `adam_epsilon`, and the decay-parameter set.

`adam_beta1` / `adam_beta2` reach Muon's internal scalar AdamW only when the config sets them to something other than the HF default; Muon's tuned `DEFAULT_SCALAR_BETAS = (0.9, 0.95)` (`halo/optimizers/muon.py`) otherwise stands, since HF's `adam_beta2` default of `0.999` would silently retune the scalar optimizer of every Muon config that never mentions betas.

Everything else takes the `build_muon_optimizer` default: `momentum=0.95`, `nesterov=True`, `scalar_lr=lr`, `ns_algorithm="gram_newton_schulz"`, `ns_use_kernels=True`. To change those, call `build_muon_optimizer` directly:

```python
from halo.optimizers.muon import build_muon_optimizer
opt = build_muon_optimizer(model, lr=3e-4, weight_decay=0.1, ns_algorithm="standard_newton_schulz")
```

## Parameter splitting

`build_muon_optimizer` splits parameters by shape and module role:

- Hidden 2D+ weight matrices (Linear, conv) → Muon with Newton-Schulz.
- Everything below 2D (biases, norms) and the token embedding / `lm_head` → an internal scalar optimizer (`scalar_lr`, betas `(0.9, 0.95)`).

The embedding and head are 2D but excluded: their rows index the vocabulary, where Newton-Schulz is ill-suited and empirically harmful.

They are identified from the module tree the model declares (`get_input_embeddings()` / `get_output_embeddings()` plus a pooled `score`/`classifier` head), so a family spelling its embedding `tok_embeddings` or its head `output` is routed correctly. `_is_embedding_or_head`'s name-marker list is a second, always-active exclusion on top of the accessors, catching models that expose none.

The scalar optimizer is [AdamWBF16](https://whitecircle.com/halo/docs/agent-docs/optimization/bf16-optimizer.md), not `torch.optim.AdamW`: Muon requires bf16 hidden weights, so the scalar params are bf16 too and a nearest-rounding write would truncate every sub-ULP update to zero (norm weights, value ~1 and ULP 3.9e-3, would never move). Stepping each parameter's local shard, it also takes a mix of sharded DTensors and plain tensors under EP/TP, which a foreach/fused AdamW rejects.

Memory per param: ~4 B for 2D (bf16 weight + bf16 momentum), ~6 B for 1D (bf16 weight + bf16 AdamW states), plus transient Newton-Schulz buffers.

Same-shape params are stacked for the NS kernel in chunks of 32 (`_GNS_CHUNK_SIZE`), and each chunk's scaled output is copied straight back into its params' update buffers, so the step's transient is a bf16 update buffer for every param plus one stacked chunk and its Newton-Schulz output. Budget headroom for the update buffers (one bf16 copy of the matrix params), and expect the same transient once on resume (the restore primes state with a zero-LR step).

**Muon-routed weights must be bf16.** Construction raises when any 2D+ param it would orthogonalize is not
bf16, naming the offender, so the fp32 master-weight knobs (`fp32_non_ep_params`, `fp32_router`,
`fp32_experts`) are incompatible with `optim: muon`. It also raises if the model has no 2D+ parameter at all.
Setting `bf16_optimizer: true` alongside `optim: muon` raises: both select an optimizer.

`state_dict`/`load_state_dict` are overridden to merge the scalar optimizer's moments into the standard
per-param index space; upstream Muon serializes only its own state, which would reset them on resume.

## Stochastic rounding on the bf16 weight write

`_weight_decay_update_kernel` computes the update in fp32 and rounds the new weight to bf16 with stochastic rounding, through the same `sr_round_to_bf16` step as [AdamWBF16](https://whitecircle.com/halo/docs/agent-docs/optimization/bf16-optimizer.md) (`halo/optimizers/stochastic_rounding.py`). The orthogonalized update has magnitude ~`lr` after RMS-norm scaling, which at typical fine-tune LRs is below the bf16 weight ULP; nearest rounding would truncate the entire update to zero.

SR keeps the write unbiased so small updates accumulate. Each seed is keyed by the parameter's step count (a `step` entry in Muon's per-param state) and its name, under a key apart from the internal AdamWBF16's (see [AdamWBF16](https://whitecircle.com/halo/docs/agent-docs/optimization/bf16-optimizer.md#stochastic-rounding)), so replicas stay bit-identical and a resume rounds as the uninterrupted run would.

A device skip flag (`skip_next_step_if`, as on [AdamWBF16](https://whitecircle.com/halo/docs/agent-docs/optimization/bf16-optimizer.md#stochastic-rounding)) reaches both legs: the momentum and weight kernels write nothing under it, and the internal AdamWBF16 skips with them.

## Newton-Schulz kernel backend

The orthogonalization matmuls can run on the `quack-kernels` / `nvidia-cutlass-dsl` CUDA backend (`ns_use_kernels`, on by default). `build_muon_optimizer` probes the kernel on a tiny tensor at construction; if the call raises — missing package, import error, or ABI mismatch — it warns and falls back to a numerically-equivalent pure-torch path that is slower.

The `Dockerfile` force-installs the ABI-compatible pair (`quack-kernels==0.6.4`, `nvidia-cutlass-dsl==4.6.2`) over `uv.lock`'s `0.4.1` / `4.4.2`, so the probe passes and the kernel path is the active one in the images. A bare host on the lockfile versions takes the fallback.

## Benchmark

Optimizer micro-benchmark on one B300, Halo 1.1.0 or earlier (`tests/gpu/optimizers/bench_muon.py --hidden 4096 --layers 8`: synthetic FFN, 8 layers × 4096 × 16384, 1.61B params; median of 20 timed steps; Blackwell image), isolating the optimizer step:

| Optimizer | Optimizer step | Peak mem vs AdamW |
|---|:---:|:---:|
| AdamW (fused) | 7.7 ms | — |
| AdamWBF16 (SR) | 4.9 ms | −0.0% |
| Muon (Gram–Newton–Schulz) | 82.4 ms | +68% |
| Muon (standard NS) | 113.7 ms | — |

Muon's step costs 11–15× fused AdamW's here; the script prints no memory row for standard NS. This FFN is worst-case: it is all matrix params, so the Newton-Schulz cost is unamortized. On a real transformer, Newton-Schulz runs on smaller per-layer matrices amortized over a larger fwd+bwd, so the end-to-end slowdown is far smaller; measure on your model.

The fused Triton kernels replace upstream Muon's per-batch Python loop + `torch.compile`.

Run `tests/gpu/optimizers/bench_muon.py` or `tests/gpu/optimizers/bench_muon_qwen3_5.py` (real Qwen3.5-2B) for tables on your hardware.

## Compatibility

Works with FSDP2, gradient checkpointing, and every parallelism mode but PP — the kernels operate on the local shard after `to_local` (`halo/runtime/dtensor.py`) unwraps DTensors. Pipeline parallelism refuses it at trainer construction (`reject_pp_named_optimizer`): no PP run validates it.

That implies an approximation: Newton-Schulz orthogonalization runs on each rank's local shard, not the full matrix, so the Muon update depends on the sharding layout and world size. The same model sharded differently takes a different update.

Shard-local NS is the accepted trade-off, keeping the step communication-free, not equivalence with single-GPU Muon; re-tune Muon hyperparameters when the world size changes materially. 3D expert tensors are exempt under EP and FSDP2: they shard on the expert dim, so each expert matrix orthogonalizes whole. The approximation bites every 2D weight a rank holds only part of — dense weights under FSDP2 and TP (either split dim), and 2D expert stacks such as gpt-oss's expert biases (`[E_local, M]` under EP, dim-0 shards under FSDP2) — and, under expert-TP (`expert_tp_size > 1`), every expert matrix, since each expert's FFN dim is split across the ETP group.

Two further consequences: NS normalizes its input, so `max_grad_norm` clipping constrains only the scalar/embedding AdamW leg, not the Muon-routed 2D updates; and the NS-kernel availability probe is an all-ranks-agree reduction, so a rank that cannot JIT the kernels drops every rank to the pure-torch path — replica updates stay bitwise identical.

## References

- [Muon is Scalable for LLM Training](https://arxiv.org/abs/2502.16982) — weight decay + per-parameter update-scale adjustment for Muon at scale
- [gram-newton-schulz](https://github.com/Dao-AILab/gram-newton-schulz) — upstream implementation (a git dependency; `uv.lock` pins the commit)
