# Cohere2 MoE (Command A+)

`Cohere2MoeForCausalLM` (`model_type: cohere2_moe`) is the text backbone of
`CohereLabs/command-a-plus-05-2026-bf16` — a `Cohere2VisionForConditionalGeneration` VLM
(`cohere2_vision`) with a SigLIP tower. Native in transformers, no `trust_remote_code`.

The block is Cohere's parallel residual: one `input_layernorm`, attention and MLP both read the
normed hidden, `hidden = residual + attn + mlp`. Attention interleaves sliding-window (RoPE,
interleaved GPT-J-style rotary) and full-attention (**NoPE** — no positional embedding) layers on a
period-4 pattern.

The real checkpoint runs GQA 128 Q / 8 KV heads at head_dim 128, 128 routed experts with top-8
sigmoid selection + top-k renorm, 4 shared experts combined by `average` (`(routed + shared) / 2`),
tied embeddings, and a `logit_scale` multiplier on the lm_head output (the config default is
`0.0625`).

Per-axis support: [Compatibility matrix](https://whitecircle.com/halo/docs/agent-docs/models.md#compatibility-matrix). Pipeline parallelism refuses the shipped checkpoints at the tie gate, before the registry check: they tie `lm_head` to `embed_tokens`.

Coverage: `tests/gpu/parallelism/test_moe_all_parallelism.py --family cohere2_moe` (ep8 / cp8 / tp8 /
ep8+cp2 / ep8+tp2 / etp8 / ep2+etp4 on a tiny synthetic checkpoint), plus EP=2 equivalence, SFT and CPU
contract suites (`tests/gpu/parallelism/ep/test_ep_family_vs_reference.py --family cohere2_moe`,
`tests/gpu/trainers/sft/test_sft_ep_roundtrip.py --family cohere2_moe`, `tests/cpu/models/test_cohere2_moe_support.py`).
LoRA is tiny-model verified by `tests/gpu/trainers/lora/test_lora_merged_save_resume.py --family cohere2_moe`
(expert and mixed adapters at ep2 / ep1 / ep2+cp2, merged save and exact resume); no full-scale LoRA
run.

## EP layer

`EPCohere2MoELayer` (`halo/distributed/expert_parallel/families/cohere2_moe.py`) replaces
`Cohere2MoeSparseMoeBlock`. Layers whose `mlp_layer_types` entry is `dense` instantiate a plain MLP
and are never wrapped.

- Routing: `Cohere2MoeTopKRouter` picks top-k on the RAW logits, then activates only the selected
  scores — softmax over the k scores, or sigmoid with `norm_topk_prob` renorm. The EP layer re-derives
  exactly that from the router's logits.

    The router module itself is still called, so the HF `router_logits` recorder keeps firing.

- Load balancing: the architecture has **no bias slot** and **no aux-loss wiring** (no
  `router_aux_loss_coef`; the loss never reads `router_logits`), so `moe_balancing: bias_update`
  **raises**, `aux_loss` cannot work, and `auto` resolves to `none` with a warning. The explicit
  `bias_update_transient` is the trainer-only opt-in ([balancing modes](https://whitecircle.com/halo/docs/agent-docs/training-methods/callbacks.md#moe-balancing-modes)).

- Routed experts: `Cohere2MoeExperts`, fused `gate_up_proj [E, 2I, H]` / `down_proj [E, H, I]` —
  the Qwen3.5/GLM-4 layout, so the shared fused-GLU helpers cover compute (grouped GEMM, 2 calls;
  fused SwiGLU Triton combine).
- Shared experts: one fused `Cohere2MoeMLP` (intermediate = `intermediate_size *
  num_shared_experts`), replicated per rank, combined after DeepEP combine by `sum` or `average`
  (the EP layer scales the summed output by 0.5).
- Checkpoints: the hub stores one tensor per expert (`experts.{i}.{gate,up,down}_proj.weight`);
  transformers fuses on load and reverts on save. The gather emits the fused pair, and
  `scripts/after_training/unfuse_moe_experts.py` rewrites a gathered save into the hub's per-expert
  layout.

## CP wrapper

`Cohere2MoeAttention` → `Cohere2MoeUlyssesAttention`
(`halo/distributed/context_parallel/families.py`), optimized `[B, S, H, D]` path. RoPE is
applied only where the HF layer applies it: sliding-window layers, plus the `force_rope`
dense-prefix override — full-attention layers pass Q/K through untouched (NoPE). The rotary is the
interleaved GPT-J form in fp32, matching `apply_rotary_pos_emb` bit-for-bit in bf16. The 8 KV heads
of the real checkpoint cap `cp_size` at 8; CP requires a flash attention implementation.

## TP

Selective TP: `Cohere2MoeAttention` is in `TP_SHARDABLE_ATTENTION_CLASSES` — standard `q_proj` /
`k_proj` / `v_proj` colwise, `o_proj` rowwise; embeddings and the tied lm_head stay replicated. MoE
blocks belong to EP, not TP.

## Packing

Isolated on every backend with **no toolkit patch**: the model forward feeds `position_ids` into
mask construction and forwards them through layer kwargs to the attention interface, so both the
dense packed mask and the flash varlen path engage. Pinned by
`tests/cpu/models/test_cohere2_moe_support.py` — if transformers drops that plumbing, the spy test
fails and the family needs the Mistral4/Zaya position-ids patch.

A text-only dataset takes the text pipeline, where packing is supported; an image dataset takes the
VLM processor pipeline, which rejects it. The shipped config keeps `packing: false` to stay on its
memory-validated shape.

## Loading and serving

- Lazy loading is off (`supports_lazy_loading=False` on its spec): no lazy load of the Command A+ checkpoint
  has been checked against `from_pretrained`, so every load routes through `from_pretrained`.
- A gathered save writes the hub namespace
  ([Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#serving-on-vllm--sglang)): the SigLIP tower under
  `model.vision_tower.vision_model.*`, the text tower under `model.language_model.*`, no
  `lm_head.weight` (the head is tied), and the fused expert pair.

    vLLM 0.32.0 registers `Cohere2VisionForConditionalGeneration` and `Cohere2MoeForCausalLM`, and
    its weight loader reads that save as written: the fused pair, the tower only with its
    `vision_model` level (a flattened key raises), no top-level `lm_head.weight` (one raises in the
    composite). Its Cohere2 MoE model computes logits from the input embeddings, so an export whose
    training untied the head does not load as the composite, and the text-only class drops the head
    and serves the tied one without an error.

    SGLang 0.5.22 registers both too, but its Cohere2 expert loader reads per-expert keys only and
    drops the fused pair without an error: run `unfuse_moe_experts.py` first. Its tower loader takes
    either spelling.

    Both verdicts come from the pinned engines' loaders; no export of this family has been served end
    to end.
- RL weight sync is refused at construction (the spec's `weight_sync_refusal`): no end-to-end sync
  has been validated on either pinned engine
  ([Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#which-families-each-engine-serves)).
- Liger covers the fused SwiGLU (`Cohere2MoeMLP`, i.e. the dense and shared-expert path), cross-entropy,
  the norm when the checkpoint sets `rms_norm_eps`, and, on a text-only `cohere2_moe` checkpoint, the
  fused loss ([Liger Kernels](https://whitecircle.com/halo/docs/agent-docs/optimization/liger-kernels.md#supported-models)).

    The norm is a llama-style `Cohere2MoeRMSNorm` only when `rms_norm_eps` is set; the `null` default
    builds `Cohere2MoeLayerNorm`, a mean-subtracting LayerNorm with no bias parameter, which stays eager.
    The fused loss folds `config.logit_scale` onto the hidden states so the scaled softmax survives the
    fusion.

    Command A+ is a `cohere2_vision` wrapper whose own head runs, so the fused loss is forced off there
    and CE serves instead. RoPE stays eager: GPT-J interleaved, on the sliding and `force_rope` layers.

## Configs

`examples/sft/cohere2_moe/command-a-plus-ultrachat-ep.yaml` runs text-only SFT of the VLM checkpoint at
EP=8. At `max_length: 1024` it peaks at ~255 of the B300's ~268 GiB; 4096 OOMs when DeepEP's elastic
buffer resizes. It pins `model_revision` to the layout those numbers hold for and leaves
attention unset, so the loader picks FA4 on Blackwell (128-wide heads, sliding window included). This shape has not run on this release.
