Cohere2 MoE (Command A+)

Cohere2MoeForCausalLM (model_type: cohere2_moe) is the text backbone of CohereLabs/command-a-plus-05-2026-bf16 — a Cohere2VisionForConditionalGeneration VLM (cohere2_vision) with a SigLIP tower. Native in transformers, no trust_remote_code.

The block is Cohere's parallel residual: one input_layernorm, attention and MLP both read the normed hidden, hidden = residual + attn + mlp. Attention interleaves sliding-window (RoPE, interleaved GPT-J-style rotary) and full-attention (NoPE — no positional embedding) layers on a period-4 pattern.

The real checkpoint runs GQA 128 Q / 8 KV heads at head_dim 128, 128 routed experts with top-8 sigmoid selection + top-k renorm, 4 shared experts combined by average ((routed + shared) / 2), tied embeddings, and a logit_scale multiplier on the lm_head output (the config default is 0.0625).

Per-axis support: Compatibility matrix. Pipeline parallelism refuses the shipped checkpoints at the tie gate, before the registry check: they tie lm_head to embed_tokens.

Coverage: tests/gpu/parallelism/test_moe_all_parallelism.py --family cohere2_moe (ep8 / cp8 / tp8 / ep8+cp2 / ep8+tp2 / etp8 / ep2+etp4 on a tiny synthetic checkpoint), plus EP=2 equivalence, SFT and CPU contract suites (tests/gpu/parallelism/ep/test_ep_family_vs_reference.py --family cohere2_moe, tests/gpu/trainers/sft/test_sft_ep_roundtrip.py --family cohere2_moe, tests/cpu/models/test_cohere2_moe_support.py). LoRA is tiny-model verified by tests/gpu/trainers/lora/test_lora_merged_save_resume.py --family cohere2_moe (expert and mixed adapters at ep2 / ep1 / ep2+cp2, merged save and exact resume); no full-scale LoRA run.

EP layer

EPCohere2MoELayer (halo/distributed/expert_parallel/families/cohere2_moe.py) replaces Cohere2MoeSparseMoeBlock. Layers whose mlp_layer_types entry is dense instantiate a plain MLP and are never wrapped.

  • Routing: Cohere2MoeTopKRouter picks top-k on the RAW logits, then activates only the selected scores — softmax over the k scores, or sigmoid with norm_topk_prob renorm. The EP layer re-derives exactly that from the router's logits.

    The router module itself is still called, so the HF router_logits recorder keeps firing.

  • Load balancing: the architecture has no bias slot and no aux-loss wiring (no router_aux_loss_coef; the loss never reads router_logits), so moe_balancing: bias_update raises, aux_loss cannot work, and auto resolves to none with a warning. The explicit bias_update_transient is the trainer-only opt-in (balancing modes).

  • Routed experts: Cohere2MoeExperts, fused gate_up_proj [E, 2I, H] / down_proj [E, H, I] — the Qwen3.5/GLM-4 layout, so the shared fused-GLU helpers cover compute (grouped GEMM, 2 calls; fused SwiGLU Triton combine).

  • Shared experts: one fused Cohere2MoeMLP (intermediate = intermediate_size * num_shared_experts), replicated per rank, combined after DeepEP combine by sum or average (the EP layer scales the summed output by 0.5).

  • Checkpoints: the hub stores one tensor per expert (experts.{i}.{gate,up,down}_proj.weight); transformers fuses on load and reverts on save. The gather emits the fused pair, and scripts/after_training/unfuse_moe_experts.py rewrites a gathered save into the hub's per-expert layout.

CP wrapper

Cohere2MoeAttention → Cohere2MoeUlyssesAttention (halo/distributed/context_parallel/families.py), optimized [B, S, H, D] path. RoPE is applied only where the HF layer applies it: sliding-window layers, plus the force_rope dense-prefix override — full-attention layers pass Q/K through untouched (NoPE). The rotary is the interleaved GPT-J form in fp32, matching apply_rotary_pos_emb bit-for-bit in bf16. The 8 KV heads of the real checkpoint cap cp_size at 8; CP requires a flash attention implementation.

TP

Selective TP: Cohere2MoeAttention is in TP_SHARDABLE_ATTENTION_CLASSES — standard q_proj / k_proj / v_proj colwise, o_proj rowwise; embeddings and the tied lm_head stay replicated. MoE blocks belong to EP, not TP.

Packing

Isolated on every backend with no toolkit patch: the model forward feeds position_ids into mask construction and forwards them through layer kwargs to the attention interface, so both the dense packed mask and the flash varlen path engage. Pinned by tests/cpu/models/test_cohere2_moe_support.py — if transformers drops that plumbing, the spy test fails and the family needs the Mistral4/Zaya position-ids patch.

A text-only dataset takes the text pipeline, where packing is supported; an image dataset takes the VLM processor pipeline, which rejects it. The shipped config keeps packing: false to stay on its memory-validated shape.

Loading and serving

  • Lazy loading is off (supports_lazy_loading=False on its spec): no lazy load of the Command A+ checkpoint has been checked against from_pretrained, so every load routes through from_pretrained.

  • A gathered save writes the hub namespace (Checkpoints): the SigLIP tower under model.vision_tower.vision_model.*, the text tower under model.language_model.*, no lm_head.weight (the head is tied), and the fused expert pair.

    vLLM 0.32.0 registers Cohere2VisionForConditionalGeneration and Cohere2MoeForCausalLM, and its weight loader reads that save as written: the fused pair, the tower only with its vision_model level (a flattened key raises), no top-level lm_head.weight (one raises in the composite). Its Cohere2 MoE model computes logits from the input embeddings, so an export whose training untied the head does not load as the composite, and the text-only class drops the head and serves the tied one without an error.

    SGLang 0.5.22 registers both too, but its Cohere2 expert loader reads per-expert keys only and drops the fused pair without an error: run unfuse_moe_experts.py first. Its tower loader takes either spelling.

    Both verdicts come from the pinned engines' loaders; no export of this family has been served end to end.

  • RL weight sync is refused at construction (the spec's weight_sync_refusal): no end-to-end sync has been validated on either pinned engine (Rollout Servers).

  • Liger covers the fused SwiGLU (Cohere2MoeMLP, i.e. the dense and shared-expert path), cross-entropy, the norm when the checkpoint sets rms_norm_eps, and, on a text-only cohere2_moe checkpoint, the fused loss (Liger Kernels).

    The norm is a llama-style Cohere2MoeRMSNorm only when rms_norm_eps is set; the null default builds Cohere2MoeLayerNorm, a mean-subtracting LayerNorm with no bias parameter, which stays eager. The fused loss folds config.logit_scale onto the hidden states so the scaled softmax survives the fusion.

    Command A+ is a cohere2_vision wrapper whose own head runs, so the fused loss is forced off there and CE serves instead. RoPE stays eager: GPT-J interleaved, on the sliding and force_rope layers.

Configs

examples/sft/cohere2_moe/command-a-plus-ultrachat-ep.yaml runs text-only SFT of the VLM checkpoint at EP=8. At max_length: 1024 it peaks at ~255 of the B300's ~268 GiB; 4096 OOMs when DeepEP's elastic buffer resizes. It pins model_revision to the layout those numbers hold for and leaves attention unset, so the loader picks FA4 on Blackwell (128-wide heads, sliding window included). This shape has not run on this release.

Source on GitHubUpdated 18 November 2026