Cohere2 MoE (Command A+)
Cohere2MoeForCausalLM (model_type: cohere2_moe) is the text backbone of
CohereLabs/command-a-plus-05-2026-bf16 — a Cohere2VisionForConditionalGeneration VLM
(cohere2_vision) with a SigLIP tower. Native in transformers, no trust_remote_code.
The block is Cohere's parallel residual: one input_layernorm, attention and MLP both read the
normed hidden, hidden = residual + attn + mlp. Attention interleaves sliding-window (RoPE,
interleaved GPT-J-style rotary) and full-attention (NoPE — no positional embedding) layers on a
period-4 pattern.
The real checkpoint runs GQA 128 Q / 8 KV heads at head_dim 128, 128 routed experts with top-8
sigmoid selection + top-k renorm, 4 shared experts combined by average ((routed + shared) / 2),
tied embeddings, and a logit_scale multiplier on the lm_head output (the config default is
0.0625).
Per-axis support: Compatibility matrix. Pipeline parallelism refuses the shipped checkpoints at the tie gate, before the registry check: they tie lm_head to embed_tokens.
Coverage: tests/gpu/parallelism/test_moe_all_parallelism.py --family cohere2_moe (ep8 / cp8 / tp8 /
ep8+cp2 / ep8+tp2 / etp8 / ep2+etp4 on a tiny synthetic checkpoint), plus EP=2 equivalence, SFT and CPU
contract suites (tests/gpu/parallelism/ep/test_ep_family_vs_reference.py --family cohere2_moe,
tests/gpu/trainers/sft/test_sft_ep_roundtrip.py --family cohere2_moe, tests/cpu/models/test_cohere2_moe_support.py).
LoRA is tiny-model verified by tests/gpu/trainers/lora/test_lora_merged_save_resume.py --family cohere2_moe
(expert and mixed adapters at ep2 / ep1 / ep2+cp2, merged save and exact resume); no full-scale LoRA
run.
EP layer
EPCohere2MoELayer (halo/distributed/expert_parallel/families/cohere2_moe.py) replaces
Cohere2MoeSparseMoeBlock. Layers whose mlp_layer_types entry is dense instantiate a plain MLP
and are never wrapped.
Routing:
Cohere2MoeTopKRouterpicks top-k on the RAW logits, then activates only the selected scores — softmax over the k scores, or sigmoid withnorm_topk_probrenorm. The EP layer re-derives exactly that from the router's logits.The router module itself is still called, so the HF
router_logitsrecorder keeps firing.Load balancing: the architecture has no bias slot and no aux-loss wiring (no
router_aux_loss_coef; the loss never readsrouter_logits), somoe_balancing: bias_updateraises,aux_losscannot work, andautoresolves tononewith a warning. The explicitbias_update_transientis the trainer-only opt-in (balancing modes).Routed experts:
Cohere2MoeExperts, fusedgate_up_proj [E, 2I, H]/down_proj [E, H, I]— the Qwen3.5/GLM-4 layout, so the shared fused-GLU helpers cover compute (grouped GEMM, 2 calls; fused SwiGLU Triton combine).Shared experts: one fused
Cohere2MoeMLP(intermediate =intermediate_size * num_shared_experts), replicated per rank, combined after DeepEP combine bysumoraverage(the EP layer scales the summed output by 0.5).Checkpoints: the hub stores one tensor per expert (
experts.{i}.{gate,up,down}_proj.weight); transformers fuses on load and reverts on save. The gather emits the fused pair, andscripts/after_training/unfuse_moe_experts.pyrewrites a gathered save into the hub's per-expert layout.
CP wrapper
Cohere2MoeAttention → Cohere2MoeUlyssesAttention
(halo/distributed/context_parallel/families.py), optimized [B, S, H, D] path. RoPE is
applied only where the HF layer applies it: sliding-window layers, plus the force_rope
dense-prefix override — full-attention layers pass Q/K through untouched (NoPE). The rotary is the
interleaved GPT-J form in fp32, matching apply_rotary_pos_emb bit-for-bit in bf16. The 8 KV heads
of the real checkpoint cap cp_size at 8; CP requires a flash attention implementation.
TP
Selective TP: Cohere2MoeAttention is in TP_SHARDABLE_ATTENTION_CLASSES — standard q_proj /
k_proj / v_proj colwise, o_proj rowwise; embeddings and the tied lm_head stay replicated. MoE
blocks belong to EP, not TP.
Packing
Isolated on every backend with no toolkit patch: the model forward feeds position_ids into
mask construction and forwards them through layer kwargs to the attention interface, so both the
dense packed mask and the flash varlen path engage. Pinned by
tests/cpu/models/test_cohere2_moe_support.py — if transformers drops that plumbing, the spy test
fails and the family needs the Mistral4/Zaya position-ids patch.
A text-only dataset takes the text pipeline, where packing is supported; an image dataset takes the
VLM processor pipeline, which rejects it. The shipped config keeps packing: false to stay on its
memory-validated shape.
Loading and serving
Lazy loading is off (
supports_lazy_loading=Falseon its spec): no lazy load of the Command A+ checkpoint has been checked againstfrom_pretrained, so every load routes throughfrom_pretrained.A gathered save writes the hub namespace (Checkpoints): the SigLIP tower under
model.vision_tower.vision_model.*, the text tower undermodel.language_model.*, nolm_head.weight(the head is tied), and the fused expert pair.vLLM 0.32.0 registers
Cohere2VisionForConditionalGenerationandCohere2MoeForCausalLM, and its weight loader reads that save as written: the fused pair, the tower only with itsvision_modellevel (a flattened key raises), no top-levellm_head.weight(one raises in the composite). Its Cohere2 MoE model computes logits from the input embeddings, so an export whose training untied the head does not load as the composite, and the text-only class drops the head and serves the tied one without an error.SGLang 0.5.22 registers both too, but its Cohere2 expert loader reads per-expert keys only and drops the fused pair without an error: run
unfuse_moe_experts.pyfirst. Its tower loader takes either spelling.Both verdicts come from the pinned engines' loaders; no export of this family has been served end to end.
RL weight sync is refused at construction (the spec's
weight_sync_refusal): no end-to-end sync has been validated on either pinned engine (Rollout Servers).Liger covers the fused SwiGLU (
Cohere2MoeMLP, i.e. the dense and shared-expert path), cross-entropy, the norm when the checkpoint setsrms_norm_eps, and, on a text-onlycohere2_moecheckpoint, the fused loss (Liger Kernels).The norm is a llama-style
Cohere2MoeRMSNormonly whenrms_norm_epsis set; thenulldefault buildsCohere2MoeLayerNorm, a mean-subtracting LayerNorm with no bias parameter, which stays eager. The fused loss foldsconfig.logit_scaleonto the hidden states so the scaled softmax survives the fusion.Command A+ is a
cohere2_visionwrapper whose own head runs, so the fused loss is forced off there and CE serves instead. RoPE stays eager: GPT-J interleaved, on the sliding andforce_ropelayers.
Configs
examples/sft/cohere2_moe/command-a-plus-ultrachat-ep.yaml runs text-only SFT of the VLM checkpoint at
EP=8. At max_length: 1024 it peaks at ~255 of the B300's ~268 GiB; 4096 OOMs when DeepEP's elastic
buffer resizes. It pins model_revision to the layout those numbers hold for and leaves
attention unset, so the loader picks FA4 on Blackwell (128-wide heads, sliding window included). This shape has not run on this release.
Source on GitHubUpdated 18 November 2026