Command A+ Cookbook
Command A+ is a 200B+ Cohere2 MoE inside a vision-language wrapper. It has 128 routed experts and picks eight per token with a sigmoid router. Four shared experts run on every token, and their output is averaged with the routed output. The recipe trains on text.
Support
Modes: the Command A+ (Cohere2 MoE) row of the Supported Matrix; online RL: Rollout engines.
Only EP8 has run on the full checkpoint, on Halo 1.0.0 or earlier. CP, TP, ETP, EP+CP, EP+TP and LoRA pass the GPU tests on a tiny model, so try a short run before a long one.
- Checkpoint:
CohereLabs/command-a-plus-05-2026-bf16, pinned to the revision in the recipe. - GPUs: eight B300s at EP8, 16 experts per GPU, at 1,024 tokens. The model card asks for four B200s for BF16 inference.
Recipes
Run these in the training container.
Full fine-tune
Bash
halo launch sft examples/sft/cohere2_moe/command-a-plus-ultrachat-ep.yaml -n 8 \
--output_dir=/data/checkpoints/command-a-plus-sftOther layouts
These have been tested only on a tiny model. Add one of them to the full fine-tune command:
- EP8 + CP2, for long sequences:
--context_parallel_size=2(the recipe already turns packing off) - EP8 + TP2, when attention memory is the limit:
--tensor_parallel_size=2 - Pure ETP8, to shard every expert instead of placing whole experts:
--expert_parallel_size=1 --expert_tensor_parallel_size=8 - EP2 + ETP4, experimental:
--expert_parallel_size=2 --expert_tensor_parallel_size=4
LoRA
Bash
halo launch sft examples/sft/cohere2_moe/command-a-plus-ultrachat-ep.yaml -n 8 \
--use_peft=true --learning_rate=1e-4 \
--lora_target_modules=q_proj,k_proj,v_proj,o_proj \
--output_dir=/data/checkpoints/command-a-plus-loraThe GPU tests cover LoRA, expert adapters included, on a tiny model: a merged save and an exact resume.
GRPO
Online and async GRPO are not available for this family (see Limits). Offline GRPO trains on pre-generated, scored completions and needs no server: see Offline GRPO.
Settings that matter
- Sequence length.
max_length: 1024peaks at about 255 GiB of the B300's ~268 GiB. That leaves DeepEP's buffer room to grow. At 4,096 the allocation fails once a longer batch forces it to grow. - Router balancing. The recipe sets
moe_balancing: bias_update_transientat the defaultrouter_balancing_rateof 1.0e-3. The model has no usable auxiliary loss and no exportable bias slot, so plainbias_updateraises. Exported checkpoints serve without the transient bias, so near-tied expert picks can differ between training and serving. Drop the line to train unbalanced but serve exactly what you trained. - Precision. The experts train in BF16 (
fp32_experts: false). FP32 masters for 200B of expert parameters do not fit. - Pinned settings. The recipe pins the model revision its memory numbers were measured on and
leaves attention to the loader, which picks FA4 on Blackwell. Those numbers come from FA2; the fit on
FA4 is computed, not measured. It keeps
packing: falseto stay on that shape, though text data supports packing.
Limits
- No online RL. Weight sync for this family has not been validated on either engine, so Halo refuses online and async GRPO when it builds the trainer.
- No pipeline parallelism: Cohere2 MoE is not a registered pipeline family.
- Serving an export is not tested end to end. The steps below follow from the engines' loaders.
Export and serve
The gathered save uses the hub's tensor names, vision tower included, and keeps the experts as transformers' fused pair.
- transformers: load the save with
AutoModelForImageTextToTextandAutoTokenizer, orAutoProcessorfor images (snippet). - vLLM 0.32.0: point it at the save. It reads the fused experts. It ties the output head to the embeddings, so a save whose training untied the head won't load; training keeps them tied unless it trained the output head on its own.
- SGLang 0.5.22: run
halo run unfuse-moe-expertsfirst. SGLang's Cohere2 loader reads one tensor per expert and skips the fused pair without an error. The rewrite gives the hub's exact layout.
Reference
- Cohere2 MoE model notes ↗: routing, the CP wrapper, test coverage
- Offline GRPO
- Model card
Source on GitHubUpdated 18 November 2026