To fine-tune Command A+ with Halo, train the BF16 checkpoint CohereLabs/command-a-plus-05-2026-bf16 on eight NVIDIA B300 GPUs with expert parallelism of size 8 (EP8). EP8 is the only layout validated at full scale.
Steps
- Start the training container. Set up the host as installation describes, then start the cookbook container. Run the commands below inside it.
- Pick the BF16 (bfloat16) checkpoint.
CohereLabs/command-a-plus-05-2026-bf16is a mixture-of-experts model with 128 routed experts. The router selects eight and also runs four shared experts. - Write the config. Create
command-a-plus-sft.yamlfor supervised fine-tuning of all weights. Among its keys:YAML
model_name_or_path: CohereLabs/command-a-plus-05-2026-bf16 moe_balancing: bias_update_transient router_balancing_rate: 1.0e-3 expert_parallel_size: 8 attn_implementation: flash_attention_2 packing: false max_length: 1024 - Launch eight processes. The recipe assumes eight NVIDIA B300 GPUs. With expert parallelism of size 8 (EP8), each GPU owns 16 routed experts.
Bash
halo launch sft command-a-plus-sft.yaml -n 8 - Load or serve the result. The BF16 model needs at least four B200 GPUs for inference.
Common mistakes
- Longer sequences.
max_length: 1024withpacking: falseis the recipe's memory-validated shape. At 4096 the run hits a CUDA out-of-memory error a few steps in. - Plain
bias_update. It raises.bias_update_transientbalances expert selection during training, and exported checkpoints serve without the bias. - Unvalidated layouts. Only EP8 has a full-scale run. Validate any other layout, or a LoRA adapter, with a short run first.
- Online reinforcement learning. Online GRPO and async GRPO with environments are refused for this family. Offline GRPO works from the fine-tuned checkpoint.
With Halo
Halo keeps Command A+ in the native Hugging Face checkpoint format. The Command A+ cookbook has the full config, layouts with context, tensor and expert-tensor parallelism, an attention LoRA adapter, and serving with vLLM.
