How to fine-tune Command A+

To fine-tune Command A+ with Halo, train the BF16 checkpoint CohereLabs/command-a-plus-05-2026-bf16 on eight NVIDIA B300 GPUs with expert parallelism of size 8 (EP8). EP8 is the only layout validated at full scale.

Steps

  1. Start the training container. Set up the host as installation describes, then start the cookbook container. Run the commands below inside it.
  2. Pick the BF16 (bfloat16) checkpoint. CohereLabs/command-a-plus-05-2026-bf16 is a mixture-of-experts model with 128 routed experts. The router selects eight and also runs four shared experts.
  3. Write the config. Create command-a-plus-sft.yaml for supervised fine-tuning of all weights. Among its keys:

    YAML

    model_name_or_path: CohereLabs/command-a-plus-05-2026-bf16
    moe_balancing: bias_update_transient
    router_balancing_rate: 1.0e-3
    expert_parallel_size: 8
    attn_implementation: flash_attention_2
    packing: false
    max_length: 1024
  4. Launch eight processes. The recipe assumes eight NVIDIA B300 GPUs. With expert parallelism of size 8 (EP8), each GPU owns 16 routed experts.

    Bash

    halo launch sft command-a-plus-sft.yaml -n 8
  5. Load or serve the result. The BF16 model needs at least four B200 GPUs for inference.

Common mistakes

  • Longer sequences. max_length: 1024 with packing: false is the recipe's memory-validated shape. At 4096 the run hits a CUDA out-of-memory error a few steps in.
  • Plain bias_update. It raises. bias_update_transient balances expert selection during training, and exported checkpoints serve without the bias.
  • Unvalidated layouts. Only EP8 has a full-scale run. Validate any other layout, or a LoRA adapter, with a short run first.
  • Online reinforcement learning. Online GRPO and async GRPO with environments are refused for this family. Offline GRPO works from the fine-tuned checkpoint.

With Halo

Halo keeps Command A+ in the native Hugging Face checkpoint format. The Command A+ cookbook has the full config, layouts with context, tensor and expert-tensor parallelism, an attention LoRA adapter, and serving with vLLM.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL