# How to fine-tune Command A+

To fine-tune Command A+ with Halo, train the BF16 checkpoint CohereLabs/command-a-plus-05-2026-bf16 on eight NVIDIA B300 GPUs with expert parallelism of size 8 (EP8). EP8 is the only layout validated at full scale.

## Steps

1. **Start the training container.** Set up the host as [installation](/halo/docs/installation) describes, then start the [cookbook container](/halo/docs/cookbooks#start-the-training-container). Run the commands below inside it.
2. **Pick the BF16 (bfloat16) checkpoint.** `CohereLabs/command-a-plus-05-2026-bf16` is a [mixture-of-experts](/glossary/mixture-of-experts) model with 128 routed experts. The router selects eight and also runs four shared experts.
3. **Write the config.** Create `command-a-plus-sft.yaml` for supervised fine-tuning of all weights. Among its keys:
   ```yaml
   model_name_or_path: CohereLabs/command-a-plus-05-2026-bf16
   moe_balancing: bias_update_transient
   router_balancing_rate: 1.0e-3
   expert_parallel_size: 8
   attn_implementation: flash_attention_2
   packing: false
   max_length: 1024
   ```
4. **Launch eight processes.** The recipe assumes eight NVIDIA B300 GPUs. With [expert parallelism](/glossary/expert-parallelism) of size 8 (EP8), each GPU owns 16 routed experts.
   ```bash
   halo launch sft command-a-plus-sft.yaml -n 8
   ```
5. **Load or serve the result.** The BF16 model needs at least four B200 GPUs for inference.

## Common mistakes

- **Longer sequences.** `max_length: 1024` with `packing: false` is the recipe's memory-validated shape. At 4096 the run hits a CUDA out-of-memory error a few steps in.
- **Plain `bias_update`.** It raises. `bias_update_transient` [balances expert selection](/glossary/moe-load-balancing) during training, and exported checkpoints serve without the bias.
- **Unvalidated layouts.** Only EP8 has a full-scale run. Validate any other layout, or a [LoRA](/glossary/lora) adapter, with a short run first.
- **Online reinforcement learning.** Online GRPO and async GRPO with environments are refused for this family. [Offline GRPO](/glossary/offline-grpo) works from the fine-tuned checkpoint.

## With Halo

Halo keeps Command A+ in the native Hugging Face checkpoint format. The [Command A+ cookbook](/halo/docs/cookbooks/halo-command-a-plus-cookbook) has the full config, layouts with context, tensor and expert-tensor parallelism, an attention LoRA adapter, and serving with vLLM.

