# How to fine-tune Qwen3.5 and Qwen3.6

To fine-tune Qwen3.5 and Qwen3.6 with Halo, train Qwen/Qwen3.6-35B-A3B or Qwen/Qwen3.5-35B-A3B on eight GPUs with expert parallelism of size 8 (EP8), Flash Attention 2 and transient bias-update balancing.

## Steps

1. **Start the training container.** Set up the host as [installation](/halo/docs/installation) describes, then start the [cookbook container](/halo/docs/cookbooks#start-the-training-container). Run the commands below inside it.
2. **Pick the checkpoint.** `Qwen/Qwen3.6-35B-A3B` and `Qwen/Qwen3.5-35B-A3B` take the same recipe. Both are [mixture-of-experts](/glossary/mixture-of-experts) models with 256 routed experts, eight selected for each token.
3. **Write the config.** Create `qwen3.6-sft.yaml` for supervised fine-tuning of all weights. Among its keys:
   ```yaml
   model_name_or_path: Qwen/Qwen3.6-35B-A3B
   moe_balancing: bias_update_transient
   expert_parallel_size: 8
   attn_implementation: flash_attention_2
   padding_free: false
   ```
4. **Launch eight processes.** The recipe starts with eight NVIDIA B300 GPUs. With [expert parallelism](/glossary/expert-parallelism) of size 8 (EP8), each GPU holds 32 experts.
   ```bash
   halo launch sft qwen3.6-sft.yaml -n 8
   ```
5. **Train Qwen3.5 instead.** Replace the model name, revision and output directory with the values in the cookbook's checkpoint table.

## Common mistakes

- **Flash Attention 4.** Its backward pass emits NaN gradients on this architecture, so the loader demotes it to SDPA (scaled dot-product attention). Keep [Flash Attention](/glossary/flash-attention) 2.
- **Another balancing mode.** `aux_loss` is refused and plain `bias_update` raises. `bias_update_transient` [balances routing](/glossary/moe-load-balancing) during training, but every exported checkpoint serves without the bias.
- **Padding-free batches.** Keep `padding_free: false`, or the run crashes.

## With Halo

Halo does not support context parallelism for this family, and tensor parallelism is limited to two GPUs. The [Qwen3.5 and Qwen3.6 cookbook](/halo/docs/cookbooks/halo-qwen3.5-qwen3.6-moe-cookbook) has the full config, the tensor and expert-tensor layouts, a [LoRA](/glossary/lora) adapter, and reinforcement learning with GRPO.

