To fine-tune Qwen3.5 and Qwen3.6 with Halo, train Qwen/Qwen3.6-35B-A3B or Qwen/Qwen3.5-35B-A3B on eight GPUs with expert parallelism of size 8 (EP8), Flash Attention 2 and transient bias-update balancing.
Steps
- Start the training container. Set up the host as installation describes, then start the cookbook container. Run the commands below inside it.
- Pick the checkpoint.
Qwen/Qwen3.6-35B-A3BandQwen/Qwen3.5-35B-A3Btake the same recipe. Both are mixture-of-experts models with 256 routed experts, eight selected for each token. - Write the config. Create
qwen3.6-sft.yamlfor supervised fine-tuning of all weights. Among its keys:YAML
model_name_or_path: Qwen/Qwen3.6-35B-A3B moe_balancing: bias_update_transient expert_parallel_size: 8 attn_implementation: flash_attention_2 padding_free: false - Launch eight processes. The recipe starts with eight NVIDIA B300 GPUs. With expert parallelism of size 8 (EP8), each GPU holds 32 experts.
Bash
halo launch sft qwen3.6-sft.yaml -n 8 - Train Qwen3.5 instead. Replace the model name, revision and output directory with the values in the cookbook's checkpoint table.
Common mistakes
- Flash Attention 4. Its backward pass emits NaN gradients on this architecture, so the loader demotes it to SDPA (scaled dot-product attention). Keep Flash Attention 2.
- Another balancing mode.
aux_lossis refused and plainbias_updateraises.bias_update_transientbalances routing during training, but every exported checkpoint serves without the bias. - Padding-free batches. Keep
padding_free: false, or the run crashes.
With Halo
Halo does not support context parallelism for this family, and tensor parallelism is limited to two GPUs. The Qwen3.5 and Qwen3.6 cookbook has the full config, the tensor and expert-tensor layouts, a LoRA adapter, and reinforcement learning with GRPO.
