How to fine-tune Qwen3.5 and Qwen3.6

To fine-tune Qwen3.5 and Qwen3.6 with Halo, train Qwen/Qwen3.6-35B-A3B or Qwen/Qwen3.5-35B-A3B on eight GPUs with expert parallelism of size 8 (EP8), Flash Attention 2 and transient bias-update balancing.

Steps

  1. Start the training container. Set up the host as installation describes, then start the cookbook container. Run the commands below inside it.
  2. Pick the checkpoint. Qwen/Qwen3.6-35B-A3B and Qwen/Qwen3.5-35B-A3B take the same recipe. Both are mixture-of-experts models with 256 routed experts, eight selected for each token.
  3. Write the config. Create qwen3.6-sft.yaml for supervised fine-tuning of all weights. Among its keys:

    YAML

    model_name_or_path: Qwen/Qwen3.6-35B-A3B
    moe_balancing: bias_update_transient
    expert_parallel_size: 8
    attn_implementation: flash_attention_2
    padding_free: false
  4. Launch eight processes. The recipe starts with eight NVIDIA B300 GPUs. With expert parallelism of size 8 (EP8), each GPU holds 32 experts.

    Bash

    halo launch sft qwen3.6-sft.yaml -n 8
  5. Train Qwen3.5 instead. Replace the model name, revision and output directory with the values in the cookbook's checkpoint table.

Common mistakes

  • Flash Attention 4. Its backward pass emits NaN gradients on this architecture, so the loader demotes it to SDPA (scaled dot-product attention). Keep Flash Attention 2.
  • Another balancing mode. aux_loss is refused and plain bias_update raises. bias_update_transient balances routing during training, but every exported checkpoint serves without the bias.
  • Padding-free batches. Keep padding_free: false, or the run crashes.

With Halo

Halo does not support context parallelism for this family, and tensor parallelism is limited to two GPUs. The Qwen3.5 and Qwen3.6 cookbook has the full config, the tensor and expert-tensor layouts, a LoRA adapter, and reinforcement learning with GRPO.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL