How to fine-tune Qwen3

To fine-tune Qwen3 with Halo, train Qwen/Qwen3-30B-A3B-Instruct-2507 on four GPUs with expert parallelism of size 4 (EP4), which places 32 of its 128 routed experts on each GPU. A LoRA adapter is a block added to the same config.

Steps

  1. Start the training container. Set up the host as installation describes, then start the cookbook container. Run the commands below inside it.
  2. Pick the checkpoint. The cookbook trains Qwen/Qwen3-30B-A3B-Instruct-2507, a mixture-of-experts model with 128 routed experts, eight selected for each token.
  3. Write the config. Create qwen3-moe-sft.yaml for supervised fine-tuning of all weights. Among its keys:

    YAML

    model_name_or_path: Qwen/Qwen3-30B-A3B-Instruct-2507
    model_init_kwargs:
      output_router_logits: true
      router_aux_loss_coef: 0.001
    moe_balancing: aux_loss
    expert_parallel_size: 4
    save_sharded_ep: false
  4. Launch four processes. The recipe starts with four NVIDIA B300 GPUs. With expert parallelism of size 4 (EP4), each GPU holds 32 experts.

    Bash

    halo launch sft qwen3-moe-sft.yaml -n 4
  5. Load the result. Because save_sharded_ep is false, Halo gathers the expert weights when it saves. The checkpoint loads with from_pretrained.

Common mistakes

  • EP4 on eight GPUs. EP4 is one dispatch group on four GPUs. The same config on eight is rejected at config time.
  • Setting the attention backend. Leave attn_implementation unset. Halo picks the Flash Attention version for the GPU.
  • Mixing modes. Do not enable tensor and expert-tensor parallelism together, and do not combine LoRA with tensor parallelism.

With Halo

Halo can distribute Qwen3's experts with expert parallelism, shard each expert with expert-tensor parallelism, and shard attention with tensor parallelism. The Qwen3 cookbook has the full config, those layouts on the same four GPUs, context parallelism for longer sequences, a LoRA adapter, and GRPO.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL