# How to fine-tune Qwen3

To fine-tune Qwen3 with Halo, train Qwen/Qwen3-30B-A3B-Instruct-2507 on four GPUs with expert parallelism of size 4 (EP4), which places 32 of its 128 routed experts on each GPU. A LoRA adapter is a block added to the same config.

## Steps

1. **Start the training container.** Set up the host as [installation](/halo/docs/installation) describes, then start the [cookbook container](/halo/docs/cookbooks#start-the-training-container). Run the commands below inside it.
2. **Pick the checkpoint.** The cookbook trains `Qwen/Qwen3-30B-A3B-Instruct-2507`, a [mixture-of-experts](/glossary/mixture-of-experts) model with 128 routed experts, eight selected for each token.
3. **Write the config.** Create `qwen3-moe-sft.yaml` for supervised fine-tuning of all weights. Among its keys:
   ```yaml
   model_name_or_path: Qwen/Qwen3-30B-A3B-Instruct-2507
   model_init_kwargs:
     output_router_logits: true
     router_aux_loss_coef: 0.001
   moe_balancing: aux_loss
   expert_parallel_size: 4
   save_sharded_ep: false
   ```
4. **Launch four processes.** The recipe starts with four NVIDIA B300 GPUs. With [expert parallelism](/glossary/expert-parallelism) of size 4 (EP4), each GPU holds 32 experts.
   ```bash
   halo launch sft qwen3-moe-sft.yaml -n 4
   ```
5. **Load the result.** Because `save_sharded_ep` is false, Halo gathers the expert weights when it saves. The checkpoint loads with `from_pretrained`.

## Common mistakes

- **EP4 on eight GPUs.** EP4 is one dispatch group on four GPUs. The same config on eight is rejected at config time.
- **Setting the attention backend.** Leave `attn_implementation` unset. Halo picks the [Flash Attention](/glossary/flash-attention) version for the GPU.
- **Mixing modes.** Do not enable tensor and expert-tensor parallelism together, and do not combine [LoRA](/glossary/lora) with tensor parallelism.

## With Halo

Halo can distribute Qwen3's experts with expert parallelism, shard each expert with expert-tensor parallelism, and shard attention with tensor parallelism. The [Qwen3 cookbook](/halo/docs/cookbooks/halo-qwen3-moe-cookbook) has the full config, those layouts on the same four GPUs, context parallelism for longer sequences, a LoRA adapter, and [GRPO](/glossary/grpo).

