To fine-tune Qwen3 with Halo, train Qwen/Qwen3-30B-A3B-Instruct-2507 on four GPUs with expert parallelism of size 4 (EP4), which places 32 of its 128 routed experts on each GPU. A LoRA adapter is a block added to the same config.
Steps
- Start the training container. Set up the host as installation describes, then start the cookbook container. Run the commands below inside it.
- Pick the checkpoint. The cookbook trains
Qwen/Qwen3-30B-A3B-Instruct-2507, a mixture-of-experts model with 128 routed experts, eight selected for each token. - Write the config. Create
qwen3-moe-sft.yamlfor supervised fine-tuning of all weights. Among its keys:YAML
model_name_or_path: Qwen/Qwen3-30B-A3B-Instruct-2507 model_init_kwargs: output_router_logits: true router_aux_loss_coef: 0.001 moe_balancing: aux_loss expert_parallel_size: 4 save_sharded_ep: false - Launch four processes. The recipe starts with four NVIDIA B300 GPUs. With expert parallelism of size 4 (EP4), each GPU holds 32 experts.
Bash
halo launch sft qwen3-moe-sft.yaml -n 4 - Load the result. Because
save_sharded_epis false, Halo gathers the expert weights when it saves. The checkpoint loads withfrom_pretrained.
Common mistakes
- EP4 on eight GPUs. EP4 is one dispatch group on four GPUs. The same config on eight is rejected at config time.
- Setting the attention backend. Leave
attn_implementationunset. Halo picks the Flash Attention version for the GPU. - Mixing modes. Do not enable tensor and expert-tensor parallelism together, and do not combine LoRA with tensor parallelism.
With Halo
Halo can distribute Qwen3's experts with expert parallelism, shard each expert with expert-tensor parallelism, and shard attention with tensor parallelism. The Qwen3 cookbook has the full config, those layouts on the same four GPUs, context parallelism for longer sequences, a LoRA adapter, and GRPO.
