Qwen3 MoE Cookbook
Qwen3-30B-A3B-Instruct-2507 has 128 routed experts and picks eight per token. Qwen3 MoE has the broadest parallelism coverage in Halo, which makes it the reference MoE family.
Support
Modes: the Qwen3 MoE row of the Supported Matrix; online RL: Rollout engines.
- Checkpoint:
Qwen/Qwen3-30B-A3B-Instruct-2507, pinned below. - GPUs: four at EP4, 32 experts per GPU.
Recipes
Run these in the training container.
Full fine-tune
No Qwen3 MoE SFT recipe ships, so save this as qwen3-moe-sft.yaml in the repo root:
YAML
model_name_or_path: Qwen/Qwen3-30B-A3B-Instruct-2507
model_revision: 0d7cf23991f47feeb3a57ecb4c9cee8ea4a17bfe
moe_balancing: aux_loss
dataset:
- HuggingFaceH4/ultrachat_200k@train_sft
conversation_field: messages
test_size: 0.01
assistant_message_template: "<|im_start|>assistant\n"
pad_token: <|endoftext|>
eos_token: <|im_end|>
expert_parallel_size: 4
fp32_router: true
packing: true
max_length: 8192
per_device_train_batch_size: 1
per_device_eval_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 5.0e-06
lr_scheduler_type: cosine
warmup_steps: 32
num_train_epochs: 1.0
save_steps: 1000
save_total_limit: 1
save_only_model: true
eval_strategy: steps
eval_steps: 300
logging_steps: 1
report_to: wandb
remove_unused_columns: false
output_dir: /data/checkpoints/qwen3-30b-a3b-sftBash
halo launch sft qwen3-moe-sft.yaml -n 4Other layouts
Each of these runs on the same four GPUs. Add one to the launch command:
- EP4 + CP2, for long sequences:
--context_parallel_size=2 --packing=false - EP4 + TP2, when attention memory is the limit:
--tensor_parallel_size=2 - Pure ETP4, to shard every expert instead of placing whole experts:
--expert_parallel_size=1 --expert_tensor_parallel_size=4
On eight GPUs, launch with -n 8 --expert_parallel_size=8. EP4 on eight GPUs is rejected at startup.
LoRA
Bash
halo launch sft qwen3-moe-sft.yaml -n 4 \
--use_peft=true --learning_rate=1e-4 \
--lora_target_modules=q_proj,k_proj,v_proj,o_proj \
--output_dir=/data/checkpoints/qwen3-30b-a3b-loraTo adapt the experts as well, add gate_proj,up_proj,down_proj and --merge_expert_lora_on_save=true.
Under EP they train as grouped expert adapters, which no tool merges afterwards.
GRPO
Read Before a GRPO run first. This setup serves the SFT checkpoint from vLLM on GPUs 0–1 and trains on GPUs 2–3.
On the host:
Bash
VLLM_MODEL=/data/checkpoints/qwen3-30b-a3b-sft VLLM_CUDA_DEVICES=0,1 VLLM_TP=2 VLLM_ENABLE_R3=1 \
docker compose -f docker-compose.vllm.yml up -d vllm-serverIn the training container, copy the template, set dataset, environment_type and rewards, then
launch:
Bash
cp examples/grpo/environmental/environmental-grpo-template.yaml qwen3-moe-grpo.yaml
CUDA_VISIBLE_DEVICES=2,3 halo launch grpo/environmental qwen3-moe-grpo.yaml -n 2 \
--model_name_or_path=/data/checkpoints/qwen3-30b-a3b-sft \
--routing_replay=rollout --beta=0.0 \
--output_dir=/data/checkpoints/qwen3-30b-a3b-grpoAdd --expert_parallel_size=2 to shard the experts across the two trainer GPUs. On SGLang, add
--rollout_backend=sglang --rollout_server_url=http://localhost:30000 and start the server with
SGLANG_ENABLE_R3=1 SGLANG_TOOL_PARSER=qwen25.
Settings that matter
- Attention. Leave
attn_implementationunset. Halo picks FA4 on Blackwell and FA3 (or FA2) on Hopper. - Router balancing.
aux_lossuses the checkpoint'srouter_aux_loss_coef(0.001). The router has no bias slot, sobias_updateraises.bias_update_transientbalances training only, and exported checkpoints serve without the bias. - Tool parser. Qwen3 tool calls parse with vLLM's default
hermesparser. On SGLang, setSGLANG_TOOL_PARSER=qwen25: the defaultautocannot read the 2507 chat template and turns the parser off with one log line, which the trainer refuses at startup on a tool-using environment (Tool parsers on SGLang ↗).
Limits
- Pipeline parallelism passes tiny-model GPU tests and a 20-step Qwen3-30B-A3B run at
pp2×ep4, and needsmoe_balancing: none, since the stage split drops the aux loss and the router has no bias slot. - The general limits apply: LoRA is refused under TP and EP+TP, and expert adapters are refused once
expert_tensor_parallel_size > 1.
Export and serve
The gathered save writes the hub's per-expert layout, which transformers, vLLM and SGLang read
directly. Smoke-test it with AutoModelForCausalLM (snippet).
Both pinned engines have served the hub checkpoint and taken its weight sync, but neither has served a
trained Qwen3-30B-A3B export, so check the first one you serve. Serve it from the
host with SGLang on port 30000:
Bash
SGLANG_MODEL="$HALO_SCRATCH/checkpoints/qwen3-30b-a3b-sft" \
docker compose -f docker-compose.sglang.yml up -dReference
- Qwen3 model notes ↗: the EP layer, CP and TP, balancing
- Async GRPO with Environments
- Model card
Source on GitHubUpdated 18 November 2026