Qwen3.5 / Qwen3.6 MoE Cookbook
Qwen3.5-35B-A3B and Qwen3.6-35B-A3B share one model family. Each has 256 routed experts plus a shared expert and picks eight per token. Of its 40 layers, 10 use full attention and 30 use GatedDeltaNet linear attention. The checkpoints are multimodal, and the recipes train their language model on text.
Support
Modes: the Qwen3.5 / Qwen3.6 MoE row of the Supported Matrix; online RL: Rollout engines.
- Checkpoints:
Qwen/Qwen3.6-35B-A3Bat revision995ad96eacd98c81ed38be0c5b274b04031597b0andQwen/Qwen3.5-35B-A3Bat59d61f3ce65a6d9863b86d2e96597125219dc754. The 122B-A10B trains fromexamples/sft/qwen3_5/qwen3.5-122b-a10b-ep.yamlat EP8 on one node. Across two nodes, add--expert_parallel_size=16 --ep_scope=global --max_length=8192. - GPUs: eight at EP8, 32 experts per GPU, at 33,000 tokens: about 17,200 tok/s per GPU at 155.5 GiB peak on a warm Triton cache.
Recipes
Run these in the training container.
Full fine-tune
The shipped recipe trains Qwen3.5. To train Qwen3.6, override the model and its revision:
Bash
halo launch sft examples/sft/qwen3_5/qwen3.5-35b-a3b-ultrachat-ep.yaml -n 8 \
--model_name_or_path=Qwen/Qwen3.6-35B-A3B \
--model_revision=995ad96eacd98c81ed38be0c5b274b04031597b0 \
--output_dir=/data/checkpoints/qwen3.6-35b-a3b-sftDrop the first two overrides to train Qwen3.5.
Other layouts
Add one of these to the full fine-tune command:
- EP8 + TP2, when attention memory is the limit:
--tensor_parallel_size=2 - Pure ETP8, to shard every expert instead of placing whole experts:
--expert_parallel_size=1 --expert_tensor_parallel_size=8 - A 4-way expert split:
--expert_parallel_size=4 --expert_tensor_parallel_size=2
LoRA
Bash
halo launch sft examples/sft/qwen3_5/qwen3.5-35b-a3b-ultrachat-ep.yaml -n 8 \
--model_name_or_path=Qwen/Qwen3.6-35B-A3B \
--model_revision=995ad96eacd98c81ed38be0c5b274b04031597b0 \
--use_peft=true --learning_rate=1e-4 \
--lora_target_modules=q_proj,k_proj,v_proj,o_proj \
--output_dir=/data/checkpoints/qwen3.6-35b-a3b-loraMTP head
Train the mtp.* head the release ships, so a server can draft with it: over a frozen trunk, or inside a fine-tune
with qwen3.6-35b-a3b-mtp-joint.yaml. A fit repairs a stale head and holds a healthy one. The recipe fits chat rows
under the think render, not a server's own samples. Its 150 steps served at 0.834 acceptance on vLLM against the
release head's 0.831, on 64 first-turn prompts, and that head alone reads 0.83 to 0.84 across launches, so no served
gain is measured. What the joint recipe does over a fine-tune is not measured.
Bash
halo launch sft examples/sft/qwen3_5/qwen3.6-35b-a3b-mtp-head-only.yaml -n 8 \
--output_dir=/data/checkpoints/qwen3.6-35b-a3b-mtp-headThe fit scores held-out rows before its first step and at its last, and exits non-zero if acceptance fell. Its last
checkpoint then holds the rejected head, so serve the one the fit started from. mtp_train_scope: detached keeps a
head current during a fine-tune without changing the trunk's objective. Serve with
VLLM_SPECULATIVE_CONFIG='{"method":"mtp","num_speculative_tokens":1}', or on SGLang with
its one-draft flags ↗.
GRPO
The shipped recipes train Qwen3.6, most of them on code contests. Read Before a GRPO run first. This one is a full fine-tune at EP4 with two vLLM servers on GPUs 4–7.
On the host, start the servers on the SFT checkpoint:
Bash
export VLLM_MODEL=/data/checkpoints/qwen3.6-35b-a3b-sft
export VLLM_CHAT_TEMPLATE=/jinja-templates/qwen3/qwen3.6-reasoning-effort.jinja
export VLLM_TOOL_PARSER=qwen3_xml VLLM_REASONING_PARSER=qwen3
VLLM_CUDA_DEVICES=4,5 VLLM_TP=2 VLLM_PORT=8000 \
docker compose -p qwen36-rollout-0 -f docker-compose.vllm.yml up -d vllm-server
VLLM_CUDA_DEVICES=6,7 VLLM_TP=2 VLLM_PORT=8001 \
docker compose -p qwen36-rollout-1 -f docker-compose.vllm.yml up -d vllm-serverIn the training container, copy the recipe, set its dataset, and launch on GPUs 0–3:
Bash
cp examples/grpo/environmental/qwen3_5/vllm/qwen3.6-35b-a3b-code-contests-full-ep4.yaml qwen3.6-grpo.yaml
CUDA_VISIBLE_DEVICES=0,1,2,3 DIST_NCCL_TIMEOUT_MINUTES=60 \
halo launch grpo/environmental qwen3.6-grpo.yaml -n 4 \
--model_name_or_path=/data/checkpoints/qwen3.6-35b-a3b-sft \
--output_dir=/data/checkpoints/qwen3.6-35b-a3b-grpoThe other configs in examples/grpo/environmental/qwen3_5/vllm/ change the environment, the adapter or
the EP size. The full fine-tune ep1 code-contests recipe is a curriculum: run -stage1-codeforces,
-stage2-hard and -stage3-extra-hard in order, each from the previous stage's checkpoint.
The sglang/ configs (ep1) run on SGLang. The full fine-tune expects two TP=1 servers on GPUs 6 and 7
and a six-GPU trainer:
Bash
export SGLANG_MODEL="$HALO_SCRATCH/checkpoints/qwen3.6-35b-a3b-sft"
export SGLANG_CHAT_TEMPLATE=/jinja-templates/qwen3/qwen3.6-reasoning-effort.jinja SGLANG_REASONING_PARSER=qwen3
SGLANG_CUDA_DEVICES=6 SGLANG_PORT=30000 \
docker compose -p qwen36-sglang-0 -f docker-compose.sglang.yml up -d sglang-server
SGLANG_CUDA_DEVICES=7 SGLANG_PORT=30001 \
docker compose -p qwen36-sglang-1 -f docker-compose.sglang.yml up -d sglang-serverBash
cp examples/grpo/environmental/qwen3_5/sglang/qwen3.6-35b-a3b-code-contests-full-ep1.yaml qwen3.6-grpo-sglang.yaml
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5 DIST_NCCL_TIMEOUT_MINUTES=60 \
halo launch grpo/environmental qwen3.6-grpo-sglang.yaml -n 6 \
--model_name_or_path=/data/checkpoints/qwen3.6-35b-a3b-sft \
--output_dir=/data/checkpoints/qwen3.6-35b-a3b-grpo-sglangSettings that matter
- Attention. The SFT recipe pins
flash_attention_2, and the GRPO recipes pinsdpa. Unpinned, Blackwell runs FA4, and FA2 on packed SFT rows (Attention backends). - Packing or padding-free. Both pack documents without cross-document attention on this family. An unset
attn_implementationloads FA2 for either, and the shipped recipes usepacking: truewith fixed-length rows. Rows that share a long prefix can pack as trees on FA2 (tree_packing: true; Tree Packing ↗). - Router balancing. The SFT recipes set
moe_balancing: bias_update_transient, and the GRPO recipes setnone. The router has no bias slot, so plainbias_updateraises. In SFT,aux_loss(whatautopicks) balances through the loss and exports by construction. The transient bias balances training only: exported checkpoints serve without it, so near-tied expert picks can differ between trainer and server. - Chat template. The SFT recipes render
qwen3-multiturn.jinja, which writes assistant turns with no think block.eval_lossreads about 3.3 at step 0 on Qwen3.6-35B-A3B and about 0.83 after two updates. Train and serve under the same render. - Long context. The SFT recipe sets
fp32_non_ep_params: truefor stable training at 33,000 tokens. - Tool parser and reasoning cap. vLLM needs
VLLM_TOOL_PARSER=qwen3_xml. The defaulthermesleaves the XML tool calls as text, and the run stops by its second round. The code-contests recipes cap each turn's reasoning at the answer room's edge, which needsVLLM_REASONING_PARSER=qwen3, or every request fails with a 400 (Reasoning caps and budgets). - SGLang rollouts.
rollout_max_thinking_tokensandcarry_reasoningare refused at startup, and onlyrollout_max_tokensbounds a turn.
Limits
- No CP: the linear-attention layers can't split the sequence across ranks.
- Pipeline parallelism takes the text tower only (
text_only_model: true), withmoe_balancing: none; packed rows keep their documents apart, one packed row per microbatch (Qwen3.5 / Qwen3.6 model notes ↗). - TP stops at 2 because the checkpoints have two KV heads. TP shards only the 10 full-attention layers,
so per-GPU memory drops much less than
1/tp_size. - Online RL refuses
text_only_model: true. - Saves drop the hub checkpoint's
mtp.*multi-token-prediction head unless the run trains it.reattach-vision-towerrestores the base's untrained head for a text-only export.
Export and serve
Gathered saves keep the language-model experts fused, like the hub checkpoints. Transformers and vLLM
0.32.0 read that layout directly. halo run unfuse-moe-experts rewrites it per expert for a loader that
needs that layout.
Smoke-test with AutoModelForImageTextToText and AutoTokenizer
(snippet). Serve from the host
with vLLM on port 8000:
Bash
VLLM_MODEL=/data/checkpoints/qwen3.6-35b-a3b-sft VLLM_CUDA_DEVICES=0,1,2,3 VLLM_TP=4 \
VLLM_TOOL_PARSER=qwen3_xml \
docker compose -f docker-compose.vllm.yml up -d vllm-servertext_only_model: true drops the vision tower and processor files from the export. To rebuild the
multimodal checkpoint:
Bash
halo run reattach-vision-tower --input_dir /data/checkpoints/<text-only-export> \
--model_id Qwen/Qwen3.6-35B-A3B --output_dir /data/checkpoints/<text-only-export>-vlReference
- Qwen3.5 / Qwen3.6 model notes ↗: why CP is blocked, the text-only export, chat templates
- Async GRPO with Environments
- Model cards: Qwen3.5-35B-A3B, Qwen3.6-35B-A3B
Source on GitHubUpdated 18 November 2026