Qwen3 Family
Three variants: dense (Qwen3ForCausalLM), MoE (Qwen3MoeForCausalLM), and the text branch of the VL model.
Per-axis support: the Qwen3 rows of the compatibility matrix.
Qwen3.5 / Qwen3.6 share the name prefix but use a different Qwen3_5* class hierarchy — see the separate page.
Qwen3 dense
Standard transformer with GQA, RoPE, SwiGLU MLP. Stock Qwen3ForCausalLM, no patching.
- FSDP2 — default under torchrun; under
accelerate launchthe--config_filedecides. - TP — native HF
tp_plan="auto"shards attention and MLP. - CP —
Qwen3Attentionuses the sameQwen3MoeUlyssesAttentionwrapper as Qwen3 MoE and Qwen3-VL. - PP — registered for the dense family; the tied 0.6B, 1.7B and 4B checkpoints are refused by the tie gate, 8B and up split. Two-node recipe:
examples/sft/qwen3/qwen3-8b-multinode-pp2.yaml(Pipeline Parallelism). - LoRA — under FSDP/DP and CP; rejected at construction under TP (PEFT).
Configs
The full fine-tune trains without gradient checkpointing. Each micro-batch, two packed 4,096-token rows, holds about 35 GB of activations, which fit under FSDP2 on several GPUs but not beside the whole model, gradient and optimizer state of a single 80 GB GPU, so a one-GPU Hopper launch passes --gradient_checkpointing=true.
The per-method canonical examples (DPO, SMPO, KTO, reward, classification, distillation, GRPO task configs) are built on Qwen3.5/3.6 — see Qwen3.5 / Qwen3.6.
Qwen3 MoE
EPQwen3MoELayer (halo/distributed/expert_parallel/families/qwen3.py) replaces Qwen3MoeSparseMoeBlock.
- Routing: the gate returns
(logits, weights, experts); softmax + top-k. - Activation:
act(gate(x)) * up(x)→down, whereactis read offQwen3MoeExperts.act_fn(the resolvedhidden_act, SiLU on every released checkpoint) rather than assumed. - Storage: pre-fused
gate_up_proj [E, 2M, H]+down_proj [E, H, M], split at EP layer construction into separategate_proj/up_proj/down_projfor sharding. - Compute: Grouped GEMM on SM90+, falling back to a per-expert loop with
index_add_. - RL weight sync:
gather_expert_state_dictemits the per-expert hub layout, which both pinned engines read — vLLM 0.32.0'sRoutedExpertsand SGLang 0.5.22'sqwen3_moeloader (make_expert_params_mapping) alike, so eitherrollout_backendtakes the sync (Rollout Servers).
Router balancing — the EP layer re-derives selection from the router's own logits, applying the DeepSeek-V3 bias to the selection scores while the gate weights stay on the unbiased softmax (honoring the family's norm_topk_prob).
The architecture has no bias slot (the gate is a bare weight), so the bias is trainer-only: moe_balancing: bias_update raises and the explicit bias_update_transient is the opt-in, with exported checkpoints serving without the trained bias. auto stays on aux_loss, since Qwen3MoeForCausalLM.forward declares output_router_logits (Callbacks).
CP — Qwen3MoeAttention → Qwen3MoeUlyssesAttention (halo/distributed/context_parallel/families.py), RoPE before all-to-all and native GQA.
TP and ETP — selective attention-only TP (q/k/v_proj colwise, o_proj rowwise); embeddings and lm_head stay replicated, expert layers belong to EP. ETP shards gate_proj / up_proj / down_proj along the intermediate dim; because Qwen3 already stores gate and up separately, the same compute path runs under both ETP and non-ETP with a smaller M.
PP — registered, at any split offset. With no exportable bias slot and the aux-loss path severed by the split, a PP run sets moe_balancing: none (Pipeline Parallelism).
LoRA on the experts works under EP through native grouped adapters (_init_expert_lora). It is rejected under TP and EP+TP, and at expert_tp_size > 1 (the replicated adapter half would take partial gradients) — attention LoRA still runs there.
Tree packing — dense and MoE Qwen3 take tree_packing: true (Tree Packing); a run pinned to flash_attention_2 takes the gather form, unset tree_full_attention resolving to it.
Configs — no Qwen3 MoE recipe ships; the EP benchmarks and GPU suites run Qwen/Qwen3-30B-A3B-Instruct-2507.
Qwen3-VL
Qwen3VLTextAttention reuses Qwen3MoeUlyssesAttention for CP.
Under TP a dense Qwen3-VL goes through HF-native tp_plan="auto", and the architecture ships no base_model_tp_plan — the plan is empty, nothing shards, and the load raises rather than running tp_size full replicas at 1/tp_size throughput. Use plain FSDP2 for dense VLM runs; CP covers only text-only rows, since the CP wrapper refuses pixel_values. The MoE variants get no EP: no EP layer claims Qwen3VLMoeTextSparseMoeBlock, and Qwen3VLMoeTextAttention is in neither the CP nor the selective-TP registry.
No dedicated SFT configs ship; tests/gpu/trainers/sft/test_sft_vlm.py loads Qwen/Qwen3-VL-2B-Instruct on text rows, and test_dpo_vlm.py, test_smpo_vlm.py and test_self_distillation_vlm.py train it on images. See Vision-language models.
Source on GitHubUpdated 18 November 2026