Qwen3 Family

Three variants: dense (Qwen3ForCausalLM), MoE (Qwen3MoeForCausalLM), and the text branch of the VL model.

Per-axis support: the Qwen3 rows of the compatibility matrix.

Qwen3.5 / Qwen3.6 share the name prefix but use a different Qwen3_5* class hierarchy — see the separate page.

Qwen3 dense

Standard transformer with GQA, RoPE, SwiGLU MLP. Stock Qwen3ForCausalLM, no patching.

  • FSDP2 — default under torchrun; under accelerate launch the --config_file decides.
  • TP — native HF tp_plan="auto" shards attention and MLP.
  • CP — Qwen3Attention uses the same Qwen3MoeUlyssesAttention wrapper as Qwen3 MoE and Qwen3-VL.
  • PP — registered for the dense family; the tied 0.6B, 1.7B and 4B checkpoints are refused by the tie gate, 8B and up split. Two-node recipe: examples/sft/qwen3/qwen3-8b-multinode-pp2.yaml (Pipeline Parallelism).
  • LoRA — under FSDP/DP and CP; rejected at construction under TP (PEFT).

Configs

ConfigMethodBase model
examples/sft/qwen3/qwen3-4b-ultrachat.yamlSFT (full), 4K seqQwen/Qwen3-4B-Instruct-2507
examples/sft/qwen3/qwen3-4b-ultrachat-lora.yamlSFT (LoRA)Qwen/Qwen3-4B-Instruct-2507
examples/sft/qwen3/qwen3-4b-ultrachat-qlora.yamlSFT (QLoRA)Qwen/Qwen3-4B-Instruct-2507
examples/sft/qwen3/qwen3-8b-multinode-pp2.yamlSFT (full), pp2 across two 8-GPU nodesQwen/Qwen3-8B
examples/embedding/qwen3/embedding-qwen3-4b-nq.yamlEmbeddingQwen/Qwen3-Embedding-4B
examples/grpo/online/qwen3/online-grpo-qwen3-{4b,8b}-smoke.yamlOnline GRPO smokeQwen/Qwen3-4B-Instruct-2507, Qwen/Qwen3-8B
examples/grpo/online/qwen3/online-grpo-qwen3-8b-rlrr-math.yamlOnline GRPO (RLVR + RLRR)Qwen/Qwen3-8B
examples/grpo/{online/rlvr-online-grpo,environmental/environmental-grpo}-template.yamlRLVR / env GRPO templatesQwen/Qwen3-4B-Instruct-2507

The full fine-tune trains without gradient checkpointing. Each micro-batch, two packed 4,096-token rows, holds about 35 GB of activations, which fit under FSDP2 on several GPUs but not beside the whole model, gradient and optimizer state of a single 80 GB GPU, so a one-GPU Hopper launch passes --gradient_checkpointing=true.

The per-method canonical examples (DPO, SMPO, KTO, reward, classification, distillation, GRPO task configs) are built on Qwen3.5/3.6 — see Qwen3.5 / Qwen3.6.

Qwen3 MoE

EPQwen3MoELayer (halo/distributed/expert_parallel/families/qwen3.py) replaces Qwen3MoeSparseMoeBlock.

  • Routing: the gate returns (logits, weights, experts); softmax + top-k.
  • Activation: act(gate(x)) * up(x) → down, where act is read off Qwen3MoeExperts.act_fn (the resolved hidden_act, SiLU on every released checkpoint) rather than assumed.
  • Storage: pre-fused gate_up_proj [E, 2M, H] + down_proj [E, H, M], split at EP layer construction into separate gate_proj / up_proj / down_proj for sharding.
  • Compute: Grouped GEMM on SM90+, falling back to a per-expert loop with index_add_.
  • RL weight sync: gather_expert_state_dict emits the per-expert hub layout, which both pinned engines read — vLLM 0.32.0's RoutedExperts and SGLang 0.5.22's qwen3_moe loader (make_expert_params_mapping) alike, so either rollout_backend takes the sync (Rollout Servers).

Router balancing — the EP layer re-derives selection from the router's own logits, applying the DeepSeek-V3 bias to the selection scores while the gate weights stay on the unbiased softmax (honoring the family's norm_topk_prob).

The architecture has no bias slot (the gate is a bare weight), so the bias is trainer-only: moe_balancing: bias_update raises and the explicit bias_update_transient is the opt-in, with exported checkpoints serving without the trained bias. auto stays on aux_loss, since Qwen3MoeForCausalLM.forward declares output_router_logits (Callbacks).

CP — Qwen3MoeAttention → Qwen3MoeUlyssesAttention (halo/distributed/context_parallel/families.py), RoPE before all-to-all and native GQA.

TP and ETP — selective attention-only TP (q/k/v_proj colwise, o_proj rowwise); embeddings and lm_head stay replicated, expert layers belong to EP. ETP shards gate_proj / up_proj / down_proj along the intermediate dim; because Qwen3 already stores gate and up separately, the same compute path runs under both ETP and non-ETP with a smaller M.

PP — registered, at any split offset. With no exportable bias slot and the aux-loss path severed by the split, a PP run sets moe_balancing: none (Pipeline Parallelism).

LoRA on the experts works under EP through native grouped adapters (_init_expert_lora). It is rejected under TP and EP+TP, and at expert_tp_size > 1 (the replicated adapter half would take partial gradients) — attention LoRA still runs there.

Tree packing — dense and MoE Qwen3 take tree_packing: true (Tree Packing); a run pinned to flash_attention_2 takes the gather form, unset tree_full_attention resolving to it.

Configs — no Qwen3 MoE recipe ships; the EP benchmarks and GPU suites run Qwen/Qwen3-30B-A3B-Instruct-2507.

Qwen3-VL

Qwen3VLTextAttention reuses Qwen3MoeUlyssesAttention for CP.

Under TP a dense Qwen3-VL goes through HF-native tp_plan="auto", and the architecture ships no base_model_tp_plan — the plan is empty, nothing shards, and the load raises rather than running tp_size full replicas at 1/tp_size throughput. Use plain FSDP2 for dense VLM runs; CP covers only text-only rows, since the CP wrapper refuses pixel_values. The MoE variants get no EP: no EP layer claims Qwen3VLMoeTextSparseMoeBlock, and Qwen3VLMoeTextAttention is in neither the CP nor the selective-TP registry.

No dedicated SFT configs ship; tests/gpu/trainers/sft/test_sft_vlm.py loads Qwen/Qwen3-VL-2B-Instruct on text rows, and test_dpo_vlm.py, test_smpo_vlm.py and test_self_distillation_vlm.py train it on images. See Vision-language models.

Source on GitHubUpdated 18 November 2026