GLM-4 MoE Lite
Glm4MoeLiteForCausalLM — MoE with sigmoid routing (group-limited top-k), shared experts, and LoRA-style attention compression (q_a_proj / q_b_proj + kv_a_proj_with_mqa / kv_b_proj).
Per-axis support: Compatibility matrix. Pipeline parallelism splits it at any layer offset (Pipeline Parallelism).
Its MLA-style attention (192 + 64 = 256-wide query/key, 256-wide value) runs FA2 on Blackwell in every run, Context Parallelism and full_determinism included: the resolver keeps latent attention at that width on FA2 as a precaution (Flash Attention → Model-specific handling). FA3 on Hopper.
EP layer
EPGlm4MoELayer (halo/distributed/expert_parallel/families/glm4.py) replaces Glm4MoeLiteMoE. Laguna shares the shape and subclasses this EP layer — see Laguna.
- Routing: sigmoid + group mask + top-k.
topk_groupselects eligible expert groups, thentop_k(=4) experts are chosen across them.route_tokens_to_experts()runs this before DeepEP dispatch. - Routed experts:
Glm4MoeLiteExperts, fusedgate_up_projanddown_proj3D tensors. - Shared experts:
Glm4MoeLiteMLP, replicated per rank, added after DeepEP combine. - Activation: SwiGLU on the split fused gate_up; Grouped GEMM on SM90+.
Fused SwiGLU
The _fused_glu_mul latch replaces the activation and the multiply with one Triton kernel on every base compute path, the ETP-sharded one included: the packed [gate | up] path runs it through GluCombine.packed, split halves through _glu_combine. It is a roster-wide seam, not a GLM-4 one: _resolve_activation latches whatever resolve_fused_glu_mul (halo/kernels/fused_glu.py) returns for the resolved activation — GluCombine.silu() for a SiLU gate (GLM-4 Lite, Laguna, Qwen3, Qwen3.5/3.6, Bailing, LFM-2, Mistral4, Cohere2 MoE, Inkling, Zaya, and Step-3.7's unclamped layers), GluCombine.gelu_tanh() for a tanh-GELU one (Gemma 4), None otherwise.
A family opts in by having a gate the kernels implement, not by setting a flag. A family whose combine is a variant rebinds the same seam: DeepSeek-V4 (clamp before the activation) and Step-3.7 (its two clamped layers clamp after it) arm their kernels off the same SiLU probe.
GLM-5 Next binds its pre-activation clamp unconditionally, since its experts hardcode that gate. GPT-OSS owns its interleaved-bias compute paths and never reaches the seam.
Each kernel hardcodes its activation (x * sigmoid(x); the tanh-GELU approximation), so the probe arms one only when the block's resolved activation computes that function exactly. Exact (erf) GELU does not arm the tanh-GeGLU kernel.
The probe is behavioral: it evaluates the activation on a fixed fp32 vector and demands bitwise equality. A type test would not do: ACT2FN["silu"] is a SiLUActivation module that is neither nn.SiLU nor F.silu, so an isinstance/identity check disarms the kernel on every real block, costing throughput with no numerical trace.
The probe fails closed: an activation it cannot vouch for runs eager rather than getting substituted.
CP wrapper
Glm4MoeLiteAttention → Glm4MoeLiteUlyssesAttention (halo/distributed/context_parallel/families.py), which declares only its HF class name. Everything below lives on the shared MLAUlyssesAttention, which runs the DeepSeek-V3 MLA geometry on the one Ulysses path. Two MLA-specific bits:
Head dims —
kv_b_projexpands the compressed KV intoqk_nope_head_dim + v_head_dim. Wherev_head_dimis narrower thanqk_head_dimthe base pads V for the flash kernel and crops the output back.On GLM-4.7-Flash the two match at 256 (192 nope + 64 rope), so the pad and crop are no-ops.
Shared rope head —
kv_a_proj_with_mqaemits[kv_lora_rank + qk_rope_head_dim]; the trailing slots are one rotary K vector broadcast to all KV heads before the Ulysses all-to-all, so the head-dim scatter gets a contiguous tensor.
GLM-4 has no llama-4 position scaling, so the base forward is unchanged. RoPE follows config.rope_interleave and applies only to the rope half.
TP
Selective TP via apply_tp_to_attention_only() needs a tailored plan for the LoRA-style attention:
- Sharded (colwise):
q_b_proj,kv_b_proj— the expansion linears. - Sharded (rowwise):
o_proj— reduces back over the head dim the expansions split. - Replicated:
q_a_proj,kv_a_proj_with_mqaand their layer norms — low-rank compressions (often dim ≤tp_size × num_heads) where sharding produces degenerate per-rank shapes.
ETP
The routed experts use fused-GLU contiguous-halves storage, so at expert_tp_size > 1 _init_fused_glu_params splits the halves before sharding and stores gate_proj / up_proj / down_proj separately. Shared experts stay replicated. See ETP.
Router balancing and precision
GLM-4 MoE Lite's router is DeepSeek-V3 aux-loss-free (sigmoid + a static e_score_correction_bias) and the forward has no aux-loss term, so aux_loss adds no balancing. Glm4MoeLiteForCausalLM.forward declares no output_router_logits parameter, so auto resolves to bias_update wherever the EP layers are present (ep_size > 1 or use_grouped_gemm, the default under torchrun); the shipped EP configs set it explicitly.
BalancingBiasCallback sign-updates the gate's own e_score_correction_bias — the native slot the family's selection arithmetic already consults, adopted whole rather than stacked with a side buffer — so the final bias exports with every checkpoint and a served copy routes on it. To freeze the router instead, use freeze_layers_patterns: ["*.mlp.gate.weight"].
Router precision comes from fp32_router, and fp32_non_ep_params (dense attention/embeddings/lm_head) turns it on regardless, since that upcast skips the EP-owned router. The example config sets both.
Chat templates
Three templates under jinja-templates/glm/ serve this family (glm5-chat.jinja beside them is GLM-5 Next's). The first two render the native non-thinking assistant format (<|assistant|></think>…):
glm-instruct.jinja— single-turn (raises on more than one assistant message).glm-chat.jinja— multi-turn system/user/assistant.glm-native.jinja— the verbatim upstreamzai-org/GLM-4.7-Flashtemplate: system messages, tools (<tool_call>XML arg pairs), thinking viareasoning_content/<think>, observation turns. Use it when the training render must match native serving byte-for-byte (tool data, RL); tool-callargumentsmust be a parsed mapping, not the OpenAI JSON-string form.
Neither injects a per-turn <|endoftext|> mid-conversation, matching native serving where an assistant turn ends at the next role marker (<|user|> / <|observation|>, both in config.eos_token_id). Completion masking stops each turn at that terminator (halo/data/turn_markers.py). The final assistant turn ends with an explicit <|endoftext|> so it carries an in-document terminator under packing.
Pair them with assistant_message_template: "<|assistant|>" — the role token alone marks every assistant turn and matches both the thinking and non-thinking renders, where <|assistant|></think> would silently miss a thinking turn. See Collators.
Configs and serving
The release's MTP tail (model.layers.47.*) drops at load and exports carry none, so method=mtp refuses them, unless the run trains the head.
GLM-4.7-Flash (47 layers, 64 routed experts, top-k=4) trains under EP=8 at ~30K max length: examples/sft/glm4/glm-4.7-flash-ultrachat-ep.yaml.
Gathered EP saves and EP-shard merges (merge_ep_shards.py) both write the per-expert hub layout both pinned engines read directly — GLM-4's spec declares a FUSED_SPLIT_PER_EXPERT ExpertLayout, which the gather and the merge both apply. A checkpoint fused before Halo loaded it stays fused through the writers without an EP layer, which keep the loaded layout (Serving on vLLM / SGLang), and needs scripts/after_training/unfuse_moe_experts.py first.
MLA on Blackwell needs vLLM --attention-backend CUTLASS_MLA or SGLang --attention-backend triton. See Serving on vLLM / SGLang.
RL weight sync runs on either engine. Two SGLang facts shape it.
SGLang 0.5.22's gate holds its weight in fp32 and reads it live, so a synced router weight reaches routing; the image build asserts it (docker/sglang/patches/patch_sglang.py), so a release that caches the weight again fails the build (Rollout Servers).
SGLang's MLA loader fuses q_a_proj with kv_a_proj_with_mqa from a cache local to one request, dropping a half that arrives alone. The client declares the pair (CO_LOADED_PARAM_GROUPS) and the chunker keeps both halves in one chunk (Rollout Servers).
Serve it with SGLANG_ATTENTION_BACKEND=triton (the engine's default backend has no kernel for its head size and exits at start) and SGLANG_EXTRA_ARGS=--enable-deterministic-inference. Under the triton backend its greedy logits differ between a prefill and a prefix-cache hit of the same prompt, which the server tier's zero-noise baseline probe refuses (training tolerates it as sampling noise; the flag is optional there).
Source on GitHubUpdated 18 November 2026