GLM-4 MoE Lite

Glm4MoeLiteForCausalLM — MoE with sigmoid routing (group-limited top-k), shared experts, and LoRA-style attention compression (q_a_proj / q_b_proj + kv_a_proj_with_mqa / kv_b_proj).

Per-axis support: Compatibility matrix. Pipeline parallelism splits it at any layer offset (Pipeline Parallelism).

Its MLA-style attention (192 + 64 = 256-wide query/key, 256-wide value) runs FA2 on Blackwell in every run, Context Parallelism and full_determinism included: the resolver keeps latent attention at that width on FA2 as a precaution (Flash Attention → Model-specific handling). FA3 on Hopper.

EP layer

EPGlm4MoELayer (halo/distributed/expert_parallel/families/glm4.py) replaces Glm4MoeLiteMoE. Laguna shares the shape and subclasses this EP layer — see Laguna.

  • Routing: sigmoid + group mask + top-k. topk_group selects eligible expert groups, then top_k (=4) experts are chosen across them. route_tokens_to_experts() runs this before DeepEP dispatch.
  • Routed experts: Glm4MoeLiteExperts, fused gate_up_proj and down_proj 3D tensors.
  • Shared experts: Glm4MoeLiteMLP, replicated per rank, added after DeepEP combine.
  • Activation: SwiGLU on the split fused gate_up; Grouped GEMM on SM90+.

Fused SwiGLU

The _fused_glu_mul latch replaces the activation and the multiply with one Triton kernel on every base compute path, the ETP-sharded one included: the packed [gate | up] path runs it through GluCombine.packed, split halves through _glu_combine. It is a roster-wide seam, not a GLM-4 one: _resolve_activation latches whatever resolve_fused_glu_mul (halo/kernels/fused_glu.py) returns for the resolved activation — GluCombine.silu() for a SiLU gate (GLM-4 Lite, Laguna, Qwen3, Qwen3.5/3.6, Bailing, LFM-2, Mistral4, Cohere2 MoE, Inkling, Zaya, and Step-3.7's unclamped layers), GluCombine.gelu_tanh() for a tanh-GELU one (Gemma 4), None otherwise.

A family opts in by having a gate the kernels implement, not by setting a flag. A family whose combine is a variant rebinds the same seam: DeepSeek-V4 (clamp before the activation) and Step-3.7 (its two clamped layers clamp after it) arm their kernels off the same SiLU probe.

GLM-5 Next binds its pre-activation clamp unconditionally, since its experts hardcode that gate. GPT-OSS owns its interleaved-bias compute paths and never reaches the seam.

Each kernel hardcodes its activation (x * sigmoid(x); the tanh-GELU approximation), so the probe arms one only when the block's resolved activation computes that function exactly. Exact (erf) GELU does not arm the tanh-GeGLU kernel.

The probe is behavioral: it evaluates the activation on a fixed fp32 vector and demands bitwise equality. A type test would not do: ACT2FN["silu"] is a SiLUActivation module that is neither nn.SiLU nor F.silu, so an isinstance/identity check disarms the kernel on every real block, costing throughput with no numerical trace.

The probe fails closed: an activation it cannot vouch for runs eager rather than getting substituted.

CP wrapper

Glm4MoeLiteAttention → Glm4MoeLiteUlyssesAttention (halo/distributed/context_parallel/families.py), which declares only its HF class name. Everything below lives on the shared MLAUlyssesAttention, which runs the DeepSeek-V3 MLA geometry on the one Ulysses path. Two MLA-specific bits:

  • Head dims — kv_b_proj expands the compressed KV into qk_nope_head_dim + v_head_dim. Where v_head_dim is narrower than qk_head_dim the base pads V for the flash kernel and crops the output back.

    On GLM-4.7-Flash the two match at 256 (192 nope + 64 rope), so the pad and crop are no-ops.

  • Shared rope head — kv_a_proj_with_mqa emits [kv_lora_rank + qk_rope_head_dim]; the trailing slots are one rotary K vector broadcast to all KV heads before the Ulysses all-to-all, so the head-dim scatter gets a contiguous tensor.

GLM-4 has no llama-4 position scaling, so the base forward is unchanged. RoPE follows config.rope_interleave and applies only to the rope half.

TP

Selective TP via apply_tp_to_attention_only() needs a tailored plan for the LoRA-style attention:

  • Sharded (colwise): q_b_proj, kv_b_proj — the expansion linears.
  • Sharded (rowwise): o_proj — reduces back over the head dim the expansions split.
  • Replicated: q_a_proj, kv_a_proj_with_mqa and their layer norms — low-rank compressions (often dim ≤ tp_size × num_heads) where sharding produces degenerate per-rank shapes.

ETP

The routed experts use fused-GLU contiguous-halves storage, so at expert_tp_size > 1 _init_fused_glu_params splits the halves before sharding and stores gate_proj / up_proj / down_proj separately. Shared experts stay replicated. See ETP.

Router balancing and precision

GLM-4 MoE Lite's router is DeepSeek-V3 aux-loss-free (sigmoid + a static e_score_correction_bias) and the forward has no aux-loss term, so aux_loss adds no balancing. Glm4MoeLiteForCausalLM.forward declares no output_router_logits parameter, so auto resolves to bias_update wherever the EP layers are present (ep_size > 1 or use_grouped_gemm, the default under torchrun); the shipped EP configs set it explicitly.

BalancingBiasCallback sign-updates the gate's own e_score_correction_bias — the native slot the family's selection arithmetic already consults, adopted whole rather than stacked with a side buffer — so the final bias exports with every checkpoint and a served copy routes on it. To freeze the router instead, use freeze_layers_patterns: ["*.mlp.gate.weight"].

Router precision comes from fp32_router, and fp32_non_ep_params (dense attention/embeddings/lm_head) turns it on regardless, since that upcast skips the EP-owned router. The example config sets both.

Chat templates

Three templates under jinja-templates/glm/ serve this family (glm5-chat.jinja beside them is GLM-5 Next's). The first two render the native non-thinking assistant format (<|assistant|></think>…):

  • glm-instruct.jinja — single-turn (raises on more than one assistant message).
  • glm-chat.jinja — multi-turn system/user/assistant.
  • glm-native.jinja — the verbatim upstream zai-org/GLM-4.7-Flash template: system messages, tools (<tool_call> XML arg pairs), thinking via reasoning_content/<think>, observation turns. Use it when the training render must match native serving byte-for-byte (tool data, RL); tool-call arguments must be a parsed mapping, not the OpenAI JSON-string form.

Neither injects a per-turn <|endoftext|> mid-conversation, matching native serving where an assistant turn ends at the next role marker (<|user|> / <|observation|>, both in config.eos_token_id). Completion masking stops each turn at that terminator (halo/data/turn_markers.py). The final assistant turn ends with an explicit <|endoftext|> so it carries an in-document terminator under packing.

Pair them with assistant_message_template: "<|assistant|>" — the role token alone marks every assistant turn and matches both the thinking and non-thinking renders, where <|assistant|></think> would silently miss a thinking turn. See Collators.

Configs and serving

The release's MTP tail (model.layers.47.*) drops at load and exports carry none, so method=mtp refuses them, unless the run trains the head.

GLM-4.7-Flash (47 layers, 64 routed experts, top-k=4) trains under EP=8 at ~30K max length: examples/sft/glm4/glm-4.7-flash-ultrachat-ep.yaml.

Gathered EP saves and EP-shard merges (merge_ep_shards.py) both write the per-expert hub layout both pinned engines read directly — GLM-4's spec declares a FUSED_SPLIT_PER_EXPERT ExpertLayout, which the gather and the merge both apply. A checkpoint fused before Halo loaded it stays fused through the writers without an EP layer, which keep the loaded layout (Serving on vLLM / SGLang), and needs scripts/after_training/unfuse_moe_experts.py first.

MLA on Blackwell needs vLLM --attention-backend CUTLASS_MLA or SGLang --attention-backend triton. See Serving on vLLM / SGLang.

RL weight sync runs on either engine. Two SGLang facts shape it.

SGLang 0.5.22's gate holds its weight in fp32 and reads it live, so a synced router weight reaches routing; the image build asserts it (docker/sglang/patches/patch_sglang.py), so a release that caches the weight again fails the build (Rollout Servers).

SGLang's MLA loader fuses q_a_proj with kv_a_proj_with_mqa from a cache local to one request, dropping a half that arrives alone. The client declares the pair (CO_LOADED_PARAM_GROUPS) and the chunker keeps both halves in one chunk (Rollout Servers).

Serve it with SGLANG_ATTENTION_BACKEND=triton (the engine's default backend has no kernel for its head size and exits at start) and SGLANG_EXTRA_ARGS=--enable-deterministic-inference. Under the triton backend its greedy logits differ between a prefill and a prefix-cache hit of the same prompt, which the server tier's zero-noise baseline probe refuses (training tolerates it as sampling noise; the flag is optional there).

Source on GitHubUpdated 18 November 2026