GLM-5 Next
Glm5NextForConditionalGeneration (zai-org/GLM-5.3-Flash) — 321B total / 18B active composite VLM, native in transformers (model_type glm5_next, text tower glm5_next_text; no remote code). 45 text layers mix KDA linear attention with DeepSeek sparse attention, hyper-connections widen the cross-layer residual 4×, and a 288-expert sigmoid-routed MoE carries most of the parameters.
Per-axis support: Compatibility matrix. Pipeline parallelism takes the text tower only (GLM5_NEXT_PP): a text-only run of the composite is admitted and re-emits the vision tower in every checkpoint, image data is refused, and no stage may begin on a shared DSA indexer layer (Pipeline Parallelism).
Architecture
Attention interleave — 34
Glm5NextTextLinearAttentionlayers (KDA: depthwise conv1d k=4 + delta-rule scan) and 11Glm5NextTextAttentionlayers (indexed_attention: MLA withq_lora_rank1536 /kv_lora_rank512, qk 256 NoPE), 1-in-4 from index 3.No text layer applies a position encoding: the config refuses a non-zero
qk_rope_head_dim, and the top-2048Glm5NextTextIndexerapplies no rotary either, so position reaches a DSA layer only through the indexer's pool boundaries.layer_typesspells"linear_attention", which is what the CP gate reads.Hyper-connections (
hc_mult: 4,hc_sinkhorn_iters: 20,hc_eps: 1e-6) — per-layerhc_attn_*/hc_ffn_*tensors route a 4×-widened residual state across layers; the mHC mapping (Sinkhorn-projected combine matrix) computes in fp32 by upcasting its inputs andfninside the forward, and the model declares_is_stateful = True. On CUDA the mapping and the mix run on the hyper-connection kernels, and together they take about a sixth of their time on upstream's forwards (Throughput).MoE — 288 routed experts, sigmoid top-8
noaux_tc(n_group1 → plain top-k),norm_topk_prob: true,routed_scaling_factor: 2.5, 1 shared expert,moe_intermediate_size2048; the first 3 layers are dense.swiglu_limit: 10.0clamps the GLU; the EP layer runs its own fused Triton clamped SwiGLU (GluCombine.clamped_silu, bound as a runtime kernel argument), latched into the generic fused-GLU seam the same way DeepSeek-V4 does (GLM-4 → Fused SwiGLU).MTP — the checkpoint ships one MTP layer (index 45); transformers drops it on load (
_keys_to_ignore_on_load_unexpected), so every export ships without it.mtp_loss_weightrefuses the family: its decoder class buildsattn_hcandffn_hclayers the release's tail lacks, and the release stores the tail in fp8.
Checkpoint conversion
The only release is fp8-e4m3 block-quantized (quantization_config: 128×128 blocks, fp32 per-block *_scale_inv sidecars; the KDA stack, norms, router, hyper-connection tensors and vision tower stay unquantized). EP requires plain BF16 experts, so convert once:
Bash
D=/path/to/large/volume # verified with df -h / findmnt
HF_HOME=$D/hf python scripts/before_training/convert_glm5_bf16.py \
--model_id zai-org/GLM-5.3-Flash --output_dir $D/models/GLM-5.3-Flash-BF16Budget ~330 GB download cache + ~650 GB output; the conversion streams shard-by-shard, so host RAM stays bounded by --max_shard_size. Unquantized tensors keep their stored dtype, and the emitted config.json drops its quantization_config.
The family's fp32 pins are transformers' _keep_in_fp32_modules_strict (e_score_correction_bias, conv1d, dt_bias, A_log); the hc_* tensors are not among them. Every training loader casts the pinned parameters to the run dtype unless fp32_non_ep_params keeps them as fp32 masters (Load precision), while the e_score_correction_bias buffer stays fp32, and every toolkit save keeps each at its trained dtype (save_dtype_caster).
Model loading
The checkpoint is a composite VLM with no text-only CausalLM sibling in transformers, so text_only_model is refused for this family. The composite class loads through the shared VLM path; a text-only dataset still takes the text data path (SFT — VLMs).
EP layer
EPGlm5NextMoELayer (halo/distributed/expert_parallel/families/glm5_next.py) replaces Glm5NextTextMoE.
Routing: sigmoid scores plus the fp32
gate.e_score_correction_biason selection only (the routed weight comes from the unbiased scores), top-8,norm_topk_prob, thenrouted_scaling_factor— fp32 top-k weights, expert load recorded for balancing.Experts: fused
Glm5NextTextExperts.gate_up_proj [E, 2M, H]/down_proj [E, H, M]in-module. The hub spells them per-expert (experts.{i}.gate_proj/up_proj/down_proj); transformers' weight converter restores the fused layout on load and, on asave_pretrainedwith no EP layer, the per-expert spelling on save.An EP-gathered save writes the hub's vendor namespace (
hc_attn_*, the KDAf_a_proj/A_logfamily, the splitq/k/v_conv1d) and the experts per expert (FUSED_SPLIT_PER_EXPERT), the hub checkpoint's own layout, so no un-fuse step follows. A checkpoint saved before that carries the fused pair:unfuse_moe_experts.pystill converts it, and the lazy loaders andfrom_pretrainedread either layout.Shared expert: one
Glm5NextTextMLP, replicated per rank, added after DeepEP combine.The 3 dense leading layers keep their plain MLP — EP swaps MoE blocks only.
ETP uses the shared fused-GLU helper: at expert_tp_size > 1 the gate/up halves store as separate shards (ETP).
EP loads take the lazy safetensors path. The hub checkpoint keeps a vendor namespace (hc tensors, the KDA forget_gate family, a three-source q/k/v_conv1d → conv1d merge, per-expert projections), and the lazy loaders plan their keys on transformers' glm5_next conversion, as from_pretrained does, while fusing only this rank's experts, bit-exact against from_pretrained (tests/gpu/parallelism/ep/test_lazy_load_converted.py).
There is no full-model host residency at 321B (host RAM under the lazy path): 628 GB of page cache node-wide at ep8 (the bf16 conversion minus the MTP tail, shared by the 8 ranks), a 73–75 GiB host RSS peak per rank while its shard handles are open, and 2.3 GiB after. The per-expert gate+up fusion's 2× transient (1.2 GB per layer) is on the GPU.
RL weight sync
Online and environmental GRPO sync the family into both pinned engines (vLLM 0.32.0 Glm5NextForConditionalGeneration, SGLang 0.5.22 Glm5NextForConditionalGeneration). The push sends the hub spelling: per-expert gate_proj/up_proj/down_proj and the shared expert, the KDA q/k/v projections, b/f_a/g_a/f_b/g_b, the split q_conv1d/k_conv1d/v_conv1d that the reverse of transformers' conv1d merge produces, the DSA indexer, the hc_attn_*/hc_ffn_* tensors, the router weight and the vision tower. gate.e_score_correction_bias stays out while moe_balancing: none: vLLM skips it on reload and SGLang keeps the launch value. tests/cpu/grpo/test_glm5_next_engine_loader_names.py holds the push to the names both loaders map, recorded from the images.
SGLang's hyper-connection path reads hc_attn_fn/hc_ffn_fn live on every forward (hc_norm_weight=None), so it has no cached padded copy of the mix weight to go stale, unlike Qwen3.8-Flash-Next. The rows of tests/gpu/trainers/grpo/test_weight_push_exact.py --family glm5_next serve a tiny random checkpoint written by --write-checkpoint (its config carries the release keys both engines read and transformers' class does not declare, such as mhc and index_kpool_compress, and the release's attention, indexer and KDA head dims, which the engines' sparse-MLA and indexer kernels gate, and the release's vision patch, merge, image and head sizes, which the engines' processors and vision rotary gate; layers, experts, hidden size, vision depth and vocabulary stay tiny) and hold it to a bit-identical unchanged push, a scaled control, and one real optimizer step; none has run against a live server yet. The name-for-name --reach rows leave the family out, since scaling an indexer weight cannot move the log-probs.
Serving flags for the two engines (set through SGLANG_EXTRA_ARGS / VLLM_EXTRA_ARGS in Rollout Servers):
- SGLang:
--mamba-track-interval 1024. The checkpoint'sindex_kpoolof 16 makes the engine's DSA page 64 x 16 = 1,024 tokens, and the hybrid-linear radix cache the family resolves to (extra_buffer) asserts thatmamba_track_interval(default 256) is a multiple of the page size, so a launch without the flag exits at argument resolution. Any multiple of 1,024 passes. - SGLang:
--skip-server-warmup(optional). For any model it advertises as image-capable, the server's startup warmup is a chat request with a 32x32 PNG, so a text-only launch still runs the vision tower once; a vision config the engine cannot run kills the scheduler there, in the prefill graph path. The tiny checkpoint's vision tower carries the release's patch, merge, image and head sizes, so the warmup passes; the flag skips it for a checkpoint whose tower is not release-sized. - vLLM:
--limit-mm-per-prompt.image 0 --limit-mm-per-prompt.video 0(optional; the same as--limit-mm-per-prompt '{"image":0,"video":0}'). RL feeds the text tower only, and the flag skips the vision-encoder profiling, which fails on a vision tower whose patch size differs from the 14 px the processor cuts (shape '[32000, -1, 2, 4, 4]').
Router balancing
The router carries a native fp32 e_score_correction_bias buffer — GLM-4's exact slot — and the forward honors output_router_logits, so moe_balancing: auto resolves to aux_loss (the Laguna case), whose switch-style aux loss softmaxes the raw router logits and so balances the unbiased routing, not the selection e_score_correction_bias shifts. Explicit bias_update adopts the native buffer instead: BalancingBiasCallback sign-updates it, and the buffer exports with every checkpoint, so a served copy routes on it.
Attention
Upstream declares _supports_flash_attn = False; SDPA is the only fast backend — pin attn_implementation: sdpa (the resolver diverts any flash request there itself). The KDA layers run their own linear-attention kernels regardless of the backend choice, and their autotune picks can move a run's logits across fresh caches (Docker); they reach fla through the hub-kernel funnel the toolkit repairs, and on Blackwell share the delta-rule state kernel the toolkit keeps on num_warps=2 (Qwen3.5 → Linear attention) — without the repair the funnel silently captures the torch scan.
The 11 DSA layers keep config._attn_implementation == "sdpa" and its boolean [B, 1, S, S] mask, but on Blackwell the sdpa slot of modeling_glm5_next holds masked_matmul_attention (halo/models/attention/masked_matmul.py, how it works). Any call it does not serve goes to the library's SDPA. HALO_DSA_MATMUL defaults off on other GPUs, and 0 keeps stock SDPA throughout. A tree forward carrying a plan attends in its own forward and never reaches this slot.
A DSA query reads about 1,800 to 1,900 keys, at most 2,051, under a mask as wide as the row. Torch's mem-efficient kernel is a general one: it scores every key of the row, and its backward costs about 7.5× its forward. The matmul kernel runs a forward, its recompute and a backward 4.95× faster at 8k and 5.13× at 16k (numbers).
On a 7-layer cut at ep1 and 32,752 tokens, the per-token gap to an fp32 reference reads 0.0195 with both kernels on and 0.0232 with neither, and top-k disagreement 1.00% against 1.34%. With hyper-connection kernels off, the DSA kernel's forward is bit-identical to SDPA's.
Throughput
Both kernels on, with torch's general kernel and upstream's hyper-connections in parentheses: B300, reentrant checkpointing (non-reentrant on pp2 × ep8, which refuses reentrant), padded rows of real text, one run per arm.
- The PP rate counts attended tokens (how a PP rate is counted). PP leads
ep16by 1.61×, 1.16× on the general kernels. ep8with DP runs out of memory in the first optimizer step, about 3 minutes in, since no gate refuses it first. At 40.2B trainable per rank, AdamWBF16's state takes the allocation to 243–250 GiB of 267.7.- The LoRA rows are long real-text rows averaging about 7,760 and 15,520 tokens, rendered with
glm-chat.jinja. The shipped recipe readsultrachat_200kat about 1,140 tokens a row underglm5-chat.jinja; it ran 5 steps on a cold cache with no steady rate to read, and its loss and rate do not compare with these rows. - Raising
max_lengthstill costs tok/s, 16% from 8k to 16k against 30% on the general kernels: a query's keys are capped at 2,051, while each tile's forward still scores its crop up to its last selected key. - MFU charges a DSA layer
index_topk, 2,048 keys, plus the pooled indexer.

On a per-part clock, which adds about 2.6 s, that step takes 24.5 s against 51.1. DSA attention (18.9%) and MoE compute (18.4%) are its largest parts, then MoE router, dispatch and combine (15.0%), KDA conv, projections, gate and output (13.4%) and the KDA scan (12.5%). Every expert call runs grouped GEMM, with no per-expert loop.
Limitations
- CP — 34 of 45 layers are a KDA linear recurrence over the sequence axis; validation rejects both the
Glm5NextTextLinearAttentionmodule and anylayer_typescontaining"linear_attention"(Context Parallelism). - TP — no shard plan: the DSA indexer and the KDA projections have no sound sharding, so a
tensor_parallel_size > 1run is rejected (zero shardable layers). - Packing — flat packing isolates neither layer type: both receive only the 2D padding mask, so the DSA layers attend across the documents of a packed row and the KDA conv/scan carries state across them (Collators). The family runs no varlen backend, so
dense_packing_leakson its spec refuses flatpackingwhateverallow_cross_document_linear_attentionsays, andpadding_freeis refused on SDPA as well; so is packing under pipeline parallelism (PP packing). Tree packing isolates both layer types; otherwise train on padded rows.
Tree packing
tree_packing: true (Tree Packing) leaves the resolved backend (SDPA) in place for plan-less forwards; under a plan neither layer type reaches the attention interface, so tree_full_attention is refused.
- KDA layers take the GatedDeltaNet treatment: the conv runs each node behind the last 3 tokens of its path, and the KDA scan (fla
chunk_kda) runs one tree depth at a time from the parent's final state. - DSA layers select along each token's own path: the indexer's 4-token pools are a token and its path predecessors, and a query's candidates are the pools whose last token precedes it on its path, plus its incomplete tail pool. Attention over the selected keys runs on the 512-wide MLA latent with
kv_b_projfolded in, not on a dense[T, T]mask; under autocast it reads the folded query and the latent in the autocast dtype. An adapter onkv_b_proj, a quantized load (load_in_4bit/load_in_8bit, or the fp8 release before conversion) and a training-timeattention_dropoutare refused before the model loads (a regex target at the first tree batch): dropkv_b_projfrom the LoRA example'slora_target_modules.
The plan comes with every batch, so plain documents packed into one row are isolated in both layer types as well. jinja-templates/glm/glm5-chat.jinja renders every turn byte for byte as the hub template renders GLM-5's own non-thinking turn (<|assistant|><think></think> and the answer), minus the hub's Reasoning Effort system line and plus <|endoftext|> on the final turn (tests/cpu/data/test_glm5_chat_template.py, against the hub template vendored at one revision). Its generation prompt closes the empty block, so a model served with it answers directly; the hub template's opens <think>. It refuses tool definitions, tool calls and tool results instead of dropping them, since it does not render the hub's tool format. On top of the options tree packing requires (packing, tree_share_markers, train_on_completions_only), a GLM-5 run sets:
YAML
attn_implementation: sdpa
tree_packing: true
chat_template: jinja-templates/glm/glm5-chat.jinja
force_chat_template: true
assistant_message_template: "<|assistant|>"Configs
examples/sft/glm5_next/glm-5.3-flash-ultrachat-ep-lora.yaml trains LoRA + expert-LoRA at EP=8 on one
8-GPU machine: 105.6 GiB allocated and 109.4 GiB on the device after a load of about 45 s. A save with
merge_expert_lora_on_save takes about 8 minutes and 612 GiB on disk. It renders with
glm5-chat.jinja, GLM-5's own non-thinking turn (Tree packing); GLM-4's glm-chat.jinja
writes a lone </think> GLM-5 never does.
The config points model_name_or_path at the BF16 conversion output above. Full fine-tuning takes two 8-GPU machines (rates), through pp2 × ep8 on padded rows or cross-node EP (ep_scope: global, e.g. EP=16 across 2×8), where the Gin dispatch ceiling caps per_device_train_batch_size × max_length at 8192 tokens/rank (DeepEP).
Source on GitHubUpdated 18 November 2026