GLM-4.7-Flash Cookbook
GLM-4.7-Flash is a GLM-4 MoE Lite model: 64 routed experts plus a shared expert, four picked per token by a sigmoid router. Its attention compresses queries and keys through low-rank projections (MLA).
Support
Modes: the GLM-4 MoE Lite row of the Supported Matrix; online RL: Rollout engines.
- Checkpoint:
zai-org/GLM-4.7-Flash. - GPUs: eight at EP8, eight experts per GPU, at 30,720 tokens.
Recipes
Run these in the training container.
Full fine-tune
Bash
halo launch sft examples/sft/glm4/glm-4.7-flash-ultrachat-ep.yaml -n 8 \
--output_dir=/data/checkpoints/glm-4.7-flash-sftOther layouts
Add one of these to the full fine-tune command:
- EP8 + CP2, for long sequences:
--context_parallel_size=2 --packing=false. EP+CP needs EP to fill the node, so EP8 is the only EP size that pairs with CP on eight GPUs. - EP8 + TP2, when attention memory is the limit:
--tensor_parallel_size=2 - Pure ETP8, to shard every expert instead of placing whole experts:
--expert_parallel_size=1 --expert_tensor_parallel_size=8 - A 4-way expert split:
--expert_parallel_size=4 --expert_tensor_parallel_size=2
LoRA
Bash
halo launch sft examples/sft/glm4/glm-4.7-flash-ultrachat-ep.yaml -n 8 \
--use_peft=true --learning_rate=1e-4 \
--lora_target_modules=q_a_proj,q_b_proj,kv_a_proj_with_mqa,kv_b_proj,o_proj \
--output_dir=/data/checkpoints/glm-4.7-flash-loraMTP head
Fit the tail the release ships over a frozen trunk, so a server can draft with it. Its export carries the head's trunk copies, which vLLM's loader requires (MTP heads ↗).
This recipe's own fit is not measured. A separate fit at its learning rate, on first-turn rows rather than the recipe's, fell from 0.7685 to 0.7649 acceptance, which the gate stops, and that export served at 0.7317 on vLLM against the release head's 0.7450. No served gain is measured.
Bash
halo launch sft examples/sft/glm4/glm-4.7-flash-mtp-head-only.yaml -n 8 \
--output_dir=/data/checkpoints/glm-4.7-flash-mtp-headGRPO
Read Before a GRPO run first. This setup serves the SFT checkpoint from vLLM on GPUs 0–3 and trains on GPUs 4–7.
RL uses jinja-templates/glm/glm-native.jinja, the upstream template with tool calls, because the SFT
template renders no tools. The server must serve the same file. On the host:
Bash
VLLM_MODEL=/data/checkpoints/glm-4.7-flash-sft VLLM_CHAT_TEMPLATE=/jinja-templates/glm/glm-native.jinja \
VLLM_CUDA_DEVICES=0,1,2,3 VLLM_TP=4 VLLM_ENABLE_R3=1 \
VLLM_TOOL_PARSER=glm47 VLLM_ATTENTION_BACKEND=CUTLASS_MLA \
docker compose -f docker-compose.vllm.yml up -d vllm-serverIn the training container, copy the template and set dataset, environment_type and rewards. The
template's generation prompt opens <think>, so rollouts start in thinking mode while SFT trained the
non-thinking render. To keep rollouts non-thinking, also add
rollout_chat_template_kwargs: {enable_thinking: false} to the copy. Then launch:
Bash
cp examples/grpo/environmental/environmental-grpo-template.yaml glm47-grpo.yaml
CUDA_VISIBLE_DEVICES=4,5,6,7 halo launch grpo/environmental glm47-grpo.yaml -n 4 \
--model_name_or_path=/data/checkpoints/glm-4.7-flash-sft \
--chat_template=jinja-templates/glm/glm-native.jinja --force_chat_template=true \
--routing_replay=rollout --beta=0.0 \
--output_dir=/data/checkpoints/glm-4.7-flash-grpoAdd --expert_parallel_size=4 to shard the experts across the four trainer GPUs. For SGLang, start the
server with SGLANG_ATTENTION_BACKEND=triton, SGLANG_CHAT_TEMPLATE on the same file and
SGLANG_ENABLE_R3=1, and add --rollout_backend=sglang --rollout_server_url=http://localhost:30000 to
the trainer.
Settings that matter
- Attention. The recipe pins
flash_attention_2, which Blackwell also picks unpinned: Halo keeps GLM's 256-wide latent attention on FA2 as a precaution, under CP andfull_determinismtoo (why ↗). CP refuses an explicitsdpalabel. - Router balancing. The recipe sets
moe_balancing: bias_update, whichautoalso picks. It updates the router's owne_score_correction_bias, so the trained bias ships in every checkpoint and the served model routes on it. The router has no auxiliary loss to use instead. - Chat templates. SFT uses
glm-chat.jinjafor multi-turn data.glm-instruct.jinjaraises on more than one assistant turn. Keepassistant_message_template: "<|assistant|>": the role token alone matches both thinking and non-thinking turns. - Tool parser. vLLM needs
VLLM_TOOL_PARSER=glm47. The defaulthermescan't read GLM tool calls, and the run stops by its second round.
Limits
- On Blackwell, both engines need an MLA attention backend that accepts GLM's head size:
VLLM_ATTENTION_BACKEND=CUTLASS_MLAorSGLANG_ATTENTION_BACKEND=triton. With SGLang's default the server exits at startup. - Pipeline parallelism passes tiny-model GPU tests only.
Export and serve
Gathered saves write the hub's per-expert layout, which both engines read, and the other writers keep
the layout the checkpoint was loaded in. A checkpoint fused before Halo loaded it holds the fused pair
and needs halo run unfuse-moe-experts before serving.
Smoke-test with AutoModelForCausalLM (snippet). Serve from the
host with SGLang on port 30000:
Bash
SGLANG_MODEL="$HALO_SCRATCH/checkpoints/glm-4.7-flash-sft" SGLANG_ATTENTION_BACKEND=triton \
docker compose -f docker-compose.sglang.yml up -dReference
- GLM-4 model notes ↗: the CP and TP plans for MLA, chat templates, serving
- Async GRPO with Environments
- Model card
Source on GitHubUpdated 18 November 2026