GLM-4.7-Flash Cookbook

GLM-4.7-Flash is a GLM-4 MoE Lite model: 64 routed experts plus a shared expert, four picked per token by a sigmoid router. Its attention compresses queries and keys through low-rank projections (MLA).

Support

Modes: the GLM-4 MoE Lite row of the Supported Matrix; online RL: Rollout engines.

  • Checkpoint: zai-org/GLM-4.7-Flash.
  • GPUs: eight at EP8, eight experts per GPU, at 30,720 tokens.

Recipes

Run these in the training container.

Full fine-tune

Bash

halo launch sft examples/sft/glm4/glm-4.7-flash-ultrachat-ep.yaml -n 8 \
  --output_dir=/data/checkpoints/glm-4.7-flash-sft

Other layouts

Add one of these to the full fine-tune command:

  • EP8 + CP2, for long sequences: --context_parallel_size=2 --packing=false. EP+CP needs EP to fill the node, so EP8 is the only EP size that pairs with CP on eight GPUs.
  • EP8 + TP2, when attention memory is the limit: --tensor_parallel_size=2
  • Pure ETP8, to shard every expert instead of placing whole experts: --expert_parallel_size=1 --expert_tensor_parallel_size=8
  • A 4-way expert split: --expert_parallel_size=4 --expert_tensor_parallel_size=2

LoRA

Bash

halo launch sft examples/sft/glm4/glm-4.7-flash-ultrachat-ep.yaml -n 8 \
  --use_peft=true --learning_rate=1e-4 \
  --lora_target_modules=q_a_proj,q_b_proj,kv_a_proj_with_mqa,kv_b_proj,o_proj \
  --output_dir=/data/checkpoints/glm-4.7-flash-lora

MTP head

Fit the tail the release ships over a frozen trunk, so a server can draft with it. Its export carries the head's trunk copies, which vLLM's loader requires (MTP heads ↗).

This recipe's own fit is not measured. A separate fit at its learning rate, on first-turn rows rather than the recipe's, fell from 0.7685 to 0.7649 acceptance, which the gate stops, and that export served at 0.7317 on vLLM against the release head's 0.7450. No served gain is measured.

Bash

halo launch sft examples/sft/glm4/glm-4.7-flash-mtp-head-only.yaml -n 8 \
  --output_dir=/data/checkpoints/glm-4.7-flash-mtp-head

GRPO

Read Before a GRPO run first. This setup serves the SFT checkpoint from vLLM on GPUs 0–3 and trains on GPUs 4–7.

RL uses jinja-templates/glm/glm-native.jinja, the upstream template with tool calls, because the SFT template renders no tools. The server must serve the same file. On the host:

Bash

VLLM_MODEL=/data/checkpoints/glm-4.7-flash-sft VLLM_CHAT_TEMPLATE=/jinja-templates/glm/glm-native.jinja \
VLLM_CUDA_DEVICES=0,1,2,3 VLLM_TP=4 VLLM_ENABLE_R3=1 \
VLLM_TOOL_PARSER=glm47 VLLM_ATTENTION_BACKEND=CUTLASS_MLA \
  docker compose -f docker-compose.vllm.yml up -d vllm-server

In the training container, copy the template and set dataset, environment_type and rewards. The template's generation prompt opens <think>, so rollouts start in thinking mode while SFT trained the non-thinking render. To keep rollouts non-thinking, also add rollout_chat_template_kwargs: {enable_thinking: false} to the copy. Then launch:

Bash

cp examples/grpo/environmental/environmental-grpo-template.yaml glm47-grpo.yaml
CUDA_VISIBLE_DEVICES=4,5,6,7 halo launch grpo/environmental glm47-grpo.yaml -n 4 \
  --model_name_or_path=/data/checkpoints/glm-4.7-flash-sft \
  --chat_template=jinja-templates/glm/glm-native.jinja --force_chat_template=true \
  --routing_replay=rollout --beta=0.0 \
  --output_dir=/data/checkpoints/glm-4.7-flash-grpo

Add --expert_parallel_size=4 to shard the experts across the four trainer GPUs. For SGLang, start the server with SGLANG_ATTENTION_BACKEND=triton, SGLANG_CHAT_TEMPLATE on the same file and SGLANG_ENABLE_R3=1, and add --rollout_backend=sglang --rollout_server_url=http://localhost:30000 to the trainer.

Settings that matter

  • Attention. The recipe pins flash_attention_2, which Blackwell also picks unpinned: Halo keeps GLM's 256-wide latent attention on FA2 as a precaution, under CP and full_determinism too (why ↗). CP refuses an explicit sdpa label.
  • Router balancing. The recipe sets moe_balancing: bias_update, which auto also picks. It updates the router's own e_score_correction_bias, so the trained bias ships in every checkpoint and the served model routes on it. The router has no auxiliary loss to use instead.
  • Chat templates. SFT uses glm-chat.jinja for multi-turn data. glm-instruct.jinja raises on more than one assistant turn. Keep assistant_message_template: "<|assistant|>": the role token alone matches both thinking and non-thinking turns.
  • Tool parser. vLLM needs VLLM_TOOL_PARSER=glm47. The default hermes can't read GLM tool calls, and the run stops by its second round.

Limits

  • On Blackwell, both engines need an MLA attention backend that accepts GLM's head size: VLLM_ATTENTION_BACKEND=CUTLASS_MLA or SGLANG_ATTENTION_BACKEND=triton. With SGLang's default the server exits at startup.
  • Pipeline parallelism passes tiny-model GPU tests only.

Export and serve

Gathered saves write the hub's per-expert layout, which both engines read, and the other writers keep the layout the checkpoint was loaded in. A checkpoint fused before Halo loaded it holds the fused pair and needs halo run unfuse-moe-experts before serving.

Smoke-test with AutoModelForCausalLM (snippet). Serve from the host with SGLang on port 30000:

Bash

SGLANG_MODEL="$HALO_SCRATCH/checkpoints/glm-4.7-flash-sft" SGLANG_ATTENTION_BACKEND=triton \
  docker compose -f docker-compose.sglang.yml up -d

Reference

Source on GitHubUpdated 18 November 2026