# GLM-4.7-Flash Cookbook

[GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash) is a GLM-4 MoE Lite model: 64 routed
experts plus a shared expert, four picked per token by a sigmoid router. Its attention compresses
queries and keys through low-rank projections (MLA).

## Support

Modes: the GLM-4 MoE Lite row of the [Supported Matrix](https://whitecircle.com/halo/docs/supported-matrix.md#model-families); online RL:
[Rollout engines](https://whitecircle.com/halo/docs/supported-matrix.md#rollout-engines).

- **Checkpoint:** `zai-org/GLM-4.7-Flash`.
- **GPUs:** eight at EP8, eight experts per GPU, at 30,720 tokens.

## Recipes

Run these in the [training container](https://whitecircle.com/halo/docs/cookbooks.md#start-the-training-container).

### Full fine-tune

```bash
halo launch sft examples/sft/glm4/glm-4.7-flash-ultrachat-ep.yaml -n 8 \
  --output_dir=/data/checkpoints/glm-4.7-flash-sft
```

### Other layouts

Add one of these to the full fine-tune command:

- EP8 + CP2, for long sequences: `--context_parallel_size=2 --packing=false`. EP+CP needs EP to fill
  the node, so EP8 is the only EP size that pairs with CP on eight GPUs.
- EP8 + TP2, when attention memory is the limit: `--tensor_parallel_size=2`
- Pure ETP8, to shard every expert instead of placing whole experts: `--expert_parallel_size=1 --expert_tensor_parallel_size=8`
- A 4-way expert split: `--expert_parallel_size=4 --expert_tensor_parallel_size=2`

### LoRA

```bash
halo launch sft examples/sft/glm4/glm-4.7-flash-ultrachat-ep.yaml -n 8 \
  --use_peft=true --learning_rate=1e-4 \
  --lora_target_modules=q_a_proj,q_b_proj,kv_a_proj_with_mqa,kv_b_proj,o_proj \
  --output_dir=/data/checkpoints/glm-4.7-flash-lora
```

### MTP head

Fit the tail the release ships over a frozen trunk, so a server can draft with it. Its export carries the head's trunk
copies, which vLLM's loader requires ([MTP heads](https://whitecircle.com/halo/docs/agent-docs/optimization/mtp-heads.md) ↗).

This recipe's own fit is not measured. A separate fit at its learning rate, on first-turn rows rather than the recipe's,
fell from 0.7685 to 0.7649 acceptance, which the gate stops, and that export served at 0.7317 on vLLM against the release
head's 0.7450. No served gain is measured.

```bash
halo launch sft examples/sft/glm4/glm-4.7-flash-mtp-head-only.yaml -n 8 \
  --output_dir=/data/checkpoints/glm-4.7-flash-mtp-head
```

### GRPO

Read [Before a GRPO run](https://whitecircle.com/halo/docs/cookbooks.md#before-a-grpo-run) first. This setup serves the SFT checkpoint from
vLLM on GPUs 0–3 and trains on GPUs 4–7.

RL uses `jinja-templates/glm/glm-native.jinja`, the upstream template with tool calls, because the SFT
template renders no tools. The server must serve the same file. On the host:

```bash
VLLM_MODEL=/data/checkpoints/glm-4.7-flash-sft VLLM_CHAT_TEMPLATE=/jinja-templates/glm/glm-native.jinja \
VLLM_CUDA_DEVICES=0,1,2,3 VLLM_TP=4 VLLM_ENABLE_R3=1 \
VLLM_TOOL_PARSER=glm47 VLLM_ATTENTION_BACKEND=CUTLASS_MLA \
  docker compose -f docker-compose.vllm.yml up -d vllm-server
```

In the training container, copy the template and set `dataset`, `environment_type` and `rewards`. The
template's generation prompt opens `<think>`, so rollouts start in thinking mode while SFT trained the
non-thinking render. To keep rollouts non-thinking, also add
`rollout_chat_template_kwargs: {enable_thinking: false}` to the copy. Then launch:

```bash
cp examples/grpo/environmental/environmental-grpo-template.yaml glm47-grpo.yaml
CUDA_VISIBLE_DEVICES=4,5,6,7 halo launch grpo/environmental glm47-grpo.yaml -n 4 \
  --model_name_or_path=/data/checkpoints/glm-4.7-flash-sft \
  --chat_template=jinja-templates/glm/glm-native.jinja --force_chat_template=true \
  --routing_replay=rollout --beta=0.0 \
  --output_dir=/data/checkpoints/glm-4.7-flash-grpo
```

Add `--expert_parallel_size=4` to shard the experts across the four trainer GPUs. For SGLang, start the
server with `SGLANG_ATTENTION_BACKEND=triton`, `SGLANG_CHAT_TEMPLATE` on the same file and
`SGLANG_ENABLE_R3=1`, and add `--rollout_backend=sglang --rollout_server_url=http://localhost:30000` to
the trainer.

## Settings that matter

- **Attention.** The recipe pins `flash_attention_2`, which Blackwell also picks unpinned: Halo keeps
  GLM's 256-wide latent attention on FA2 as a precaution, under CP and `full_determinism` too
  ([why](https://whitecircle.com/halo/docs/agent-docs/optimization/flash-attention.md#model-specific-handling) ↗). CP refuses an
  explicit `sdpa` label.
- **Router balancing.** The recipe sets `moe_balancing: bias_update`, which `auto` also picks. It updates
  the router's own `e_score_correction_bias`, so the trained bias ships in every checkpoint and the
  served model routes on it. The router has no auxiliary loss to use instead.
- **Chat templates.** SFT uses `glm-chat.jinja` for multi-turn data. `glm-instruct.jinja` raises on more
  than one assistant turn. Keep `assistant_message_template: "<|assistant|>"`: the role token alone
  matches both thinking and non-thinking turns.
- **Tool parser.** vLLM needs `VLLM_TOOL_PARSER=glm47`. The default `hermes` can't read GLM tool calls,
  and the run stops by its second round.

## Limits

- On Blackwell, both engines need an MLA attention backend that accepts GLM's head size:
  `VLLM_ATTENTION_BACKEND=CUTLASS_MLA` or `SGLANG_ATTENTION_BACKEND=triton`. With SGLang's default the
  server exits at startup.
- Pipeline parallelism passes tiny-model GPU tests only.

## Export and serve

Gathered saves write the hub's per-expert layout, which both engines read, and the other writers keep
the layout the checkpoint was loaded in. A checkpoint fused before Halo loaded it holds the fused pair
and needs `halo run unfuse-moe-experts` before serving.

Smoke-test with `AutoModelForCausalLM` ([snippet](https://whitecircle.com/halo/docs/cookbooks.md#smoke-test-a-checkpoint)). Serve from the
[host](https://whitecircle.com/halo/docs/cookbooks.md#serve-from-the-host) with SGLang on port 30000:

```bash
SGLANG_MODEL="$HALO_SCRATCH/checkpoints/glm-4.7-flash-sft" SGLANG_ATTENTION_BACKEND=triton \
  docker compose -f docker-compose.sglang.yml up -d
```

## Reference

- [GLM-4 model notes](https://whitecircle.com/halo/docs/agent-docs/models/glm4.md) ↗: the CP and TP plans for MLA, chat templates,
  serving
- [Async GRPO with Environments](https://whitecircle.com/halo/docs/training-methods/async-grpo-environments.md)
- [Model card](https://huggingface.co/zai-org/GLM-4.7-Flash)
