# How to fine-tune GLM-4.7-Flash

To fine-tune GLM-4.7-Flash with Halo, copy the shipped config for zai-org/GLM-4.7-Flash and launch it on eight GPUs with expert parallelism of size 8 (EP8), Flash Attention 2 and a 30,720-token sequence length.

## Steps

1. **Start the training container.** Set up the host as [installation](/halo/docs/installation) describes, then start the [cookbook container](/halo/docs/cookbooks#start-the-training-container). Run the commands below inside it.
2. **Copy the shipped config.**
   ```bash
   cp examples/sft/glm4/glm-4.7-flash-ultrachat-ep.yaml glm47-sft.yaml
   ```
3. **Set the output directory.** In the copy, set `output_dir: /data/checkpoints/glm-4.7-flash-ultrachat-ep8` so the checkpoint lands on the scratch volume.
4. **Check what it trains.** The config runs supervised fine-tuning of all weights on `zai-org/GLM-4.7-Flash`. Among its settings:
   ```yaml
   model_name_or_path: zai-org/GLM-4.7-Flash
   moe_balancing: bias_update
   expert_parallel_size: 8
   attn_implementation: flash_attention_2
   max_length: 30720
   ```
5. **Launch eight processes.** The recipe uses eight NVIDIA B300 GPUs.
   ```bash
   halo launch sft glm47-sft.yaml -n 8
   ```

## Common mistakes

- **Flash Attention 4.** Its backward pass emits NaN gradients on GLM's compressed attention, so the loader demotes it to SDPA (scaled dot-product attention). [Flash Attention](/glossary/flash-attention) 2 is the stable choice.
- **Serving on Blackwell.** Set `SGLANG_ATTENTION_BACKEND=triton` when you serve the checkpoint with [SGLang](/glossary/sglang), or the server exits at start.

## With Halo

GLM-4.7-Flash is a [mixture-of-experts](/glossary/mixture-of-experts) model with 64 routed experts, four selected per token. With [expert parallelism](/glossary/expert-parallelism) of size 8 (EP8), each GPU owns eight routed experts, and Halo gathers the expert weights when it saves. The [GLM-4.7-Flash cookbook](/halo/docs/cookbooks/halo-glm-4.7-flash-cookbook) adds context, tensor and expert-tensor parallelism, [LoRA](/glossary/lora) on GLM's compressed attention, and reinforcement learning with GRPO.

