[v4] GLM-4.7-Flash-Coder

More reasoning, fewer turns

Although GLM-4.7-Flash-Coder does 2.4× more reasoning than the base model, it completes tasks faster: 45% fewer turns, which translates to 14% lower inference cost.

The key to turn efficiency is concurrent tool calls

By allowing the model to dispatch multiple bash and read calls in a single turn, we significantly reduced the latency bottleneck of sequential execution.

SWE-rebench-V2

We trained and evaluated models on nebius/SWE-rebench-V2 — a dataset of 32k agentic coding tasks. It consists of real issues from public repositories, closely representing real-world SWE challenges.

Teacher — GLM-5.1

The training data comes from GLM-5.1 acting as a teacher. We ran it on the train split with 4 rollouts per task and kept only trajectories that passed the tests — 7,777 successful multi-turn traces after rejection sampling.

We're releasing the training dataset alongside the train/test splits. All models run inside PI — a lightweight agent harness with four tools: bash, read, edit, write.

Training

We trained Coder with Supervised Fine-Tuning (SFT) using halo, our post-training framework we're open-sourcing today.

Halo is driven entirely by a single YAML config:

model_name_or_path: zai-org/GLM-4.7-Flash
trust_remote_code: true
attn_implementation: flash_attention_4_hybrid

dataset:
- whitecircle/swe-rebench-v2-glm-5.1-pi-agent-successful-traces
test_size: 0.01

conversation_field: messages
tools_field: tools
interleaved_thinking: true

max_length: 75000
packing: true

learning_rate: 1e-5
lr_scheduler_type: cosine
warmup_steps: 25
per_device_train_batch_size: 1
per_device_eval_batch_size: 1

moe_balancing: none
liger_kernel_config:
  fused_linear_cross_entropy: true

assistant_message_template: "<|assistant|>"

output_dir: /mnt/research/GLM-4.7-Flash-Coder
save_steps: 50
save_only_model: true

eval_strategy: steps
eval_steps: 50
logging_steps: 1
logging_first_step: true
log_decoded_samples: true

report_to: wandb
project_name: agentic-sft
run_name: glm-4.7-flash-coder

enable_efficiency_metrics: true

Emergent skills

The model picked up several of the teacher's skills and patterns during fine-tuning:

• A 10× higher rate of self-corrections in reasoning
• Self-directed questions ("But wait, could this ever be True?")
• Advanced tool calling patterns, e.g. setting timeout for the bash tool

git stash && python -m pytest tests/test_markdown.py::test_markdown_table -xvs

Loss decomposition into Reasoning & Tool Calls

We only train on assistant responses. Of 177 million tokens in the dataset, only 46 million carry loss — the remaining 131 million (system prompts and tool responses) are masked.

To track convergence separately, we decompose loss into two components — negative log-likelihood on reasoning and tool calling tokens, respectively.

Test scores

Below is the comparison of GLM-4.7-Flash, GLM-4.7-Flash-Coder, and its teacher GLM-5.1 on the test set of SWE-rebench-V2:

How we enabled FA4

FlashAttention-4 is a fast attention kernel — but it's incompatible with GLM-4.7-Flash's architecture:

FlashAttention-4 produces NaN gradients for this model's head_dim-256 partial-rotary
attention; falling back to attn_implementation='sdpa'.

We found that the incompatibility only appears in the backward pass — the forward computation is correct. So we designed a hybrid scheme: FA4 handles the forward pass, while FA2 takes over for the backward. The hybrid approach gave a 1.36× increase in attention throughput.

Kernels

GLM-4.7-Flash is a Mixture-of-Experts model. Tokens are distributed unevenly across experts — that means MLP blocks receive tensors of varying width. The Grouped-GEMM kernel batches matrix multiplications of varying shapes into a single kernel launch. Enabling it gave a 1.6× speedup on the FFN step. Additionally, with Liger Fused Linear Cross-Entropy, we were able to compute loss without full logit matrix materialization in VRAM, preventing OOM on long sequences. Together, these improvements yield up to a 1.63× training speedup over TRL, raising throughput to 3000 tokens/s on a single B300 GPU:

Agentic infra challenge

We faced a real challenge with agentic evaluation infrastructure. At a scale of thousands of concurrent agents, Docker introduces high startup overhead and heavy disk pressure, and quickly saturates the local network — resulting in a high rate of environment startup errors. Kubernetes struggled at high concurrency on a single host as well: it couldn't set up and tear down pods fast enough, triggering timeouts. The problem was worst on powerful hosts: they could support enormous concurrency, but pod scheduling couldn't keep up, so we never reached full CPU or RAM utilization.

Overlay sandboxes

Our solution is the Overlay Sandbox: instead of a container image, the agent runs on the bare host, in an isolated filesystem and process group. It has three layers: a base filesystem with the OS and repository, a hidden layer with the verifier suite, and an upper layer holding all edits and pip-installed packages. Because overlay mounts are lazy (they don't even index the filesystem!) and only track modified files, sandbox startup is essentially free.

Since SWE-rebench-V2 is built from public repositories, agents must run without network access — otherwise, a model can simply fetch a real fix from GitHub. Before restricting network, we caught Fable 5 and Sonnet 5 doing exactly this — retrieving the upstream patches directly on 20-40% of tasks.

Traffic recorder

Our sandbox also includes a traffic recorder — a tiny proxy that forwards requests to the provider (OpenRouter / vLLM / SGLang) and records every response. It's a harness-agnostic way to assemble full trajectories in OpenAI format and collect performance metrics.

Evaluation

Every agent runs with max_turns = 200, medium reasoning effort, and default sampling parameters. Inference costs are calculated from OpenRouter pricing tables. We report the theoretical price under the assumption of perfect caching — actual OpenRouter costs were higher due to inconsistent routing. For locally served GLM-4.7-Flash-Coder, we applied the per-request pricing of GLM-4.7-Flash via OpenRouter's DeepInfra provider.

Prompt template

You are solving a real GitHub issue in the `{repo}` repository. The repo is already
cloned and set up in the current working directory.

## Workflow

1. Understand the issue. Read the problem statement carefully. Identify the
   expected vs actual behavior.
2. Explore the codebase. Find the relevant files using grep, find, or by reading
   the project structure. Read the code that needs to change.
3. Reproduce the bug if possible — run the relevant test file or write a small
   script to confirm the issue exists.
4. Implement a minimal fix. Change only what is necessary. Do not refactor
   unrelated code, do not add features beyond what the issue asks for.
5. Verify your fix. Run the project's existing test suite to make sure you
   haven't introduced regressions. Look at the project's test configuration
   (Makefile, tox.ini, pyproject.toml, CI config) to find the right test command.
6. Mark the task complete only after you've verified existing tests pass with
   your changes.

## Issue

{problem_statement}

## Expected Interfaces

The following interfaces are expected by the test suite. Pay close attention to
function names, parameter signatures, and return types — the tests verify these
exactly.

{interface}