# Async GRPO with environments

Async GRPO is reinforcement learning for multi-turn, tool-using tasks. The model acts in an environment until the episode ends, the environment grades the whole episode, and new episodes are generated while the trainer updates.

## How it works

The model talks to an [environment](/glossary/rl-environments): it calls tools, runs code, searches or edits a workspace until the episode ends. The environment then grades the whole trajectory. [GRPO](/glossary/grpo) (Group Relative Policy Optimization) trains on the group of episodes that shared a prompt. Each episode's advantage is its reward minus the group's baseline, and every assistant turn of the episode is trained with it.

Use it when the reward comes from what the model did. Single-turn tasks with a checkable answer are cheaper on [online GRPO](/glossary/online-grpo).

## Why asynchronous

A multi-turn episode is slow to generate, because its turns run one after another. When generation and training take turns, each waits for the other. Async GRPO generates the next round of [rollouts](/glossary/rollouts) while the current update runs.

## In Halo

The launch name is `environmental-grpo`. Rollouts run on a separate vLLM or SGLang container (a [rollout server](/glossary/rollout-server)), driven by Ray actors that hold the environments and their tools. A shipped recipe sets, among other keys:

```yaml
environment_type: react_math
max_turns: 10
num_rollout_workers: 16
num_generations: 8
rollout_max_tokens: 8192
```

`rollout_max_tokens` caps one turn, not the trajectory. With one server a step is serial: sync, generate, train. With two or more, listed in `rollout_server_configs`, `enable_prefetch: true` overlaps the next round's generation with the current update. See [Async GRPO with environments](/halo/docs/training-methods/async-grpo-environments).

