Async GRPO is reinforcement learning for multi-turn, tool-using tasks. The model acts in an environment until the episode ends, the environment grades the whole episode, and new episodes are generated while the trainer updates.
How it works
The model talks to an environment: it calls tools, runs code, searches or edits a workspace until the episode ends. The environment then grades the whole trajectory. GRPO (Group Relative Policy Optimization) trains on the group of episodes that shared a prompt. Each episode's advantage is its reward minus the group's baseline, and every assistant turn of the episode is trained with it.
Use it when the reward comes from what the model did. Single-turn tasks with a checkable answer are cheaper on online GRPO.
Why asynchronous
A multi-turn episode is slow to generate, because its turns run one after another. When generation and training take turns, each waits for the other. Async GRPO generates the next round of rollouts while the current update runs.
In Halo
The launch name is environmental-grpo. Rollouts run on a separate vLLM or SGLang container (a rollout server), driven by Ray actors that hold the environments and their tools. A shipped recipe sets, among other keys:
YAML
environment_type: react_math
max_turns: 10
num_rollout_workers: 16
num_generations: 8
rollout_max_tokens: 8192rollout_max_tokens caps one turn, not the trajectory. With one server a step is serial: sync, generate, train. With two or more, listed in rollout_server_configs, enable_prefetch: true overlaps the next round's generation with the current update. See Async GRPO with environments.
