# What is GRPO?

GRPO (Group Relative Policy Optimization) is a reinforcement learning method for language models. It samples a group of completions for each prompt, scores them, and reinforces the ones that beat their group's average.

## How it works

For each prompt the model samples a group of completions. A grader scores each one: a rule, a test suite, a judge model or a [reward model](/glossary/reward-model).

Each completion's advantage is its reward minus the group's mean, usually divided by the group's standard deviation. The update raises the probability of completions with a positive advantage and lowers it for those with a negative one. A group whose completions all score the same gives no signal.

## How it differs from PPO

[PPO](/glossary/ppo) (Proximal Policy Optimization) trains a second network, the value network, to estimate the baseline that rewards are compared with. GRPO takes the baseline from the group, so there is no value network to train or keep in memory. It keeps PPO's clipped update. GRPO was introduced in the DeepSeekMath paper in 2024.

## In Halo

Halo runs GRPO in three variants:

| Variant | Trains on | `halo launch` |
| --- | --- | --- |
| [Offline GRPO](/glossary/offline-grpo) | prompts with completions you already scored | `offline-grpo` |
| [Online GRPO](/glossary/online-grpo) | prompts with a verifiable answer, generated live | `rlvr` |
| [Async GRPO](/glossary/async-grpo) | multi-turn, tool-using trajectories | `environmental-grpo` |

In online and async GRPO the group size is `num_generations`. In offline GRPO a group is one dataset row. [Choosing a method](/halo/docs/choosing-a-method) compares the three.

