GRPO (Group Relative Policy Optimization) is a reinforcement learning method for language models. It samples a group of completions for each prompt, scores them, and reinforces the ones that beat their group's average.
How it works
For each prompt the model samples a group of completions. A grader scores each one: a rule, a test suite, a judge model or a reward model.
Each completion's advantage is its reward minus the group's mean, usually divided by the group's standard deviation. The update raises the probability of completions with a positive advantage and lowers it for those with a negative one. A group whose completions all score the same gives no signal.
How it differs from PPO
PPO (Proximal Policy Optimization) trains a second network, the value network, to estimate the baseline that rewards are compared with. GRPO takes the baseline from the group, so there is no value network to train or keep in memory. It keeps PPO's clipped update. GRPO was introduced in the DeepSeekMath paper in 2024.
In Halo
Halo runs GRPO in three variants:
In online and async GRPO the group size is num_generations. In offline GRPO a group is one dataset row. Choosing a method compares the three.
