What is GRPO?

GRPO (Group Relative Policy Optimization) is a reinforcement learning method for language models. It samples a group of completions for each prompt, scores them, and reinforces the ones that beat their group's average.

How it works

For each prompt the model samples a group of completions. A grader scores each one: a rule, a test suite, a judge model or a reward model.

Each completion's advantage is its reward minus the group's mean, usually divided by the group's standard deviation. The update raises the probability of completions with a positive advantage and lowers it for those with a negative one. A group whose completions all score the same gives no signal.

How it differs from PPO

PPO (Proximal Policy Optimization) trains a second network, the value network, to estimate the baseline that rewards are compared with. GRPO takes the baseline from the group, so there is no value network to train or keep in memory. It keeps PPO's clipped update. GRPO was introduced in the DeepSeekMath paper in 2024.

In Halo

Halo runs GRPO in three variants:

VariantTrains onhalo launch
Offline GRPOprompts with completions you already scoredoffline-grpo
Online GRPOprompts with a verifiable answer, generated liverlvr
Async GRPOmulti-turn, tool-using trajectoriesenvironmental-grpo

In online and async GRPO the group size is num_generations. In offline GRPO a group is one dataset row. Choosing a method compares the three.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL