# GRPO

Group Relative Policy Optimization scores a prompt's completions against each other, not against a learned value model. For prompt `x` with completions `{y1..yk}` and rewards `{r1..rk}`, the advantage is, in its simplest form, `A(yi) = normalize(ri - mean(r))` and the loss `L = -E[A(yi) * log pi(yi|x)]`; each method's page gives the advantage rule and the loss it runs.

| | Offline GRPO | Online GRPO (RLVR) | Async GRPO with Environments |
|---|---|---|---|
| Trainer | `OfflineGRPOTrainer` | `DistributedOnlineGRPOTrainer` | `DistributedAsyncEnvironmentalGRPOTrainer` |
| Script | `grpo/offline.py` | `grpo/online.py` | `grpo/environmental.py` |
| Generation | None, pre-collected | Online, vLLM only | Online, vLLM or SGLang (`rollout_backend`), driven by Ray actors |
| Rewards | Pre-scored in the dataset | [`rewards:` terms](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/rewards.md): `accuracy`, `format` and the [external terms](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/rewards.md#common-options) | [`rewards:` terms](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/rewards.md): the `environment` grade plus the [external terms](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/rewards.md#common-options) |
| Turns | Single | Single | Multi-turn (single-turn environments exist) |
| Config | `OfflineGRPOConfig` | TRL `GRPOConfig` | TRL `GRPOConfig` + `EnvironmentConfig` + `AsyncRolloutConfig` |
| Infrastructure | Training only | Training + one vLLM server | Training + rollout servers + Ray |
| CP | Full fine-tuning (CP, EP+CP) | No | No |
| PP | Full fine-tuning (PP, PP+EP, PP+ETP) | No | No |
| Best for | Large pre-scored datasets, no generation cost | Single-turn verifiable answers: math, structured output | Tool use, code generation, environment feedback |

All three extend `DistributedTrainerMixin` and run under EP, TP, pure ETP, EP+TP and EP+ETP. Online GRPO runs only against a vLLM server ([Server mode only](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/online.md#server-mode-only)).

## Launch

Offline GRPO needs no server ([Launch](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/offline.md#launch)). Online GRPO and Async GRPO generate against servers on GPUs the trainer does not use; by compose default vLLM takes GPU 7 and the trainer the other seven ([Online launch](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/online.md#launch), [Async quickstart](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo.md#quickstart)).

## Pages

- [Offline GRPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/offline.md)
- [Online GRPO (RLVR)](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/online.md)
- [Reward Terms](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/rewards.md): the `rewards:` list the online and environment arms read.
- [Async GRPO with Environments](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo.md) and its [environments](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/environments.md)

Pairwise preference alternatives: [SMPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/preference/smpo.md) · [DPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/preference/dpo.md). All methods: [Training Methods Overview](https://whitecircle.com/halo/docs/agent-docs/training-methods.md).
