GRPO
Group Relative Policy Optimization scores a prompt's completions against each other, not against a learned value model. For prompt x with completions {y1..yk} and rewards {r1..rk}, the advantage is, in its simplest form, A(yi) = normalize(ri - mean(r)) and the loss L = -E[A(yi) * log pi(yi|x)]; each method's page gives the advantage rule and the loss it runs.
All three extend DistributedTrainerMixin and run under EP, TP, pure ETP, EP+TP and EP+ETP. Online GRPO runs only against a vLLM server (Server mode only).
Launch
Offline GRPO needs no server (Launch). Online GRPO and Async GRPO generate against servers on GPUs the trainer does not use; by compose default vLLM takes GPU 7 and the trainer the other seven (Online launch, Async quickstart).
Pages
- Offline GRPO
- Online GRPO (RLVR)
- Reward Terms: the
rewards:list the online and environment arms read. - Async GRPO with Environments and its environments
Pairwise preference alternatives: SMPO · DPO. All methods: Training Methods Overview.
Source on GitHubUpdated 18 November 2026