GRPO

Group Relative Policy Optimization scores a prompt's completions against each other, not against a learned value model. For prompt x with completions {y1..yk} and rewards {r1..rk}, the advantage is, in its simplest form, A(yi) = normalize(ri - mean(r)) and the loss L = -E[A(yi) * log pi(yi|x)]; each method's page gives the advantage rule and the loss it runs.

Offline GRPOOnline GRPO (RLVR)Async GRPO with Environments
TrainerOfflineGRPOTrainerDistributedOnlineGRPOTrainerDistributedAsyncEnvironmentalGRPOTrainer
Scriptgrpo/offline.pygrpo/online.pygrpo/environmental.py
GenerationNone, pre-collectedOnline, vLLM onlyOnline, vLLM or SGLang (rollout_backend), driven by Ray actors
RewardsPre-scored in the datasetrewards: terms: accuracy, format and the external termsrewards: terms: the environment grade plus the external terms
TurnsSingleSingleMulti-turn (single-turn environments exist)
ConfigOfflineGRPOConfigTRL GRPOConfigTRL GRPOConfig + EnvironmentConfig + AsyncRolloutConfig
InfrastructureTraining onlyTraining + one vLLM serverTraining + rollout servers + Ray
CPFull fine-tuning (CP, EP+CP)NoNo
PPFull fine-tuning (PP, PP+EP, PP+ETP)NoNo
Best forLarge pre-scored datasets, no generation costSingle-turn verifiable answers: math, structured outputTool use, code generation, environment feedback

All three extend DistributedTrainerMixin and run under EP, TP, pure ETP, EP+TP and EP+ETP. Online GRPO runs only against a vLLM server (Server mode only).

Launch

Offline GRPO needs no server (Launch). Online GRPO and Async GRPO generate against servers on GPUs the trainer does not use; by compose default vLLM takes GPU 7 and the trainer the other seven (Online launch, Async quickstart).

Pages

Pairwise preference alternatives: SMPO · DPO. All methods: Training Methods Overview.

Source on GitHubUpdated 18 November 2026