Online GRPO is reinforcement learning with verifiable rewards (RLVR). During training the model generates several completions for each prompt, a grader checks each one, and the scores inside each group drive the update.
How it works
Each training step is one cycle:
- The current model generates several completions for each prompt.
- A grader scores each one. The reward is verifiable: a rule checks it, such as a final answer equal to the ground truth or an output in a required format.
- The rewards inside each group become advantages, as in any GRPO (Group Relative Policy Optimization) run.
- The model takes an optimizer step.
The next round of rollouts comes from the new weights, which makes the method on-policy.
When to use it
It fits tasks with one prompt, one answer and one turn: math, structured output, anything a grader can check. Multi-turn tool use belongs to async GRPO, and data that is already scored to offline GRPO.
In Halo
The launch name is rlvr. Generation runs on a separate vLLM container, the rollout server, which receives the updated weights before each round and must own GPUs no trainer process uses. Start the server, wait for /health, then launch the trainer on the other GPUs:
Bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6 halo launch rlvr \
examples/grpo/online/rlvr-online-grpo-template.yaml -n 7The data has two columns, prompt and answer. num_generations is the group size: below about 4 a group's rewards are too often all equal, and an all-equal group produces no gradient. See the online GRPO guide.
