Online GRPO (RLVR)

Online GRPO is reinforcement learning with verifiable rewards (RLVR). During training the model generates several completions for each prompt, a grader checks each one, and the scores inside each group drive the update.

How it works

Each training step is one cycle:

  1. The current model generates several completions for each prompt.
  2. A grader scores each one. The reward is verifiable: a rule checks it, such as a final answer equal to the ground truth or an output in a required format.
  3. The rewards inside each group become advantages, as in any GRPO (Group Relative Policy Optimization) run.
  4. The model takes an optimizer step.

The next round of rollouts comes from the new weights, which makes the method on-policy.

When to use it

It fits tasks with one prompt, one answer and one turn: math, structured output, anything a grader can check. Multi-turn tool use belongs to async GRPO, and data that is already scored to offline GRPO.

In Halo

The launch name is rlvr. Generation runs on a separate vLLM container, the rollout server, which receives the updated weights before each round and must own GPUs no trainer process uses. Start the server, wait for /health, then launch the trainer on the other GPUs:

Bash

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6 halo launch rlvr \
    examples/grpo/online/rlvr-online-grpo-template.yaml -n 7

The data has two columns, prompt and answer. num_generations is the group size: below about 4 a group's rewards are too often all equal, and an all-equal group produces no gradient. See the online GRPO guide.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL