Offline GRPO is reinforcement learning on completions that were generated and scored before training. Each prompt comes with several scored completions, and the model generates nothing during the run.
How it works
The dataset holds a group of completions for each prompt and one reward per completion. As in any GRPO (Group Relative Policy Optimization) run, each reward is compared with the rest of its group to get an advantage. Training raises the probability of completions with a positive advantage and lowers it for those with a negative one.
Nothing is generated or scored during training, so no inference server runs beside the trainer and the dataset can be re-run.
When to use it
Use it when scoring is expensive, already done, or must stay reproducible. The limit is that the data came from an earlier policy: as the model moves away from it, the rewards say less about what the model now writes. When the model should generate as it learns, use online GRPO or async GRPO.
In Halo
One dataset row is one group, with three columns:
JSON Lines
{"prompt": [{"role": "user", "content": "What is 2+2?"}], "completions": [[{"role": "assistant", "content": "4"}], [{"role": "assistant", "content": "5"}]], "rewards": [1.0, -0.5]}One command starts the run:
Bash
halo launch offline-grpo examples/grpo/offline/qwen3_5/offline-grpo-qwen3.6-35b-a3b-gsm8k.yaml -n 8Two rows that share a prompt are two groups, and a group of one contributes no gradient. Without a scored dataset, halo run rm-rejection-sampling --output_format offline_grpo generates candidates against a served model and scores them with a reward model. See the offline GRPO guide.
