PPO (Proximal Policy Optimization) is a reinforcement learning algorithm that updates a policy in small, clipped steps, with a value network as its baseline. It is the classic optimizer for RLHF (reinforcement learning from human feedback).
How it works
The policy acts, and each action gets an advantage: how much better it turned out than expected. The expectation comes from the value network, a second network trained alongside the policy.
PPO raises the probability of actions with a positive advantage, within a limit. It tracks the ratio between the new and the old probability of each action, and the objective stops rewarding a change once that ratio leaves a narrow band around 1. The policy stays close to the one that generated the data, so one batch of rollouts can serve several gradient steps. OpenAI published PPO in 2017.
PPO and language models
In RLHF the policy is the language model, an action is a token, and the reward comes from a reward model trained on human preferences. The cost of PPO there is the value network: more weights to train and keep in memory.
How GRPO differs
GRPO (Group Relative Policy Optimization) keeps the clip and drops the value network. Its baseline is the mean reward of a group of completions sampled for the same prompt.
In Halo
Halo's reinforcement learning methods are GRPO variants. In online GRPO the clip is the epsilon key, which binds only when a generation batch outlives one optimizer step. See the online GRPO guide.
