# What is PPO?

PPO (Proximal Policy Optimization) is a reinforcement learning algorithm that updates a policy in small, clipped steps, with a value network as its baseline. It is the classic optimizer for RLHF (reinforcement learning from human feedback).

## How it works

The policy acts, and each action gets an advantage: how much better it turned out than expected. The expectation comes from the value network, a second network trained alongside the policy.

PPO raises the probability of actions with a positive advantage, within a limit. It tracks the ratio between the new and the old probability of each action, and the objective stops rewarding a change once that ratio leaves a narrow band around 1. The policy stays close to the one that generated the data, so one batch of [rollouts](/glossary/rollouts) can serve several gradient steps. OpenAI published PPO in 2017.

## PPO and language models

In [RLHF](/glossary/rlhf) the policy is the language model, an action is a token, and the reward comes from a [reward model](/glossary/reward-model) trained on human preferences. The cost of PPO there is the value network: more weights to train and keep in memory.

## How GRPO differs

[GRPO](/glossary/grpo) (Group Relative Policy Optimization) keeps the clip and drops the value network. Its baseline is the mean reward of a group of completions sampled for the same prompt.

## In Halo

Halo's reinforcement learning methods are GRPO variants. In online GRPO the clip is the `epsilon` key, which binds only when a generation batch outlives one optimizer step. See the [online GRPO guide](/halo/docs/training-methods/online-grpo).

