# What is RLHF?

RLHF (reinforcement learning from human feedback) tunes a model on people's judgments of its answers. A reward model learns to predict which answer people prefer, and the model is then trained to score higher on it.

## How it works

The classic pipeline has three stages:

1. **Supervised fine-tuning.** The model learns the task from example answers.
2. **Reward model.** People rank the model's answers to a prompt. A [reward model](/glossary/reward-model) learns from those rankings to score any answer.
3. **Reinforcement learning.** The model generates answers, the reward model scores them, and [PPO](/glossary/ppo) (Proximal Policy Optimization) updates the model toward higher scores. A penalty keeps it close to the stage 1 model.

The [InstructGPT paper](https://arxiv.org/abs/2203.02155) describes this recipe.

## Why it matters

Comparing two answers is easier than writing the ideal one, so RLHF can teach qualities that are hard to specify, such as helpfulness and tone. The costs are human labeling, a training loop that is hard to tune, and a model that can learn to exploit the reward model's mistakes.

## How DPO and GRPO relate

- [DPO](/glossary/dpo) (Direct Preference Optimization) optimizes the same objective directly from preference pairs, with no reward model and no sampling.
- [GRPO](/glossary/grpo) (Group Relative Policy Optimization) can take PPO's place in stage 3. It drops the value network and compares each answer with the others sampled for its prompt.

## In Halo

`halo launch rewards` trains a Bradley-Terry reward model on preference pairs. Online GRPO can use the served model as a `reward_model` reward term. See [reward modeling](/halo/docs/training-methods/reward-and-classification#reward-modeling).

