What is RLHF?

RLHF (reinforcement learning from human feedback) tunes a model on people's judgments of its answers. A reward model learns to predict which answer people prefer, and the model is then trained to score higher on it.

How it works

The classic pipeline has three stages:

  1. Supervised fine-tuning. The model learns the task from example answers.
  2. Reward model. People rank the model's answers to a prompt. A reward model learns from those rankings to score any answer.
  3. Reinforcement learning. The model generates answers, the reward model scores them, and PPO (Proximal Policy Optimization) updates the model toward higher scores. A penalty keeps it close to the stage 1 model.

The InstructGPT paper describes this recipe.

Why it matters

Comparing two answers is easier than writing the ideal one, so RLHF can teach qualities that are hard to specify, such as helpfulness and tone. The costs are human labeling, a training loop that is hard to tune, and a model that can learn to exploit the reward model's mistakes.

How DPO and GRPO relate

  • DPO (Direct Preference Optimization) optimizes the same objective directly from preference pairs, with no reward model and no sampling.
  • GRPO (Group Relative Policy Optimization) can take PPO's place in stage 3. It drops the value network and compares each answer with the others sampled for its prompt.

In Halo

halo launch rewards trains a Bradley-Terry reward model on preference pairs. Online GRPO can use the served model as a reward_model reward term. See reward modeling.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL