Reward model

Reward models read a prompt and an answer and return one number for how good the answer is. They are trained on preference pairs so that the chosen answer scores above the rejected one.

How it works

A reward model is usually a language model with a small head that outputs one score, not text. Each training row is a prompt with a chosen and a rejected answer. The Bradley-Terry objective treats the gap between the two scores as the log-odds that the chosen answer wins, and training widens that gap.

The score is unbounded and only means something next to another answer's score.

What it is for

  • Reinforcement learning. The score is the reward a model learns to raise, as in RLHF (reinforcement learning from human feedback) or GRPO (Group Relative Policy Optimization).
  • Ranking. Generate several answers and keep the best.
  • Filtering. Drop low-scoring rows from a dataset.

A model trained against a reward model can learn to exploit its blind spots, so read outputs, not only scores. To change the model itself from the same pairs, with no scorer, use preference tuning.

In Halo

Reward modeling reads pairs in prompt, chosen and rejected columns:

Bash

halo launch rewards examples/reward/qwen3_5/rm-qwen3.5-9b-skywork-pref80k.yaml -n 8

accuracy is the share of pairs ranked correctly, and 0.5 is chance. Once trained, halo run rm-scoring and halo run rm-rejection-sampling generate against a served endpoint and score the results. See the reward modeling guide.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL