# Reward model

Reward models read a prompt and an answer and return one number for how good the answer is. They are trained on preference pairs so that the chosen answer scores above the rejected one.

## How it works

A reward model is usually a language model with a small head that outputs one score, not text. Each training row is a prompt with a chosen and a rejected answer. The Bradley-Terry objective treats the gap between the two scores as the log-odds that the chosen answer wins, and training widens that gap.

The score is unbounded and only means something next to another answer's score.

## What it is for

- **Reinforcement learning.** The score is the reward a model learns to raise, as in [RLHF](/glossary/rlhf) (reinforcement learning from human feedback) or [GRPO](/glossary/grpo) (Group Relative Policy Optimization).
- **Ranking.** Generate several answers and keep the best.
- **Filtering.** Drop low-scoring rows from a dataset.

A model trained against a reward model can learn to exploit its blind spots, so read outputs, not only scores. To change the model itself from the same pairs, with no scorer, use [preference tuning](/glossary/preference-tuning).

## In Halo

Reward modeling reads pairs in `prompt`, `chosen` and `rejected` columns:

```bash
halo launch rewards examples/reward/qwen3_5/rm-qwen3.5-9b-skywork-pref80k.yaml -n 8
```

`accuracy` is the share of pairs ranked correctly, and 0.5 is chance. Once trained, `halo run rm-scoring` and `halo run rm-rejection-sampling` generate against a served endpoint and score the results. See the [reward modeling guide](/halo/docs/training-methods/reward-and-classification#reward-modeling).

