# Reward Modeling

A reward model scores a (prompt, completion) pair with one scalar, fit to pairwise preferences under the Bradley-Terry loss. Use it for rejection sampling or as an RL reward signal; to train a policy on the same pairs use [DPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/preference/dpo.md) or [SMPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/preference/smpo.md).

Trainer `DistributedRewardTrainer`, script `scripts/training/reward/reward_model.py`, model class `AutoModelForSequenceClassification` with `num_labels=1`. EP, TP, ETP, EP+TP and PP apply; CP does not — the score head pools the whole sequence ([matrix](https://whitecircle.com/halo/docs/agent-docs/reference/trainer-architecture.md#trainer-compatibility)).

## Dataset

The pairwise `prompt` / `chosen` / `rejected` format DPO and SMPO use, plus implicit-prompt datasets carrying no `prompt` column, with the shared turns repeated inside both sides (`Skywork/Skywork-Reward-Preference-80K-v0.2`). See [Dataset Formats](https://whitecircle.com/halo/docs/agent-docs/data/dataset-formats.md#reward-modeling).

TRL's `RewardTrainer` chat-templates and tokenizes the raw columns itself, concatenating `prompt + chosen` / `prompt + rejected` with no hub-shape normalization: a `prompt` column whose completions repeat those turns renders the prompt twice, silently. DPO and SMPO strip that prefix, and so does the reward VLM path.

An optional `margin` column widens the target gap per row.

## Configuration

```yaml
model_name_or_path: Qwen/Qwen3.5-9B
dataset: Skywork/Skywork-Reward-Preference-80K-v0.2
test_size: 0.05
attn_implementation: sdpa          # the shipped recipe's pin; unset takes FA4 on Blackwell

max_length: 4096
per_device_train_batch_size: 2
gradient_accumulation_steps: 8
learning_rate: 1.0e-05             # the shipped recipes' value
num_train_epochs: 1
gradient_checkpointing: true
output_dir: checkpoints/rm-qwen3.5-9b-skywork-pref80k
```

`max_length` is a **filter**: TRL drops pairs whose chosen or rejected side exceeds it. `center_rewards_coefficient` penalizes `(chosen + rejected)²`, pulling scores toward zero mean. `tools_field` names a column of per-row tool schemas, aliased onto `tools` and rendered on conversational rows only; `images_field` is under [Vision-language](#vision-language). Types and defaults: [RewardConfig](https://whitecircle.com/halo/docs/agent-docs/reference/configuration-reference.md#trl-trainer-configs) and [RewardScriptArguments](https://whitecircle.com/halo/docs/agent-docs/reference/configuration-reference.md#other-script-arguments).

`max_length: null` resolves to the model's context window. `learning_rate` defaults to TRL's `1e-4`, high for a full fine-tune. A `chat_template_kwargs` column is splatted into the template too.

Left unpinned, `attn_implementation` takes the padded-batch default: the loader's own pick on Blackwell (SDPA for a remote-code checkpoint, FA2 for a softcapped policy), SDPA elsewhere, and nothing under `reset_sinks: false`, where only a sink-carrying backend is accepted ([padded workloads](https://whitecircle.com/halo/docs/agent-docs/optimization/flash-attention.md#model-specific-handling)). `text_only_model` and `log_decoded_samples` are refused, not silently ignored.

Where the tokenizer pads with eos (DeepSeek-V4), an INFO line names the cost: the recorded `config.pad_token_id` binds `nn.Embedding(padding_idx=<eos id>)` on the next load, masking that row's input-embedding gradient. Give such a base its own `pad_token:` to keep the EOS embedding training on the input side.

## Launch

```bash
# Dense, plain FSDP2 data parallel
torchrun --nproc_per_node=8 scripts/training/reward/reward_model.py \
    examples/reward/qwen3_5/rm-qwen3.5-9b-skywork-pref80k.yaml

# MoE, expert parallel
torchrun --nproc_per_node=8 scripts/training/reward/reward_model.py \
    examples/reward/gptoss/rm-gptoss-20b-skywork-pref80k-ep.yaml --expert_parallel_size=8
```

`halo launch reward/reward-model <config> --nproc 8` builds the same line.

## PEFT / LoRA

```yaml
use_peft: true
lora_r: 16
lora_alpha: 32
lora_target_modules: [q_proj, k_proj, v_proj, o_proj]
lora_task_type: SEQ_CLS         # required for a pooled score head
lora_modules_to_save: [score]
```

LoRA is rejected under TP, EP+TP and PP ([PEFT](https://whitecircle.com/halo/docs/agent-docs/optimization/peft.md#parallelism-compatibility)).

## Vision-language

Image-carrying pairs train on the families with both a vision tower and a sequence-classification head. transformers 5.20 ships one for `gemma3`, `qwen3_5` (dense), `t5gemma2` and `modernvbert`; the toolkit registers Gemma 4 and MoE Qwen3.5/3.6 in `halo/models/upstream/seq_cls_heads.py`. A checkpoint outside that live registry is refused before the model load, naming the supported list.

The data path follows the **run**, not the checkpoint: a vision run is one whose dataset carries an `images`/`image` column, or has `images_field` name one. A text recipe on a multimodal checkpoint stays on TRL's tokenize map.

```yaml
model_name_or_path: google/gemma-4-26B-A4B-it
dataset: path/to/vision_preference_dataset
images_field: images
max_length: 4096
```

Images merge into the **prompt** conversation, which both sides of a pair share: they fill unset `{"type": "image"}` placeholders in order, and otherwise lead the first user turn. Images inside a `chosen`/`rejected` completion are refused, as is `tools_field`. An `AutoProcessor` is the `processing_class` here, so every checkpoint exports `processor_config.json`.

`max_length` filters on rendered **text** only. Vision tokens are counted at collation, where a batch over budget raises. Lower the image resolution or raise `max_length`.

## Testing a setup

```bash
torchrun --nproc_per_node=2 scripts/training/reward/reward_model.py <config> \
    --max_steps=5 --save_strategy=no --report_to=none
```

Covering tests: `pytest tests/cpu/trainers -m cpu`, `tests/gpu/trainers/other/test_reward.py` and `test_reward_vlm_e2e.py`; under EP, TP and ETP with save, exact resume and export, `tests/gpu/parallelism/ep/test_ep_pooled_head_trainers.py --trainer reward`; under PP, `tests/gpu/parallelism/pp/test_pp_reward_bt.py` and `test_pp_trainer_checkpoints.py --trainer reward`.

Once trained, score fresh generations or build preference pairs with `scripts/inference/reward_model/score.py` and `rejection_sample.py` ([Scripts](https://whitecircle.com/halo/docs/agent-docs/reference/scripts-reference.md)). Both generate against a served endpoint from a `--dataset` prompt file before scoring; neither scores an existing (prompt, completion) corpus. Keep their `--rm_dtype` at `bfloat16`: reward logits are unbounded, and fp16 saturates on the completions the score must separate.

## What to watch

| Signal | Reading |
|---|---|
| `accuracy` | Share of pairs scoring chosen above rejected (chance is 0.5) |
| `margin` | Mean chosen − rejected score; the separation the model is buying |
| `min_reward` / `mean_reward` / `max_reward` | Score range; `center_rewards_coefficient` is what pulls `mean_reward` toward zero |

Failure signatures:

- Rows vanishing at prep — `max_length` filters rather than truncates, and the text path drops them silently (only the VLM path reports a count). Compare the split sizes, then raise it or shorten the pairs.
- Accuracy pinned near 0.5 under LoRA — the score head is freshly initialized, so it must be in `lora_modules_to_save`.
