Reward Modeling

A reward model scores a (prompt, completion) pair with one scalar, fit to pairwise preferences under the Bradley-Terry loss. Use it for rejection sampling or as an RL reward signal; to train a policy on the same pairs use DPO or SMPO.

Trainer DistributedRewardTrainer, script scripts/training/reward/reward_model.py, model class AutoModelForSequenceClassification with num_labels=1. EP, TP, ETP, EP+TP and PP apply; CP does not — the score head pools the whole sequence (matrix).

Dataset

The pairwise prompt / chosen / rejected format DPO and SMPO use, plus implicit-prompt datasets carrying no prompt column, with the shared turns repeated inside both sides (Skywork/Skywork-Reward-Preference-80K-v0.2). See Dataset Formats.

TRL's RewardTrainer chat-templates and tokenizes the raw columns itself, concatenating prompt + chosen / prompt + rejected with no hub-shape normalization: a prompt column whose completions repeat those turns renders the prompt twice, silently. DPO and SMPO strip that prefix, and so does the reward VLM path.

An optional margin column widens the target gap per row.

Configuration

YAML

model_name_or_path: Qwen/Qwen3.5-9B
dataset: Skywork/Skywork-Reward-Preference-80K-v0.2
test_size: 0.05
attn_implementation: sdpa          # the shipped recipe's pin; unset takes FA4 on Blackwell

max_length: 4096
per_device_train_batch_size: 2
gradient_accumulation_steps: 8
learning_rate: 1.0e-05             # the shipped recipes' value
num_train_epochs: 1
gradient_checkpointing: true
output_dir: checkpoints/rm-qwen3.5-9b-skywork-pref80k

max_length is a filter: TRL drops pairs whose chosen or rejected side exceeds it. center_rewards_coefficient penalizes (chosen + rejected)², pulling scores toward zero mean. tools_field names a column of per-row tool schemas, aliased onto tools and rendered on conversational rows only; images_field is under Vision-language. Types and defaults: RewardConfig and RewardScriptArguments.

max_length: null resolves to the model's context window. learning_rate defaults to TRL's 1e-4, high for a full fine-tune. A chat_template_kwargs column is splatted into the template too.

Left unpinned, attn_implementation takes the padded-batch default: the loader's own pick on Blackwell (SDPA for a remote-code checkpoint, FA2 for a softcapped policy), SDPA elsewhere, and nothing under reset_sinks: false, where only a sink-carrying backend is accepted (padded workloads). text_only_model and log_decoded_samples are refused, not silently ignored.

Where the tokenizer pads with eos (DeepSeek-V4), an INFO line names the cost: the recorded config.pad_token_id binds nn.Embedding(padding_idx=<eos id>) on the next load, masking that row's input-embedding gradient. Give such a base its own pad_token: to keep the EOS embedding training on the input side.

Launch

Bash

# Dense, plain FSDP2 data parallel
torchrun --nproc_per_node=8 scripts/training/reward/reward_model.py \
    examples/reward/qwen3_5/rm-qwen3.5-9b-skywork-pref80k.yaml

# MoE, expert parallel
torchrun --nproc_per_node=8 scripts/training/reward/reward_model.py \
    examples/reward/gptoss/rm-gptoss-20b-skywork-pref80k-ep.yaml --expert_parallel_size=8

halo launch reward/reward-model <config> --nproc 8 builds the same line.

PEFT / LoRA

YAML

use_peft: true
lora_r: 16
lora_alpha: 32
lora_target_modules: [q_proj, k_proj, v_proj, o_proj]
lora_task_type: SEQ_CLS         # required for a pooled score head
lora_modules_to_save: [score]

LoRA is rejected under TP, EP+TP and PP (PEFT).

Vision-language

Image-carrying pairs train on the families with both a vision tower and a sequence-classification head. transformers 5.20 ships one for gemma3, qwen3_5 (dense), t5gemma2 and modernvbert; the toolkit registers Gemma 4 and MoE Qwen3.5/3.6 in halo/models/upstream/seq_cls_heads.py. A checkpoint outside that live registry is refused before the model load, naming the supported list.

The data path follows the run, not the checkpoint: a vision run is one whose dataset carries an images/image column, or has images_field name one. A text recipe on a multimodal checkpoint stays on TRL's tokenize map.

YAML

model_name_or_path: google/gemma-4-26B-A4B-it
dataset: path/to/vision_preference_dataset
images_field: images
max_length: 4096

Images merge into the prompt conversation, which both sides of a pair share: they fill unset {"type": "image"} placeholders in order, and otherwise lead the first user turn. Images inside a chosen/rejected completion are refused, as is tools_field. An AutoProcessor is the processing_class here, so every checkpoint exports processor_config.json.

max_length filters on rendered text only. Vision tokens are counted at collation, where a batch over budget raises. Lower the image resolution or raise max_length.

Testing a setup

Bash

torchrun --nproc_per_node=2 scripts/training/reward/reward_model.py <config> \
    --max_steps=5 --save_strategy=no --report_to=none

Covering tests: pytest tests/cpu/trainers -m cpu, tests/gpu/trainers/other/test_reward.py and test_reward_vlm_e2e.py; under EP, TP and ETP with save, exact resume and export, tests/gpu/parallelism/ep/test_ep_pooled_head_trainers.py --trainer reward; under PP, tests/gpu/parallelism/pp/test_pp_reward_bt.py and test_pp_trainer_checkpoints.py --trainer reward.

Once trained, score fresh generations or build preference pairs with scripts/inference/reward_model/score.py and rejection_sample.py (Scripts). Both generate against a served endpoint from a --dataset prompt file before scoring; neither scores an existing (prompt, completion) corpus. Keep their --rm_dtype at bfloat16: reward logits are unbounded, and fp16 saturates on the completions the score must separate.

What to watch

SignalReading
accuracyShare of pairs scoring chosen above rejected (chance is 0.5)
marginMean chosen − rejected score; the separation the model is buying
min_reward / mean_reward / max_rewardScore range; center_rewards_coefficient is what pulls mean_reward toward zero

Failure signatures:

  • Rows vanishing at prep — max_length filters rather than truncates, and the text path drops them silently (only the VLM path reports a count). Compare the split sizes, then raise it or shorten the pairs.
  • Accuracy pinned near 0.5 under LoRA — the score head is freshly initialized, so it must be in lora_modules_to_save.

Source on GitHubUpdated 18 November 2026