Served Reward Model
A reward_model term scores with a *ForSequenceClassification or *ForRewardModel checkpoint served by the rollout engines' pooling routes, on both arms.
The scorer renders the conversation with the tokenizer's chat template, adds no special tokens, and maps the head's logit to sigmoid((logit − logit_shift) / logit_scale). On the online arm it renders the original conversation, not TRL's generation prompt.
Serving
TRL's in-process path stays reachable through the online trainer's API (not the YAML): a model id in reward_funcs loads a classification head on every rank, in trainer memory.
Options
The raw logit logs as reward_model/<name>/logit, the verdict rate as reward_model/<name>/scored.
Source on GitHubUpdated 18 November 2026