Reward Terms

A GRPO reward is a sum of terms. Each term names a source of a score in [0, 1] and prices it as weight × score ^ exponent. Terms are listed under rewards: and parsed at config time (halo/rewards/terms.py), so a bad term fails before any server is contacted.

The online arm turns each term into one TRL reward function; the environment arm composes them into the episode reward.

YAML

rewards:
  - source: accuracy                 # online arm only
  - source: judge
    name: quality
    weight: 0.3
    requirements:
      - {name: correctness, description: "The final answer is correct and fully justified."}
      - {name: clarity, description: "The solution is easy to follow.", weight: 0.5}
  - source: reward_model
    name: preference
    weight: 0.5
    url: http://localhost:8100
    model: Skywork/Skywork-Reward-V2-Llama-3.1-8B
FieldDefaultEffect
source—Where the score comes from (table below)
nameRequired for the external terms (judge, reward_model); defaulted otherwiseThe metric key: reward/<name> on the environment arm, rewards/<name>/mean on the online arm. Unique. The environment term is always objective; any other name raises
weight1.0 (0 for a veto judge)Signed; negative makes a penalty
exponent1.0> 0. Above 1 is convex credit (half a score earns under half the weight), below 1 concave

There is no constant-offset knob, failure offset included: a constant cancels in the group baseline.

sourceScoreArm
environmentthe environment's own gradeenvironments
judgea generative judge: a score over requirements, or a veto over checksboth; a veto judge environments only
reward_modela served Bradley-Terry or sequence-classification modelboth
accuracy, formatthe RLVR gradersonline

Common options

The external sources (judge, reward_model) score the finished completion and take:

OptionDefaultEffect
viewfinalWhat the scorer reads: final (the final answer), full (every turn) or digest (a compact account of every turn; judge only)
on_errorinvalid; neutral for a veto judgeWhat a missing verdict means: invalid voids the completion's verdict, neutral prices the term 0 (Outages)

Outages

A scorer never raises for a sample. A failed request, an unparseable reply or an unbuildable prompt returns no verdict, and <source>/<name>/scored logs 0 for that completion.

on_errorOnline arm (TRL row)Environment arm (episode)
invalid (default)The term returns None, summed as 0. The row leaves the baseline only when every term returned NoneThe term prices 0 and the episode is episode_invalid, out of the group baseline, with the scorer's error as episode_invalid_reason. The eval runner reports an error row
neutral (veto judge default)The term returns 0The term prices 0; the episode stays valid

On the environment arm, a second step in a row with no valid episode on any rank stops the run (Batch construction).

Two configuration faults raise instead of booking an outage: a reward model whose tokenizer does not load, and, on the online arm, a term whose dataset column TRL did not forward (Rewards).

Probe and retries

At launch, rank 0 probes every external term with a one-line sample in the run's exact request shape, so a bad URL, key, model or parameter fails the launch, not every episode. The environment arm probes through the environment's verify_backend.

Judge requests retry (halo/inference/openai_client.py):

  • up to four times, with request_timeout bounding each attempt, not the whole call;
  • four more times, at 2, 4, 8 and 16 s, on a reply with no choices that carries a 408, 429, 500, 502, 503 or 504 error code (how OpenRouter reports a rate-limited model).

One stuck call can hold a step for minutes; lower the timeout first.

Environment arm

The environment grades; the terms price. _grade_episode(trajectory, context) returns an EpisodeGrade: objective (the score in [0, 1]) and shaping (the environment's own episode-level terms, by bare name). Writing one: Custom Environments.

The episode reward is the sum of its reward/* components:

  • reward/turn_shaping: per-turn deltas (tool credit and penalties, ReAct thought credit).
  • reward/tool_shaping: native-protocol episode knobs (no_tool_use_penalty, turn_overflow_penalty, length_cutoff_penalty).
  • reward/<name>: each environment shaping term (code contests: submission, resubmission) and each external term.
  • reward/objective: weight × grade ^ exponent from the environment term.

The trainer adds its length charges, reward/reasoning_price and reward/reasoning_floor (each logged while its term is on), outside these components (Reasoning length reward). Components and rationales are saved with each completion (Saved trajectories).

  • rewards defaults to [{source: environment}] and reaches the environment constructor as reward_terms.
  • turn_shaping and the class's SHAPING_COMPONENTS names (unioned over the hierarchy) are reserved; no reward term may take one.
  • A grade with no signal (a null answer, a grader outage, a code grade that stopped before any test failed) is 0 and marks the episode episode_invalid.

What a scorer reads

One sample serves every external term, each through its own view. It carries:

  • the prompt turns (everything before the first assistant turn) and the tool schemas;
  • the policy's turns, with reasoning and the capped, cut, empty or rejected flags;
  • the calls a cut turn never ran (Message.cut_tool_calls), which only the judge's full and digest views show;
  • the final answer and the reference.

Three EnvironmentBase hooks fill the episode-specific parts:

  • _final_answer(trajectory): what the episode delivered (the native protocol's final_response, ReAct's Final Answer, code contests' submitted program as a fenced code block), or None (the base) when it delivered none.
  • _scoring_reference(trajectory): the reference, context["answer"] by default. exam_qa hands the choice letter; code contests hands None, so the hidden tests never reach a judge.
  • _episode_facts(trajectory): the episode facts the environment decides itself (declared in EPISODE_FACTS), which a judge check names in unless to stand exempt under; none in the base. Code contests establishes failed_with_no_runs_left.

Settlement

External terms are scored after the episode ends. At the terminal step the environment prices its own side and marks the reward pending. The episode dispatcher (Ray actor and eval runner alike) then awaits settle_async; a synchronous caller uses env.settle(ids). Settlement prices each term, applies a fired veto and books a missing verdict by its on_error.

  • Reading rollout_metrics on an unsettled episode raises.
  • episode/reward_scored is 1 only when every external term reached a verdict; a miss settled neutral still logs 0.
  • Judge concurrency is max_concurrency per Ray actor (one scorer per environment instance).

Some episodes never reach a scorer:

  • A generation that failed past its retries. The training actor returns an error row at reward 0. The eval driver grades what the episode earned, but reports a generation error instead unless its own request caused the failure.
  • A sandbox fault. The episode logs no episode/reward_scored.

Failure and cost

  • Under TP or ETP every rank of a group scores its copy and the leader's result is broadcast, so judge calls scale with tp_size or expert_tp_size.
  • Judge cost is per graded completion and per eval sample; watch the judge's completion-token and cost metrics.

Pages

Source on GitHubUpdated 18 November 2026