Reward Terms
A GRPO reward is a sum of terms. Each term names a source of a score in [0, 1] and prices it as weight × score ^ exponent. Terms are listed under rewards: and parsed at config time (halo/rewards/terms.py), so a bad term fails before any server is contacted.
The online arm turns each term into one TRL reward function; the environment arm composes them into the episode reward.
YAML
rewards:
- source: accuracy # online arm only
- source: judge
name: quality
weight: 0.3
requirements:
- {name: correctness, description: "The final answer is correct and fully justified."}
- {name: clarity, description: "The solution is easy to follow.", weight: 0.5}
- source: reward_model
name: preference
weight: 0.5
url: http://localhost:8100
model: Skywork/Skywork-Reward-V2-Llama-3.1-8BThere is no constant-offset knob, failure offset included: a constant cancels in the group baseline.
Common options
The external sources (judge, reward_model) score the finished completion and take:
Outages
A scorer never raises for a sample. A failed request, an unparseable reply or an unbuildable prompt returns no verdict, and <source>/<name>/scored logs 0 for that completion.
On the environment arm, a second step in a row with no valid episode on any rank stops the run (Batch construction).
Two configuration faults raise instead of booking an outage: a reward model whose tokenizer does not load, and, on the online arm, a term whose dataset column TRL did not forward (Rewards).
Probe and retries
At launch, rank 0 probes every external term with a one-line sample in the run's exact request shape, so a bad URL, key, model or parameter fails the launch, not every episode. The environment arm probes through the environment's verify_backend.
Judge requests retry (halo/inference/openai_client.py):
- up to four times, with
request_timeoutbounding each attempt, not the whole call; - four more times, at 2, 4, 8 and 16 s, on a reply with no choices that carries a 408, 429, 500, 502, 503 or 504 error code (how OpenRouter reports a rate-limited model).
One stuck call can hold a step for minutes; lower the timeout first.
Environment arm
The environment grades; the terms price. _grade_episode(trajectory, context) returns an EpisodeGrade: objective (the score in [0, 1]) and shaping (the environment's own episode-level terms, by bare name). Writing one: Custom Environments.
The episode reward is the sum of its reward/* components:
reward/turn_shaping: per-turn deltas (tool credit and penalties, ReAct thought credit).reward/tool_shaping: native-protocol episode knobs (no_tool_use_penalty,turn_overflow_penalty,length_cutoff_penalty).reward/<name>: each environment shaping term (code contests:submission,resubmission) and each external term.reward/objective:weight × grade ^ exponentfrom theenvironmentterm.
The trainer adds its length charges, reward/reasoning_price and reward/reasoning_floor (each logged while its term is on), outside these components (Reasoning length reward). Components and rationales are saved with each completion (Saved trajectories).
rewardsdefaults to[{source: environment}]and reaches the environment constructor asreward_terms.turn_shapingand the class'sSHAPING_COMPONENTSnames (unioned over the hierarchy) are reserved; no reward term may take one.- A grade with no signal (a null
answer, a grader outage, a code grade that stopped before any test failed) is 0 and marks the episodeepisode_invalid.
What a scorer reads
One sample serves every external term, each through its own view. It carries:
- the prompt turns (everything before the first assistant turn) and the tool schemas;
- the policy's turns, with reasoning and the capped, cut, empty or rejected flags;
- the calls a cut turn never ran (
Message.cut_tool_calls), which only the judge'sfullanddigestviews show; - the final answer and the reference.
Three EnvironmentBase hooks fill the episode-specific parts:
_final_answer(trajectory): what the episode delivered (the native protocol'sfinal_response, ReAct'sFinal Answer, code contests' submitted program as a fenced code block), orNone(the base) when it delivered none._scoring_reference(trajectory): the reference,context["answer"]by default.exam_qahands the choice letter; code contests handsNone, so the hidden tests never reach a judge._episode_facts(trajectory): the episode facts the environment decides itself (declared inEPISODE_FACTS), which a judge check names inunlessto stand exempt under; none in the base. Code contests establishesfailed_with_no_runs_left.
Settlement
External terms are scored after the episode ends. At the terminal step the environment prices its own side and marks the reward pending. The episode dispatcher (Ray actor and eval runner alike) then awaits settle_async; a synchronous caller uses env.settle(ids). Settlement prices each term, applies a fired veto and books a missing verdict by its on_error.
- Reading
rollout_metricson an unsettled episode raises. episode/reward_scoredis 1 only when every external term reached a verdict; a miss settledneutralstill logs 0.- Judge concurrency is
max_concurrencyper Ray actor (one scorer per environment instance).
Some episodes never reach a scorer:
- A generation that failed past its retries. The training actor returns an error row at reward 0. The eval driver grades what the episode earned, but reports a generation error instead unless its own request caused the failure.
- A sandbox fault. The episode logs no
episode/reward_scored.
Failure and cost
- Under TP or ETP every rank of a group scores its copy and the leader's result is broadcast, so judge calls scale with
tp_sizeorexpert_tp_size. - Judge cost is per graded completion and per eval sample; watch the judge's completion-token and cost metrics.
Pages
Source on GitHubUpdated 18 November 2026