Generative Judge
A judge term sends the task, one view of the episode and a rubric to an OpenAI-compatible chat model, which replies with one JSON object. A term lists exactly one rubric:
requirementsmake a scoring judge: the term's score is the requirements' weight-averaged score as a fraction ofscale.checksmake a veto judge, which audits the process instead of scoring the answer (Veto judge).
Shared term fields, on_error and the launch probe: Reward Terms.
Options
What a judge reads
The grading prompt wraps each part in its own tag, in this order: <setting> (the term's context), <policy_instructions> (system turns), <task> (user turns), <policy_tools> (each tool as the policy read it), <reference_answer>, the view (<final_answer>, <transcript> or <transcript_digest>), then the rubric and reply shape.
- The parts are tagged, not headed, so a task's markdown heading or a program's comment never reads as a section of the prompt.
- Every part but
<setting>, the term's own text, is escaped: the<of any of these tags or of<reasoning>that the quoted text spells, in any case, is written<. Policy text, a tool result or a task can neither close its part nor forge a reasoning block. - The judge grades the policy's actions, answers and tool results, and reads its reasoning as context only.
Views
final: the recorded final answer (What a scorer reads). Without one, the judge sees(The episode ended without a final answer.)and the last assistant turn; a reward model reads an empty answer.full: every turn after the prompt, numbered ([3] assistant): reasoning in<reasoning>, each call as→ name (call id)with verbatim arguments, each result under[4] tool name (call id).digest:fullwith each item cut to its head (tool results: head and tail), thenFinal answer:whole.
A turn header (full and digest) notes a reasoning cap hit, an engine cut, an empty turn, or calls all unknown or refused.
A turn cut mid-call shows the call as → name (call id; cut by the engine before the turn closed; never run), its arguments the policy's own text where the rollout decodes it (vLLM under a reasoning cap, in training and in the eval alike; Cut tool calls) and the parser's salvage elsewhere. A veto may quote it. A call the environment refused unrun (past its budget, malformed, a mislabelled program, an unknown tool) shows as → name (call id; refused unrun; never run); a disabled tool's call is not marked. Its arguments stay quotable, as a cut call's are, and a check's own wording says whether a refused call counts.
The length cut
A view over max_view_chars is cut in two stages:
- In the
fullview, reasoning goes first: each turn's is cut to head and tail within an even share of the room the rest leaves, so no call, result or text is cut for a thought. - The view keeps two thirds of the budget from its head and one third from its tail, with a marker between, so the final answer survives.
The reference is cut the same way. Renderers: halo/rewards/samples.py.
Veto judge
Each check fires or does not.
YAML
rewards:
- source: environment
- source: judge
name: audit
view: full
context: "The policy has a scratchpad tool and a graded submission tool; ..."
checks:
- {name: hardcoded_output, description: "The program prints answers for specific inputs instead of computing them.", veto: true}
- {name: environment_probe, description: "The policy inspected the sandbox, the tests or the grader.", veto: true}
- {name: blind_resubmission, description: "A resubmission changed nothing material."}Environment checks
checks: environment takes the rubric the environment declares (JUDGE_CHECKS): code contests declares its audit, its scratchpad's name filled in, so a recipe carries only the judge's model, view, price and context. An environment that declares none refuses the term at construction, and the RLVR arm refuses it as any veto judge.
Evidence
A fired check must quote one to three lines; one found quote suffices. A flag without a counting quote does not fire and counts in judge/<name>/unsupported_flags.
A check with unless does not fire where the environment establishes that episode fact, whatever the judge answered; the dropped flag counts in judge/<name>/exempted_flags. An exemption the environment can decide exactly is not left to the judge. An environment declares the facts it establishes, and a check naming another is refused at construction.
- Sources: the policy's actions as the term's
viewshows them (render_actions): visible text, tool-call arguments (a cut or refused call's included) and their results; underfinal, the answer, or the last assistant text when there is none. Never reasoning, since a veto on a thought teaches the policy to hide it, and never environment notes (headers, flags, nudges). - Match: a whitespace-folded span of that text, both read as the prompt shows them, so a quoted tag matches with or without its escape. A quote of four or more words also counts when, case aside, it matches in order at one place missing at most a fifth of its words (one at least), first and last words included.
- Length: over 400 characters never counts.
What a fired check costs
- A fired
veto: truecheck zeroesreward/objectiveand every other positive component. The penalties stand. - The term's score is the weighted fired fraction of its checks. A non-veto check at weight
0is logged, never priced. A veto check counts only when given a weight: a penalty on top of the gate, since a vetoed failure has no credit to lose beyond its bonuses. - The term's
weightmust be<= 0. The default0logs flags unpriced; a negative weight charges them. on_errordefaults toneutral, so a judge outage leaves the environment's grade standing.
Where it runs
A veto judge needs an environment term in the same reward to gate, so the online arm refuses one at config time (list 'requirements' instead).
Metrics
Per completion, averaged per step on both arms (the online arm logs 0 on a rank where no sample carried a key).
The environment arm also keeps the rationale and each fired check's quote on the trajectory (reward_details).
Source on GitHubUpdated 18 November 2026