Reward
An episode's reward is the solve, small shaping rungs around it, the trainer's reasoning length terms (not charged in a training group with no success), and an audit judge whose veto zeroes an episode's credit and whose weighted checks charge a fixed price. Recipe values: Shipped recipes.
Reward ladder
The grade is read off the last submit_solution call and is all-or-nothing, matching the accept verdict pass@1
counts. Partial credit would pay a brute force that times out on the large tests.
- An unsubmitted solution, an infra outage and an inconclusive grade score 0 and pay no shaping rung; the resubmission penalty and tool shaping still apply. The last two mark the episode invalid.
- The
environmentterm'sweightprices a solve; itsexponenthas nothing to reshape (Reward Terms).
- Keep each shaping rung small next to the objective's weight: rungs bootstrap a base that never submits and cancel out within a group once every completion reaches them.
- Components log as
reward/*and sum exactly to the reward. The trainer's reasoning length terms log outside them, asreward/reasoning_priceandreward/reasoning_floor.
Resubmissions
resubmission_penalty is flat, whether a resubmission fixes or re-rolls, and never stated to the model. A
resubmission identical to a program the episode already graded is not regraded: it counts and pays like any
other, and its reply repeats the earlier verdict.
episode/resubmission_improved is the share of resubmissions whose pass fraction beat every earlier one.
What scorers read
A judge or reward_model term's final view reads the submitted program as the episode's final answer, a
fenced code block, never the hidden tests. view: full reads every turn instead, and a judge's digest the turns
in brief ahead of that answer (What a scorer reads).
Audit judge
Every recipe runs a veto judge named audit on what no counter
can judge. It reads the whole episode, reasoning included, beside a context that explains the environment and
its caps. Its checks are the environment's own: checks: environment takes the rubric CodeContestsEnvironment
declares (JUDGE_CHECKS, built from AUDIT_CHECKS in code_contests_prompts.py), the run's scratchpad (python_repl or run_code) named
in it, so a recipe states only the judge's model, view, price and context. The context comes from a shared
context_file under examples/grpo/environmental/code-contests-judge/, one file per chat template, rollout
backend and scratchpad: gptoss.txt (gpt-oss on either engine), gemma4-vllm.txt, gemma4-qwen3.6-sglang.txt,
qwen3.6-vllm.txt, and qwen3.6-vllm-run-code.txt for the Python and C++ stages. The term's weight (−0.7) prices
the weighted checks by their check weights, each fired one adding its own price. The check weights are the
environment's (7 in all), so −0.7 is 0.1 per unit; a recipe of your own that lists checks: environment takes the
same check weights, so its unit is its weight / 7.
The judge is served through OpenRouter and reads OPENROUTER_API_KEY: pass it with --env-file, or drop the term
to run without a judge.
A veto zeroes every credit (the solve and the submission bonus); penalties stand, the veto's own price among them, so a guess or a probe
on a problem nobody solves still costs more than an honest failure. wasted_run is priced and reasoning_in_runs only logged, not
vetoed: a veto there would teach a policy to submit untested. untested_submission prices failed episodes only: the policy tests when
it is unsure, so within a group an outcome-only reward charges testing, and pricing the untested failure is what pays for testing
beyond the samples. Prices add when several checks fire. A call the environment refused unrun is marked so in the judge's view: it ran nothing and spent no submission or run, but a closing-tag refusal is priced as a tool error (tool_error_penalty).
Judge outages
The recipes' on_error: invalid drops an episode the judge
could not score from its group's baseline and the loss, so an outage (an expired key, a provider failure) trains
nothing rather than every episode unjudged. judge/audit/scored and sampling/invalid_episode_frac show it, and a
second step in a row with no valid episode stops the run (Batch construction).
Cap context
Where the engine closes reasoning at a one-token marker (Qwen3.6 and Gemma 4 on vLLM), a turn whose reasoning
reaches the run's reasoning cap is marked for the judge. Each recipe's context states what the policy sees of its
earlier reasoning (carry_reasoning) and of its turn limits.
Economy invariants
tests/cpu/config/test_env_grpo_reward_economy.py holds every shipped recipe to these relations:
- The floor's weight and the price's cap each stay under one more failed submission at its worst honest price
(the resubmission penalty plus
untested_submission), so reasoning length never outweighs whether to resubmit. An episode pays the floor or the price, never both. - A graded submission that passes nothing scores above an episode that never attempts; raising every audit process flag, it scores no lower at the same length terms.
- At every level, a solve that pays every per-episode price (its resubmissions, admitted recoveries, the worst
length term, the process flags a solve can raise) out-scores any zero-objective episode, before
tool_error_penaltyper failed call. A solve never pays the overflow: the accept ends the episode. - The overflow costs more than one more failed submission at its worst honest price (the same bound). A truncated episode never solves and always has a submission left, so running out is never the cheapest way to fail.
- A recovered cut costs less than the submission bonus and no more than an overflow.
- Under its budget a failed episode at every level is paid to reason more; a higher level is priced lower and given a larger budget, and every level's budget leaves the episode output budget room for a turn.
- The episode output budget binds below
max_turns × rollout_max_tokens; on vLLM the answer room binds below the turn ceiling. - Each audit check fired alone costs what the audit table states, so moving a judge
weightor a check weight alone, which re-prices every weighted check, fails here; each shaping price is the one the recipe table states.
Behavior counters
episode/submission_rateepisode/test_calls: scratchpad runs that countedepisode/tested_before_submission: over submitting episodesepisode/grading_budget_hitepisode/identical_resubmissions: resubmissions identical to an already graded programepisode/unrun_failed_submission: a failed graded submission was a program never run as submitted while a run was left, comments, whitespace and dropped output statements aside. The match is textual, so a tested program that a guard, a rename or a harness changed counts as unrun;untested_submissionis the price that reads the changeepisode/resubmission_improved: Resubmissionsepisode/language_switches: where the model picks the language
Grade outcomes (episode/grade_inconclusive, episode/grading_infra_outage, episode/tests_graded_frac):
Grading.
Source on GitHubUpdated 18 November 2026