Tools
The model gets two tools: a scratchpad that runs a program on an input it supplies, and submit_solution, the
only graded channel. Their per-episode budgets, max_test_calls and max_submissions, are set per
effort level. Both tools:
- State no budget, in their description or in any reply. A refusal names what to do, never a number.
- Neither refuse nor price a program's comments or what a run showed; the audit judge handles both.
- Cut output that would push a reply past
max_observation_chars(…[truncated N chars]), so the notes after it survive.
Scratchpad
The scratchpad is python_repl when the run fixes python, else run_code.
- It runs a program in the grading sandbox on the
stdinthe call supplies (empty by default), never on the graded tests. - Each call is one-shot: nothing a run writes survives into the next.
- A run gets the per-test time limit its language is graded at.
- The reply leads with any error (the compiler's first diagnostics, or a crash's signal and stderr tail), then stdout (Using it from Python).
- A sandbox fault ends the episode.
- An empty or whitespace-only
stdinadds a note that none was passed, unless the build failed. The run still spends one ofmax_test_calls; it is never refused, since a self-test may embed its input.
submit_solution
submit_solution grades a complete stdin/stdout program against the hidden tests (Grading). There
is no fenced-code-block fallback.
- A submission that passes every hidden test ends the episode, as reaching
max_submissionsdoes. A later submission in the same turn is not graded. - A program identical to one this episode already graded is not regraded: its reply repeats that grade under
Identical to submission k, already graded: …(episode/identical_resubmissions). It is an ordinary submission otherwise: it spends one ofmax_submissions, pays the resubmission price, and its turn trains. Comments and trailing whitespace are no change. The offline re-grader gives it its slot.
Refused calls
A call is refused when:
- it is past its budget: the scratchpad replies
Error: Not run: this task's scratchpad budget is spent. Submit your solution with submit_solution., andsubmit_solutionpastmax_submissionsin the same turnError: Not graded: this task's submission budget is spent.; - its arguments do not bind: an undeclared argument such as
languageon a single-language run (Error: <tool>: unknown argument 'language'; …), a list or object ascodeorstdin, alanguageoutside the run's list, or a value ending in its own closing tag (astdinending</stdin>, see Argument binding); - its program is evidently in another language than its
languagenames.
A refused call pays tool_error_penalty, never tool_success_reward. It spends its turn but none of
max_test_calls or max_submissions, and grades nothing. A turn whose every call was refused or unknown is
untrainable.
Not refusals:
- A call to a tool whose budget is 0 (
ToolDisabled: theleaderboardscratchpad,max_test_calls: 0) gets the same reply and price, but its turn is not flagged. - A submission after an accept in the same turn that the budget still admits is returned to its budget, neither
paid nor charged. It counts in
episode/tool_calls, but not as a successful call.
Languages
language takes python, cpp, c, or a list the model picks from
(sandbox languages).
- With a list, both tools take a required
languageargument, and each program is graded in the language its call names. The episode's last language is itslanguagemetrics slice. - With one language, the tools declare no
languageargument and refuse a call naming one. - The tool descriptions name the toolchain (compile flags, Python version) on
localandbubblewrap;remotestates none.
Evident language
With a language list, a program evidently written in another of its languages is not run:
Text
Error: Not run: this looks like cpp code sent with language "python"; send it again with language "cpp". No scratchpad run was spent.submit_solution says Not graded instead. The call is refused: it pays tool_error_penalty,
spends no budget, grades nothing and leaves the language slice unchanged, and a turn of such calls alone is
untrainable, so no solve reinforces the wasted turn. The check (find_evident_language in program_source.py) acts
only when a Python parse and the program's textual marks point the same way; a program it misses runs as labeled.
A submission after an accept takes the post-accept reply whatever its label.
Recovery nudges
A final text answer ends the episode ungraded, so the nudge after a cut or empty turn
(Unproductive turns) asks for a tool call and names submit_solution. A
turn cut while writing its call is asked to call again with its reasoning kept out of the program's comments.
Source on GitHubUpdated 18 November 2026