Grading
submit_solution grades through grade_solution (halo/environments/tasks/coding/grading.py; the per-test
verdicts, comparators and checker in verdicts.py). The offline
re-grader shares it, so the same rules grade online and offline; the re-grader alone stops at the first failing test
and runs without max_grading_seconds (Re-grading). How the grade is paid:
Reward ladder.
Grading rules
- One sandbox session serves the whole grade: on
localandbubblewrapa compiled submission builds once, and the session resets after every test. - The verdict lists non-passing tests only, one entry per distinct verdict with the tests that failed the same way folded in, at most five.
Build failures
A compile failure is graded once against the whole pool, as is Python source that does not compile on local
and bubblewrap.
bubblewrapshows the error: its one build runs before any test, on an empty stdin.localhides it: a program can force a rebuild that reads a test's input.remotehides it: its build shares each test's request. It may run another Python version, so Python source runs there.- A Python syntax error quotes the source alone (
SyntaxError: invalid syntax (line 3)).
Comparison
exactcomparison spuriously fails correct Codeforces solutions, hence thecodeforcespreset.- Token comparison accepts a real-valued token within an absolute or a relative error of
1e-6of the expected one, only when the expected token looks like a float, so integer answers stay exact. A statement that states a looser bound is graded under it, each bound in its own kind: "absolute or relative error does not exceed 10^{-4}" loosens both, "an absolute error of 10^{-2} or a relative error of 10^{-6}" each its own, and a bound that names no kind loosens both. A tighter one never tightens the default, since the expected outputs are a reference solution's printed digits.
Verdict detail
outcome shows each failed test's verdict class alone: FAIL, RUNTIME ERROR, TIME LIMIT EXCEEDED,
OUTPUT LIMIT EXCEEDED, COMPILATION ERROR (build failures), or ERROR for a test lost to
infra (its text goes to the log). Stderr, exit status and output size are withheld because each can carry the
hidden input. stop_on_first_failure narrows what reaches the policy to the first failure, the Codeforces contract.
full adds a runtime error's signal or exit status and stderr tail, an output overrun's size against its cap, an
infra error's text, and a wrong answer's expected and produced output. A second submission then becomes a free test
oracle, and probing out-earns scratchpad testing.
Scratchpad runs on the model's own inputs show their output in both modes.
Time limits
- The per-test cap is the payload's
time_limit, elsetimeout_per_test. An interpreted language is floored attimeout_per_test, so a C++-tuned limit cannot fail a correct CPython solution. A compiled one is scaled bycompiled_time_limit_scale. Both are clamped tomax_time_limit. - On
localandbubblewrapthe cap is wall clock from the program's exec, the interpreter's startup included and the sandbox's launch excluded (Limits);remotekeeps its service's own clock. The build runs before it, under its own limit. - A list mixing interpreted and compiled languages states the floor, scale and clamp in the system prompt.
- A payload without
time_limitgets each language's limit appended to the task message (Each test of this problem runs under 5 s in python, 10 s in cpp.). - On
local/bubblewrapa compiled program's stack is raised to the memory limit, as on Codeforces, up to the process's hard stack limit (Limits).
Grading budget
Tests run sequentially, so a problem with hundreds of tests stalls the round. max_grading_seconds bounds one
grade, checked between tests. Size it for an honest solution: an ungraded test never counts as passed.
- Stopped after a failed test: a wrong answer.
- Stopped before any failure: grade 0, and the episode is marked invalid, out of the group baseline
(
episode/grade_inconclusive). episode/tests_graded_fracshows a partial grade.
Special judges
A per-problem checker (Python) in the payload overrides comparison:
python checker.py input.txt correct_output.txt solution_output.txt. The submission is accepted only when the
checker exits cleanly and its last stdout token is 1. The checker runs at the 15 s infra default, never the
solution's limit.
Infra errors
- Tests lost to the backend with none passed and none failed mark the episode invalid, out of the group baseline
(
episode/grading_infra_outage). - Lost tests beside passes and no failure leave the grade inconclusive, as a budget stop does.
- Verdicts, not infra: a build past the compile limit (compile error), a program that replaces its working directory (runtime error), one that floods its output (output-limit or runtime error). A program that removes its working directory runs the next test in a fresh one.
Other routes into an infra error: Sandbox faults.
Source on GitHubUpdated 18 November 2026