Grading

submit_solution grades through grade_solution (halo/environments/tasks/coding/grading.py; the per-test verdicts, comparators and checker in verdicts.py). The offline re-grader shares it, so the same rules grade online and offline; the re-grader alone stops at the first failing test and runs without max_grading_seconds (Re-grading). How the grade is paid: Reward ladder.

Grading rules

  • One sandbox session serves the whole grade: on local and bubblewrap a compiled submission builds once, and the session resets after every test.
  • The verdict lists non-passing tests only, one entry per distinct verdict with the tests that failed the same way folded in, at most five.

Build failures

A compile failure is graded once against the whole pool, as is Python source that does not compile on local and bubblewrap.

BackendCompile error, outcomeCompile error, fullPython syntax error
bubblewrapCompiler's first errorCompiler's first errorFirst error in both modes, no run
localVerdict class aloneCompiler's first errorFirst error in both modes, no run
remoteVerdict class aloneCompiler's first errorRuns; graded as the service reports it
  • bubblewrap shows the error: its one build runs before any test, on an empty stdin.
  • local hides it: a program can force a rebuild that reads a test's input.
  • remote hides it: its build shares each test's request. It may run another Python version, so Python source runs there.
  • A Python syntax error quotes the source alone (SyntaxError: invalid syntax (line 3)).

Comparison

  • exact comparison spuriously fails correct Codeforces solutions, hence the codeforces preset.
  • Token comparison accepts a real-valued token within an absolute or a relative error of 1e-6 of the expected one, only when the expected token looks like a float, so integer answers stay exact. A statement that states a looser bound is graded under it, each bound in its own kind: "absolute or relative error does not exceed 10^{-4}" loosens both, "an absolute error of 10^{-2} or a relative error of 10^{-6}" each its own, and a bound that names no kind loosens both. A tighter one never tightens the default, since the expected outputs are a reference solution's printed digits.

Verdict detail

outcome shows each failed test's verdict class alone: FAIL, RUNTIME ERROR, TIME LIMIT EXCEEDED, OUTPUT LIMIT EXCEEDED, COMPILATION ERROR (build failures), or ERROR for a test lost to infra (its text goes to the log). Stderr, exit status and output size are withheld because each can carry the hidden input. stop_on_first_failure narrows what reaches the policy to the first failure, the Codeforces contract.

full adds a runtime error's signal or exit status and stderr tail, an output overrun's size against its cap, an infra error's text, and a wrong answer's expected and produced output. A second submission then becomes a free test oracle, and probing out-earns scratchpad testing.

Scratchpad runs on the model's own inputs show their output in both modes.

Time limits

  • The per-test cap is the payload's time_limit, else timeout_per_test. An interpreted language is floored at timeout_per_test, so a C++-tuned limit cannot fail a correct CPython solution. A compiled one is scaled by compiled_time_limit_scale. Both are clamped to max_time_limit.
  • On local and bubblewrap the cap is wall clock from the program's exec, the interpreter's startup included and the sandbox's launch excluded (Limits); remote keeps its service's own clock. The build runs before it, under its own limit.
  • A list mixing interpreted and compiled languages states the floor, scale and clamp in the system prompt.
  • A payload without time_limit gets each language's limit appended to the task message (Each test of this problem runs under 5 s in python, 10 s in cpp.).
  • On local / bubblewrap a compiled program's stack is raised to the memory limit, as on Codeforces, up to the process's hard stack limit (Limits).

Grading budget

Tests run sequentially, so a problem with hundreds of tests stalls the round. max_grading_seconds bounds one grade, checked between tests. Size it for an honest solution: an ungraded test never counts as passed.

  • Stopped after a failed test: a wrong answer.
  • Stopped before any failure: grade 0, and the episode is marked invalid, out of the group baseline (episode/grade_inconclusive).
  • episode/tests_graded_frac shows a partial grade.

Special judges

A per-problem checker (Python) in the payload overrides comparison: python checker.py input.txt correct_output.txt solution_output.txt. The submission is accepted only when the checker exits cleanly and its last stdout token is 1. The checker runs at the 15 s infra default, never the solution's limit.

Infra errors

  • Tests lost to the backend with none passed and none failed mark the episode invalid, out of the group baseline (episode/grading_infra_outage).
  • Lost tests beside passes and no failure leave the grade inconclusive, as a budget stop does.
  • Verdicts, not infra: a build past the compile limit (compile error), a program that replaces its working directory (runtime error), one that floods its output (output-limit or runtime error). A program that removes its working directory runs the next test in a fresh one.

Other routes into an infra error: Sandbox faults.

Source on GitHubUpdated 18 November 2026