Evaluation
scripts/environments/inference/eval_code_contests.py scores a served model on this environment. Flags, output
files and re-grading shared by every environment are on Evaluation.
Evaluation protocols
eval_protocol names the contract a run is scored under:
- A configured value that contradicts a pin raises at construction. An effort profile's tool budgets are validated, then give way to the pins (logged).
- Under
leaderboardthe model learns the scratchpad is closed only from its reply, a budget-0 call that is not a refusal.
Running an evaluation
Bash
python scripts/environments/inference/eval_code_contests.py --adapter codeforces \
--dataset open-r1/codeforces --config verifiable --split test \
--training_config examples/grpo/environmental/qwen3_5/vllm/qwen3.6-35b-a3b-code-contests-full-ep1-stage1-codeforces.yaml \
--base_url http://localhost:8000/v1 --model <served-name> \
--num_examples 100 --num_samples 4 --reasoning_effort high--adaptertakes the adapters that score raw rows: every adapter buthardtests, whose suites join only in its prepared pool.--env_typepickscodeforcesorcode_contests, else the training config'senvironment_type, elsecodeforces.--training_configsupplies the recipe's rollout settings and language list. A policy trained on[python, cpp]names a language in every call, which--language pythonalone refuses.--eval_protocolpicks the protocol, else the training config's, elseharness; the report title and trajectory meta name it.- A training config written under another protocol yields the tool budgets the flag's protocol pins (logged); one
naming the protocol itself, or
--env_kwargs, that contradicts a pin raises. - Grading knobs with no flag go through
--env_kwargs, recorded in the trajectory meta.
Scoring
success@1 / success@k are bucketed by the adapter's field (rating here); neither is a benchmark's
mean-over-samples pass@1 (Running an evaluation).
- A problem counts solved when the episode's last graded submission passes every test. Shaping never moves it,
and the coding CLI takes no
--success_threshold. - Examples-only problems are left out and counted in the log unless
--include_examples_only, which the trajectory meta records with the selection. On an adapter that marks none, the flag exits before a row is read. success@1is each row's first scored sample. The re-grader scores each episode on its first submission (s@1) or any within its budget (s@2); the two agree only on aleaderboardrun at--num_samples 1.
Generation budget
--reasoning_effortdefaults to the training config's level, wherereasoning_effort: nullisnone. A config without the key, or no config, takesmedium, as training does.--max_tokensdefaults to the training config'srollout_max_tokens, else 32768. The level never changes it.- Without
--training_configno reasoning cap is sent. A training config's cap needs a reasoning parser on the vLLM server, which otherwise rejects the request. - Evaluate a non-thinking model with
--reasoning_effort none; a think-tag parser reads its whole answer as reasoning.nonedoes not switch a thinking model's reasoning off: the request carries noreasoning_effort, so the template keeps its own default. Qwen3/3.5/3.6 still think, and stop only onenable_thinking: false(the server's--default-chat-template-kwargs, orrollout_chat_template_kwargsunder--training_config). - The eval treats the server as SGLang only when
--training_confignamesrollout_backend: sglang, and then drops the reasoning cap as training does.
LiveCodeBench window
A LiveCodeBench release is cumulative (every problem since May 2023), so score a window after the model's training cutoff:
Bash
python scripts/environments/inference/eval_code_contests.py --adapter livecodebench \
--dataset livecodebench/code_generation_lite --config release_v6 \
--start_date 2025-01-01 --end_date 2025-04-30 --platform atcoder,codeforces \
--eval_protocol leaderboard --base_url http://localhost:8000/v1 --model <served-name> \
--num_examples 0 --num_samples 4--start_date/--end_date(YYYY-MM-DD) are inclusive, compare the row'scontest_dateby calendar day, and may each stay open.--platformtakesatcoderandcodeforces.leetcodeis refused: its rows are functional, which a stdin/stdout environment cannot grade.- Rows come newest release file first, each file in its stored order, which is not newest-first.
--num_examples(default 50) takes the window's first problems in that order;0scores the whole window. - Another date spelling, a start after the end, an unknown platform, or a window or platform filter on any adapter
but
livecodebenchexits before a row is read. A row without a parsablecontest_dateraises under a window.
The selection is recorded in the trajectory meta (re-grading).
Source on GitHubUpdated 18 November 2026