Evaluation
A run evaluates in two ways: eval rounds inside training, and offline CLIs that drive a served model through an environment. Environment-specific commands live on each environment's page (Code Contests, QA Benchmarks).
During training
Eval metrics carry the eval_ prefix. Eval rounds run without prefetch, and a round waits for its slowest episode. An eval pushes first, whatever sync_weights_every_n_steps says, so outcome/success@N and best-model selection read the weights being evaluated and saved; the next step's push is then skipped as redundant.
num_generations_eval sets the eval group size (1 gives a fast pass@1 eval) and eval_rollout_batch_size the rows, not prompts, per rank in one eval round; types and defaults are in GRPOConfig and AsyncRolloutConfig.
eval_rollout_batch_sizemust be a multiple ofnum_generations_eval, at mostmax_concurrent_rollouts, and needsdataloader_drop_last: false; each raises at startup (eval_rollout_batch_size). Unset,per_device_eval_batch_sizemust hold whole groups instead; the loss forward chunks by it either way.- Under
reasoning_effort: random(Effort levels) each problem's level is drawn from its text, here and under the offline--training_config, so checkpoints compare level for level. - With
num_generations_evalabove 1 and a solve verdict,eval_outcome/success@<k>is the share of prompts any of theirkvalid samples solved;eval_outcome/solve_ratestays the per-sample mean.
Padding and round means
Every rank runs the same rounds, so a short last round is padded with repeats of the split's first rows, never rolled out or counted. Two biases remain:
- TRL's loss-time eval metrics (
eval_entropy,eval_clip_ratio/*,eval_kl) count a rank whose final round is all padding as 0. - Every logged eval metric weighs rounds equally, however full.
For exact means, make len(split) × num_generations_eval a multiple of eval_rollout_batch_size × data_parallel_size, or, for the rollout metrics, run the split in one round.
Running an evaluation
Two CLIs drive training's own episode loop (play_episode, through halo/environments/rollouts/eval_runner.py) against an OpenAI-compatible endpoint (vLLM, SGLang, OpenRouter): eval_env.py for generic environments, eval_code_contests.py for code contests; their runs are halo/entrypoints/generic_env_eval.py and halo/entrypoints/code_contests_eval.py. Replies are read by the training rollout's own parser (turn_from_chat_completion).
Bash
python scripts/environments/inference/eval_env.py --env_type qa_search \
--dataset basicv8vc/SimpleQA --split test --prompt_field problem \
--base_url http://localhost:8000/v1 --model <served-name> --num_examples 100Every request also sends top_k, min_p and repetition_penalty (the run's rollout_* values under --training_config, else off), so a served model's generation_config.json never filters the eval.
A message-list prompt reaches the environment as its last user turn, as in training; a row with no user turn, or a non-text content part in it, is refused before any episode runs.
Script-specific flags
eval_env.py reads --prompt_field / --answer_field, passes extra columns through --context_fields, buckets by --group_by and names examples by --id_field (default id). A field naming no column exits before any row is read (the default answer and id columns may be absent). Like the trainer, it refuses a dataset without the answer field when the environment declares requires_answer.
eval_code_contests.py takes instead --adapter (fixing the bucket and id fields per benchmark), --language, --eval_protocol, --reasoning_effort (low / medium / high / none), --include_examples_only and, on dated benchmarks, --start_date / --end_date / --platform (Code Contests evaluation). An option with a flag of its own is refused in --env_kwargs.
Grading under a training config
--training_config <yaml> parses the YAML with the training script's config classes (TrainingContract, halo/entrypoints/eval_contract.py), adopting its RolloutConfig (backend, template variables, stop tokens, reasoning cap and episode output budget, sampling, request timeout) and environment config (rewards, max_turns, environment_kwargs, environment_type). An explicit flag beats the YAML; the YAML beats the default.
- It refuses what the trainer refuses, through the same resolvers, before any episode: a
rollout_stop_tokensentry the config's tokenizer does not know (resolve_rollout_stop_token_ids), and under vLLM a reasoning cap the served model type cannot enforce (Mistral 4;reject_unenforceable_thinking_budget). - The YAML's dataset keys (
dataset,prompt_field,answer_field,context_fields) are not read: pass the matching flags. - A relative path in the YAML, a judge's
context_fileincluded, resolves against the cwd: run from the repo root, wherehalo launchruns training. - The eval joins no process group, so a recipe whose
episode_timeoutneeds a raisedDIST_NCCL_TIMEOUT_MINUTESin training evaluates without it. - To compare a checkpoint with its base, serve each under the same
--served-model-nameand rerun the same command.
The report
Besides mean reward and aggregate telemetry, the report logs:
Neither success score is a benchmark's mean-over-samples pass@1. Reasoning models need a large --max_tokens, or the chain of thought is cut before any answer and scores 0.
A sample counts as solved on the environment's own verdict where it reports one (the flag behind outcome/solve_rate), whatever its shaped reward.
Failed generations
The OpenAI client retries transport failures itself. Training's retry policy (max_retries, retry_base_wait from --training_config, else 3 and 1 s) covers what the client does not: vLLM's not JSON compliant 400 is retried and an engine abort re-issued. A generation that still fails ends the episode:
- Its own length (a conversation over the served context, or a turn that outran
--request_timeouton every client retry): the sample is graded on what it earned, like a miss. - Any other cause (a rejected key, an unknown model or route, a malformed request): a generation error.
rewardandsuccessare null, the sample leaves every score but the telemetry line, andgeneration_errorscounts it.
A score left with no sample reads nan.
Output files
--output_path <path.json> dumps per-example results: group, id, and per sample reward, success, stats, error or generation_error where set, plus any recorded trajectory.
--save_trajectories <path.jsonl> records the full run; --trajectory_dir <folder>, which it excludes, auto-names one file per run: <model>__<env_type>__<split>.jsonl, or for the coding script <model>__<adapter>__<split>__<language>.jsonl plus __<eval_protocol> (other than harness) and any contest selection.
Line 1 is a meta record of the run's contract: model, env type, dataset/config/split, effective max_turns, rollout, training_config, system_prompt, tool schemas, and for coding the adapter, contest selection, language, eval_protocol, effort and env_grading.
Each later line is an episode addressed by index and id: reward, success, generation_error, stats, the messages, reasoning_effort and info.
context, the_-prefixedinfofields (code contests' hidden tests and checker) and assistant chain-of-thought are stripped; otherinfokeys stay (exam_qakeepsexpected_answer).- Messages keep their
tool_calls, which the re-grader replays; a call the engine cut stays as unruncut_tool_calls, which it skips, its arguments decoded as training decodes them (Cut tool calls). - Each assistant turn under a reasoning cap keeps its
thinking_cap, the run's cap (recorded on SGLang too, which does not enforce it; Thinking-cap metric). - A turn counted under a resolved single-token reasoning close (a vLLM
--training_configrun with a reasoning cap, against a server started with--return-tokens-as-token-ids) also recordsreasoning_tokens, the reasoning it sampled.
Re-grading recorded trajectories
Only eval_code_contests.py dumps carry the meta a re-grade needs; an eval_env.py dump is refused. A wall-clock grade inflates under host load, so a high-concurrency sweep can read a correct, fast solution as TIME-LIMIT-EXCEEDED. Generate in parallel, then re-grade in one bounded process:
Bash
python scripts/environments/inference/regrade_trajectories.py \
"$HALO_DATA_ROOT/eval/trajectories"/*.jsonl --max_workers 64 --output_path regraded.jsonlIt rebuilds each problem's hidden tests by index under the meta line's contest selection and replays every admitted submit_solution (a refused call, a mislabelled language or a repeat of a graded program takes no slot) under the recorded env_grading (the grading settings the meta line records):
- up to the episode's own submission budget (the rebuilt environment's
max_submissionsfor an episode that stamped none); - stopping at the first failing test, without
max_grading_seconds.
A file is refused before any grade when:
- an episode's
indexlies past the rebuilt problems: the dataset is not the one the run scored; env_gradingnames a field the grading contract does not declare, or holds a value it refuses (an unknowncomparisonorverdict_detail, a non-positive time limit).
A selection that records no include_examples_only is read as including the examples-only problems.
Per file it reports the protocol and, over the n episodes with a verdict, s@1 (the first admitted submission passes every test) and s@2 (any submission within the budget does). An episode with a generation_error leaves n and counts in generation_errors. Keep --max_workers at or below the core count.
Smoke-testing a config
Point the eval at the training YAML and a few rows before a long run:
Bash
python scripts/environments/inference/eval_env.py --training_config <config>.yaml \
--dataset <the config's dataset> --model <served-name> \
--num_examples 5 --max_workers 4 --save_trajectories /tmp/smoke.jsonlIt checks what otherwise costs a whole run: the server's tool-call parser, rollout_stop_tokens against the tokenizer (an unresolved name raises), the chat template, the sandbox or judge backend, and the dataset columns.
Source on GitHubUpdated 18 November 2026