Environments

An environment is the multi-turn task an Async GRPO with Environments run trains against. The protocol (EnvironmentBase, halo/environments/environment_base.py):

  • reset(prompts, contexts) opens one episode per prompt.
  • step(episode_ids, actions, contexts) executes a model turn and returns the next observation.
  • get_trajectories(episode_ids) hands back the messages and the final reward.

An episode ends when the environment reports it done, or at max_turns.

Registry

environment_type resolves through build_environment (halo/environments/registry.py). get_registered_environments() lists the live set; register_environment extends it (Custom Environments).

environment_typeClassToolsDefault max_turns
react_mathReActEnvironmentcalculate, python10
react_searchReActEnvironmentweb_search10
native_mathNativeToolUseEnvironmentcalculate, python10
native_codingNativeToolUseEnvironmentpython_repl10
native_combinedNativeToolUseEnvironmentcalculate, python, web_search, file ops10
qa_searchNativeToolUseEnvironment (factory)web_search (+ python under include_python_tools)10
exam_qaExamQAEnvironmentnone; web_search under open_book8
sweSweEnvironmentrun_code, run_bash_command, workspace file ops20
code_contestsCodeContestsEnvironmentscratchpad test tool, submit_solution15
codeforcesCodeContestsEnvironmentsame, graded by token comparison15
mcpNativeMCPClientEnvironmentwhatever the server advertises10

The last column is the class's own turn budget, which max_turns: null keeps.

The ReAct presets parse the action out of the text. Every other environment uses native tool calls, so the server needs the family's --tool-call-parser (Tool calls).

Actor runtime

Actors run environments on CPU and inherit the ray start daemon's environment, so export what they need before ray start on each node (Ray Cluster).

EnvironmentActor nodes need
code_contests, codeforces, sweA sandbox backend and a large TMPDIR (Code Execution Sandboxes)
react_math, native_math, native_coding, closed-book exam_qaNothing beyond the image (in-process tools)
react_search, qa_search, native_combined, open-book exam_qaOutbound network. SERPER_API_KEY / BRAVE_API_KEY / TAVILY_API_KEY / KEENABLE_API_KEY picks a keyed search backend, else keyless DuckDuckGo. search_backend names one instead (QA Benchmarks)
mcpThe preset's launcher on PATH (npx; uvx for fetch) and its credential. An SSE server needs only network to server_url

Configuration

EnvironmentConfig (halo/configs/grpo_environmental.py) has four top-level fields. Everything else goes in environment_kwargs.

YAML

environment_type: react_math
rewards:
  - source: environment    # the environment's grade in [0, 1], priced weight × grade ^ exponent
max_turns: null            # null keeps the class default

environment_kwargs:
  carry_reasoning: false
  tool_error_penalty: 0.1
  • Any other constructor parameter of the resolved class that YAML can spell is settable from environment_kwargs; one taking a Python object, and mcp's custom server_command / server_args / server_env (From Python), is not. A key no constructor binds (a typo, or another environment_type's option) raises TypeError at construction.
  • A boolean knob takes YAML true or false; any other value, no and off included, raises at construction.
  • The ReAct presets refuse a set system_prompt, mcp reads mcp_server in its factory, and native_combined reads search_backend in its own.
  • rewards reaches the constructor as reward_terms (Reward Terms).

Common knobs

Every environment accepts these environment_kwargs, beside the top-level max_turns and rewards (Configuration).

KnobDefaultEffect
tool_success_reward / tool_error_penalty0.05 / 0.1 (mcp pays 0.1; code contests 0 / 0)Paid per successful tool call, charged per failed one, logged as reward/turn_shaping. A call a sandbox outage ended is unpriced (Sandbox faults)
tool_reward_captool_success_reward × max_turnsEpisode cap on successful-call pay, so call spam cannot out-earn the objective. A closed-book exam_qa, with no tool to price, refuses all three pricing knobs
max_observation_chars16384Longer tool observations are truncated; an integer >= 0, 0 = no cap
reasoning_effortNone (code contests medium)low / medium / high / random CoT steer (Effort levels)
carry_reasoningfalseSends the last assistant turn's reasoning back to the engine; vLLM only (Carried reasoning)
max_length_cutoff_recoveriesnullUnproductive turns (engine-cut, or no visible content and no tool call) one episode may retry; an integer >= 0, null = all within max_turns (Unproductive turns)
requires_answerfalse; true for code contests, exam_qa, qa_search, the ReAct presets and swe (unless judge-only)The reward grades against the dataset's answer column, so a dataset without one is refused at trainer construction; code contests refuse false

Reward and penalty knobs are magnitudes (≥ 0); a negative value raises. A bare environment_kwargs: header reads as null and raises: comment it out, or list its keys.

Pages

Tool protocols

  • ReAct — Thought / Action / Observation in plain text.
  • Native Tool-Use — function calling and tool factories; most environments extend it.
  • MCP — tools from an MCP server.

Tasks

Infrastructure and authoring

Source on GitHubUpdated 18 November 2026