Custom Environments
Subclass EnvironmentBase (halo/environments/environment_base.py) and implement three methods, then register a factory for YAML or pass the class from Python. The base composes the tool accounting (tools/accounting.py), turn recovery (episodes/turn_recovery.py), effort profiles (episodes/reasoning_effort.py) and scoring (episodes/episode_scoring.py, EpisodeGrade) your class inherits.
Python
from halo.environments.environment_base import EnvironmentBase
from halo.environments.episodes.episode_scoring import EpisodeGrade
from halo.environments.episodes.messages import StepOutcome
class GuessEnvironment(EnvironmentBase):
DEFAULT_MAX_TURNS = 6
requires_answer = True # the grade reads context["answer"]
def _reset_single(self, prompt, context=None):
expected = (context or {}).get("answer")
return self._init_trajectory(prompt, context, extra_info={"expected": expected})
def _step_single(self, trajectory, action, context=None):
done = "Final Answer:" in action
trajectory.info["completed"] = done
return StepOutcome(trajectory=trajectory, reward=0.0, done=done, truncated=False, info={})
def _grade_episode(self, trajectory, context=None):
expected = trajectory.info.get("expected")
answered = expected is not None and str(expected) in trajectory.messages[-1].content
return EpisodeGrade(1.0 if answered else 0.0)Rules that bite:
- Grade, do not price.
_grade_episodereturns anEpisodeGrade; a class that defines_compute_rewardis refused at construction (Environment arm).objectiveis the task grade in[0, 1], priced by the reward'senvironmentterm (reward/objective).shapingmaps episode-level terms to values ({"submission": 0.25}), each added asreward/<name>. Declare the names inSHAPING_COMPONENTS, or the first settled episode fails.objectiveandturn_shapingare reserved, and no reward term may reuse a declared name.- Protocol-level shaping goes in
_episode_shaping(trajectory) -> dict. - An episode whose grade carries no signal (a grader outage, a null
answercell) setstrajectory.info[EPISODE_INVALID_KEY] = True, its cause underEPISODE_INVALID_REASON_KEY, and grades 0, so it leaves the group baseline.
- Hand a scorer what to read. Override
_final_answer,_scoring_referenceand_episode_facts(withEPISODE_FACTS) forjudgeandreward_modelterms (What a scorer reads). - Forward
**kwargstosuper().__init__. The base bindsrewards(asreward_terms),max_turnsand the common knobs; a keyword no constructor binds raisesTypeErrorat construction. - Declare differing defaults on the class (
DEFAULT_MAX_TURNS,DEFAULT_TOOL_SUCCESS_REWARD,DEFAULT_TOOL_ERROR_PENALTY,requires_answer), not in the factory, so a YAMLmax_turnsstill wins. - Build through
self._init_trajectory(...), so the system turn, the task turn and the context land where the base expects them. - Route an unproductive turn, never grade it (Unproductive turns).
- An engine cut arrives with
context["finish_reason"] in ENGINE_CUT_FINISH_REASONS(halo/inference/response.py): callself._recover_length_cutoff(trajectory). A turn with neither visible content nor a tool call: callself._recover_empty_turn(trajectory). Both nudge and retry withinmax_turnsand the sharedmax_length_cutoff_recoveriescap. - Declare
LENGTH_CUTOFF_NUDGEandEMPTY_TURN_NUDGE, or the base raisesNotImplementedErroron the first such turn. - A token-cap cut's unfinished tool calls arrive under
context[CUT_TOOL_CALLS_KEY], notcontext["tool_calls"]; never execute them. Passin_tool_call=bool(context.get(CUT_TOOL_CALLS_KEY))to_recover_length_cutoffso the turn counts underepisode/length_cutoff_in_call_turnsand getsLENGTH_CUTOFF_IN_CALL_NUDGE(the plain nudge if unset). - A native-protocol environment with no tools sends the
TOOLLESS_*nudges, which ask for the final answer. A subclass with its own wording declares those too.
- An engine cut arrives with
- For I/O-bound steps subclass
AsyncEnvironmentBaseand override_reset_single_async/_step_single_async; free per-episode resources in_release_episode, whichcleanupcalls after every episode, error and timeout paths included. - Probe your own backend. Extend
verify_backendand callsuper(), so a bad URL fails the launch instead of every episode.
Register it
Python
from halo.environments.registry import register_environment
register_environment("guess", lambda env_config: GuessEnvironment(**env_config))- Forward the whole
env_config: a factory that ignores it drops everyenvironment_kwargskey silently. - A name resolves against the factories registered in the process that builds the environment. A training run builds it inside each Ray actor, where the registry holds only what
halo/environments/registry.pyregisters at import: add yours there (_register_builtins), then setenvironment_type: guess. A registration made only in the entry script passes the startup check and then fails every rollout withUnknown environment type. - The eval scripts build the environment in their own process, so a module their wrapper imports is enough (Test it).
- From Python, skip the registry:
DistributedAsyncEnvironmentalGRPOTrainer(..., environment_cls=GuessEnvironment, environment_kwargs={...}); the class itself travels to the actors.
A custom tool
Python
from halo.environments.tools.definitions import NativeTool, NativeToolRegistry, ToolParameter
registry = NativeToolRegistry().register(NativeTool(
name="lookup",
description="Look up a record by id",
parameters=[ToolParameter(name="record_id", type="string", description="Record id")],
handler=lookup_record, # called as lookup_record(record_id="...")
))Pass the registry as NativeToolUseEnvironment(tool_registry=registry, ...).
- Calls are bound against the declared parameters before the handler runs; a refused call is a tool error that spends no tool budget (Argument binding). A tool with no declared parameters takes its arguments as given.
- A handler that raises fails the call and charges
tool_error_penalty. Return a string for a legitimate negative answer. - A sandbox-backed handler raises its sandbox fault instead (Sandbox faults).
Trainable trajectory shape
The trainer builds one row per assistant turn from the engine's sampled ids (Training rows). Four rules keep that possible:
- Let
step()append the assistant turn. Only it carries the sampled ids, logprobs and reasoning. One hand-appended assistant message sends the whole trajectory to a re-tokenization fallback that trains text the policy never emitted. - Give every advertised tool call a result message. Set
max_tool_calls_per_turnwhen the environment executes fewer calls than the model may request. - Return tool output as a message:
Message.tool(content, tool_call_id, name), or anObservation: …user turn for a text protocol. The loss mask comes from spans, not roles. - Keep
tool_callsrenderable. A trajectory the chat template cannot render, or whose turn spans cannot be located, is dropped as a masked row, not trained on a guess.
To log per-episode diagnostics, override rollout_metrics(trajectory) -> dict[str, float] and merge super().rollout_metrics(trajectory); keys log under their full paths (outcome/*, episode/*). A string stamped under a slice name in trajectory.info["slices"] ({"language": "cpp"}) splits the episode/* keys as <slice>/<value>/* (Logged metrics).
Dataset
Python
{"prompt": str | list[dict], "answer": Any, "difficulty": "hard"}answerand the columns named in the script'scontext_fieldsreachreset()/step()incontext; other columns are dropped. Acontext_fieldsentry (or a renamedanswer_field) that names no column raises at startup.requires_answer = Truemakes the trainer refuse a dataset withoutanswer(defaultFalse; per-run override inenvironment_kwargs).- A list-valued prompt reaches the environment as its last
usermessage only, so put framing in the environment's system prompt (system_prompton the protocol classes;_init_trajectory(system_prompt=...)on your own).
Test it
A CPU test, in the shape of tests/cpu/environments/test_environments.py:
Python
import pytest
from halo.configs.grpo_environmental import EnvironmentConfig
from halo.environments.registry import register_environment, build_environment
def test_guess_env_grades_a_correct_answer(isolated_registry):
register_environment("guess", lambda c: GuessEnvironment(**c))
env = build_environment("guess", EnvironmentConfig(environment_type="guess").to_env_config())
ids, _ = env.reset(["What is 2 + 2?"], [{"answer": "4"}])
step = env.step(ids, ["Final Answer: 4"], [{}])[0]
assert step.done
assert step.trajectory.total_reward == 1.0 # reward/objective 1.0 + reward/turn_shaping 0.0
if __name__ == "__main__":
raise SystemExit(pytest.main([__file__, "-v"]))- The
isolated_registryfixture unregisters the name afterwards. Run withpytest tests/cpu/environments -m cpu. - With an external reward term, call
env.settle(ids)before reading the reward.
Then run it end to end against a served model:
Bash
python scripts/environments/inference/eval_env.py \
--env_type guess --dataset <hf-id-or-dir> --split test \
--base_url http://localhost:8000/v1 --model <served-model> \
--num_examples 5 --save_trajectories /tmp/guess.jsonlThe script resolves --env_type through the same registry, so call its main() from a wrapper that imports your module first.
Related pages
- Environments — registry, common knobs, actor runtime.
- Native Tool-Use — the protocol most environments extend.
- Async GRPO with Environments — the trainer and its dataset surface.
Source on GitHubUpdated 18 November 2026