RL environments are the tasks a language model acts in during reinforcement learning (RL). An environment hands the model a task and tools, runs its tool calls, keeps state between turns, and grades the finished episode.
How it works
An environment runs one task as a loop:
- It builds the conversation: the system prompt, the tool schema and the task.
- The model generates a turn.
- The environment parses the tool calls, executes them and returns what they produced.
- Steps 2 and 3 repeat until the model finishes or a turn limit is reached.
- The environment grades the whole trajectory and returns a reward.
The finished episode is one rollout. GRPO (Group Relative Policy Optimization) then compares the rewards of the episodes that shared a prompt.
What an environment holds
- Tools. A calculator, a Python sandbox, web search.
- State. A code workspace that survives across turns, for example.
- A grader. An answer match, hidden tests, or a judge model.
In Halo
Async GRPO runs environments as Ray actors. environment_type picks one:
Your own class registers alongside them. The environment list has the rest.
