RL environments for LLMs

RL environments are the tasks a language model acts in during reinforcement learning (RL). An environment hands the model a task and tools, runs its tool calls, keeps state between turns, and grades the finished episode.

How it works

An environment runs one task as a loop:

  1. It builds the conversation: the system prompt, the tool schema and the task.
  2. The model generates a turn.
  3. The environment parses the tool calls, executes them and returns what they produced.
  4. Steps 2 and 3 repeat until the model finishes or a turn limit is reached.
  5. The environment grades the whole trajectory and returns a reward.

The finished episode is one rollout. GRPO (Group Relative Policy Optimization) then compares the rewards of the episodes that shared a prompt.

What an environment holds

  • Tools. A calculator, a Python sandbox, web search.
  • State. A code workspace that survives across turns, for example.
  • A grader. An answer match, hidden tests, or a judge model.

In Halo

Async GRPO runs environments as Ray actors. environment_type picks one:

environment_typeThe task
react_math, react_searchReAct, with actions written as plain text: calculator and Python, or web search
native_math, native_coding, native_combinedthe same tools as native function calls
swean edit-run-test loop over a workspace
code_contests, codeforceswrite a program that passes every hidden test
mcpwhatever tools an MCP (Model Context Protocol) server advertises

Your own class registers alongside them. The environment list has the rest.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL