# RL environments for LLMs

RL environments are the tasks a language model acts in during reinforcement learning (RL). An environment hands the model a task and tools, runs its tool calls, keeps state between turns, and grades the finished episode.

## How it works

An environment runs one task as a loop:

1. It builds the conversation: the system prompt, the tool schema and the task.
2. The model generates a turn.
3. The environment parses the tool calls, executes them and returns what they produced.
4. Steps 2 and 3 repeat until the model finishes or a turn limit is reached.
5. The environment grades the whole trajectory and returns a reward.

The finished episode is one [rollout](/glossary/rollouts). [GRPO](/glossary/grpo) (Group Relative Policy Optimization) then compares the rewards of the episodes that shared a prompt.

## What an environment holds

- **Tools.** A calculator, a Python sandbox, web search.
- **State.** A code workspace that survives across turns, for example.
- **A grader.** An answer match, hidden tests, or a judge model.

## In Halo

[Async GRPO](/glossary/async-grpo) runs environments as Ray actors. `environment_type` picks one:

| `environment_type` | The task |
| --- | --- |
| `react_math`, `react_search` | ReAct, with actions written as plain text: calculator and Python, or web search |
| `native_math`, `native_coding`, `native_combined` | the same tools as native function calls |
| `swe` | an edit-run-test loop over a workspace |
| `code_contests`, `codeforces` | write a program that passes every hidden test |
| `mcp` | whatever tools an MCP (Model Context Protocol) server advertises |

Your own class registers alongside them. The [environment list](/halo/docs/training-methods/async-grpo-environments#the-environments) has the rest.

