Async GRPO with Environments
Multi-turn RL: the model converses with an environment (tools, code, repositories, search) until a final reward, then trains with GRPO. For pre-collected rewards use Offline GRPO; for single-turn verifiable rewards, Online GRPO; for pairwise preferences, SMPO or DPO.
- Trainer
DistributedAsyncEnvironmentalGRPOTrainer, scriptscripts/training/grpo/environmental.py. - Pick an environment with
environment_typeor register your own (Environments). - Reward Terms defines how the episode reward is composed.
- A rollout server must be serving before you launch; Ray configures itself on one node.
How a step works
- The trainer pushes its weights to the engines over NCCL, then hands one prompt per batch row to
RolloutManager, which dispatches round-robin across Ray actors and server URLs. - Each actor holds its own environment instance: POST
/v1/chat/completions, parse the reply and itstool_calls, run the step, repeat until done ormax_turns. - The environment grades each trajectory; the trainer builds training rows from the engine's sampled tokens and computes the GRPO loss.

Configuration surface
One YAML, six dataclasses.
Dataset
prompt is the row; answer is carried whenever the dataset has it. Where the environment requires answer, a
dataset without it is refused at trainer construction:
code_contests/codeforces(hidden-test payload),exam_qa,qa_searchand thereact_*presets (graded final answer) require it, as doesswewithout atest_function, unless judge-only;native_*andmcpneed none.
Rename columns with prompt_field / answer_field and forward extras with context_fields. A context_fields
entry, or an answer_field renamed away from answer, that names no column raises at startup.
- A message-list
promptreduces to its lastuserturn in training and the eval runner alike, since the environment builds the conversation itself. A content-part list hands over itstextparts concatenated. A conversation with nouserturn, or whose lastuserturn carries any other part, fails the batch on every rank; the script refuses an image part earlier, while it processes the dataset. - The trainer forces
remove_unused_columns: false(warning when a config sets it true): pruning would stripanswerand thecontext_fieldscolumns. - The environment owns the system turn and the tool schema: there is no
system_promptfield andtools_fieldis rejected. An override, where accepted, isenvironment_kwargs.system_prompt; the ReAct presets refuse it. text_only_modelis rejected: the sync forwards names verbatim, and the text-only sibling'smodel.layers.*would miss a multimodal server'smodel.language_model.layers.*.
Quickstart
Point the template's dataset: at your data, start a server on a GPU the trainer will not use, wait for /health,
launch. The template runs a ReAct environment, which is served without a tool-call parser
(VLLM_TOOL_CALLING_FLAGS=). The vLLM compose file needs HALO_SCRATCH, a large host volume, exported or set in the
repo-root .env:
Bash
make build-vllm # or set VLLM_IMAGE to the public image
VLLM_MODEL=Qwen/Qwen3-4B-Instruct-2507 VLLM_CUDA_DEVICES=7 VLLM_TOOL_CALLING_FLAGS= \
docker compose -f docker-compose.vllm.yml up -d vllm-server
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6 torchrun --nproc_per_node=7 \
scripts/training/grpo/environmental.py \
examples/grpo/environmental/environmental-grpo-template.yamlCheckpoints land in <output_dir>/checkpoint-<step>/, rollouts in <output_dir>/completions/
(Saved trajectories). Launch options: Servers and Launch.
From Python
Python
trainer = DistributedAsyncEnvironmentalGRPOTrainer(
model=model,
args=grpo_config, # TRL GRPOConfig
train_dataset=train_dataset,
processing_class=tokenizer,
environment_config=EnvironmentConfig(environment_type="react_math"),
async_config=AsyncRolloutConfig(rollout_server_url="http://localhost:8000"),
parallelism_config=ParallelismConfig(), # required; the default is plain data parallelism
peft_config=lora_config, # optional
)For an unregistered EnvironmentBase subclass, pass environment_cls=MyEnvironment with environment_kwargs={...}
instead. One of the two is required.
Testing a setup
Run the environment on a few rows first. --training_config samples under the training YAML's own contract
(Smoke-testing a config).
Bash
python scripts/environments/inference/eval_env.py \
--training_config examples/grpo/environmental/environmental-grpo-template.yaml \
--dataset <hub-id> --split test --num_examples 20 \
--base_url http://localhost:8000/v1 --model <served-model-id>pytest tests/cpu/environments tests/cpu/grpo -m cpucovers the construction gates and row accounting.make test-gpu-vllm/make test-gpu-sglangrun the end-to-end suites against a live server.
Pages
- Servers and Launch: GPU split, backends, weight sync, multi-node
- Episodes: sampling, turn and episode budgets, tool calls, unproductive turns
- Reasoning Budgets: effort levels, the engine's reasoning cap, the per-level budget and price
- Objective and Stability: training rows, importance sampling, advantages, KL
- Memory and Throughput: the logits wall, chunked log-probs, sizing
- Monitoring: logged metrics, saved trajectories, early stop
- Evaluation: eval during training, the offline runner, re-grading
- Troubleshooting: symptoms and fixes
- Environments: the registered tasks, tools and rewards
Source on GitHubUpdated 18 November 2026