ReAct Environments
The model writes a Thought:, then an Action: or a Final Answer:, and the environment parses the action out of the plain text (ReAct, halo/environments/protocols/react.py).
Serve the model without a tool-call parser: one would lift the call out of the text, leaving no Action: line to parse.
Registry names, each with a fixed system prompt: react_math (calculate, python) and react_search (web_search). Shipped config: examples/grpo/environmental/qwen3_5/vllm/qwen3.6-35b-a3b-react-math-full-ep4.yaml.
Text
Thought: I need 25 * 4 first.
Action: calculate(expression="25 * 4")
Observation: 100
Final Answer: 110An action's arguments are read as a Python call of literals (a triple-quoted code argument arrives whole), else by a name=value pattern. Arguments neither reads bind as none, which a tool with a required argument refuses as a tool error. An Action: that spells no call gets the format hint. Neither is an episode error.
Configuration
YAML
environment_type: react_math
environment_kwargs:
thought_reward: 0.02The common knobs apply too. The native protocol's no_tool_use_penalty, turn_overflow_penalty and length_cutoff_penalty raise TypeError here.
Tools
calculate— restricted arithmetic and math functions, oneexpression.python— the in-process restricted REPL, imports blocked.web_search—queryand optionalmax_results(default 5).
Reward
The grade is 1 when the final answer matches, 0 otherwise, priced as reward/objective (Reward Terms).
- Matching is all-or-nothing: exact match after normalization (
\boxed{}, bold, answer prefixes), then a numeric compare at 1% relative tolerance. - An
answer_validator(final_answer, expected) -> boolgrades instead wherever one is passed, and clearsrequires_answer(From Python). Without one the presets require theanswercolumn. - A validator that raises marks the episode
episode_invalidand grades it 0.
Per-turn shaping, logged as reward/turn_shaping: +thought_reward or -no_thought_penalty for the Thought, and +tool_success_reward / -tool_error_penalty for the call, paid up to tool_reward_cap.
Refused and unproductive turns
- An
Action:naming an unregistered tool is a tool error whose observation lists the real tools; the turn trains only on a negative advantage (Untrainable turns). - An
Action:whose arguments do not bind (Argument binding) or past itstool_budgetscap is refused unrun:tool_error_penalty, no budget spent, and the turn trains only on a negative advantage. A cap of0disables the tool, and its turn stays trainable. - A call that ends on a sandbox fault is booked by its class and ends the episode (Sandbox faults).
- A cut or empty turn is neither executed nor graded. It is nudged and retried (Unproductive turns) and trains only on a negative advantage.
- Text with neither an Action nor a Final Answer gets the format hint and stays trainable.
Dataset
{"prompt": str | list[dict], "answer": str | number}; the answer column is required (requires_answer). Strip dataset markup from answer during preparation (answer grading).
Evaluation
Bash
python scripts/environments/inference/eval_env.py \
--env_type react_math --dataset <hf-id-or-dir> --split test \
--prompt_field prompt --answer_field answer \
--base_url http://localhost:8000/v1 --model <served-model> --num_examples 20Flags and the --training_config contract: Evaluation.
From Python
answer_validator is a callable, so only the Python path sets it; a YAML string raises:
Python
from halo.environments.protocols.react import build_react_math_environment
env = build_react_math_environment(max_turns=10, answer_validator=my_validator)Related pages
- Environments — registry and common knobs.
- Native Tool-Use — the function-calling protocol.
- Async GRPO with Environments — the trainer.
Source on GitHubUpdated 18 November 2026