# Episodes

An episode is one multi-turn conversation: the engine samples an assistant turn, the environment answers it, and the
loop runs until a final answer, `max_turns`, or a spent budget. Knobs are top-level
[`AsyncRolloutConfig`](https://whitecircle.com/halo/docs/agent-docs/reference/configuration-reference.md#asyncrolloutconfig) fields except
`carry_reasoning`, `max_length_cutoff_recoveries` and `length_cutoff_penalty` (`environment_kwargs`), `max_turns`
(`EnvironmentConfig`), and `max_prompt_length`, `chat_template` and `force_chat_template` (script arguments).
Reasoning budgets have [their own page](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/reasoning.md).

## Sampling

`rollout_temperature`, `rollout_top_p`, `rollout_top_k`, `rollout_min_p` and `rollout_repetition_penalty` set the
engine's sampling; their ranges, and the cuts refused under a sequence-level ratio
([Trust region masks](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/objective.md#trust-region-masks)), are in [AsyncRolloutConfig](https://whitecircle.com/halo/docs/agent-docs/reference/configuration-reference.md#asyncrolloutconfig).
Rollout and eval requests send all five, so the model's `generation_config.json` defaults never apply. TRL's own
sampling fields reach no sampler and are refused when set ([Rollout backend](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/setup.md#rollout-backend)).

## Trajectory length

`rollout_max_tokens` caps **one turn**, reasoning and answer together, and `rollout_max_episode_tokens` what an
**episode's** assistant turns sample together, recovery turns included; an episode that burns its `max_turns`
([per environment](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/environments.md#registry)) ends truncated. `max_prompt_length` drops dataset rows whose
rendered prompt exceeds it, and `max_train_row_tokens` bounds a training row's memory.

### Episode output budget

`rollout_max_episode_tokens` binds per turn and is never stated to the model.

- A turn's `max_tokens` is the smaller of its own cap and what the episode has left. Its reasoning cap shrinks by as
  much, so the turn keeps its [answer room](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/reasoning.md#answer-room).
- An episode left with less than that room ends truncated, priced like a `max_turns` overflow. Watch
  `episode/output_budget_exhausted`.
- It travels as `max_tokens`, so it holds on both engines.

### Context window

The trajectory is **never truncated**: a row over the model's context window raises on every rank together. The
startup check warns when the worst-case episode overflows the served window
([Rollout backend](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/setup.md#rollout-backend)); size `rollout_max_episode_tokens` so it fits. The check counts the
environment's system prompt and tool schema as the chat template renders them, and a template that cannot render
that preamble fails the run at startup.

An episode that grows past the served window closes on what it earned, as in an [eval](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/evaluation.md), so the longest
failures keep their row and reward; `episode/context_overflow` counts them. SGLang refuses any request whose prompt plus
`max_tokens` passes the window where vLLM clamps `max_tokens`, so there such an episode ends up to `rollout_max_tokens`
earlier. Under `train_on_sampled_tokens: false` its whole render can pass the window, ending on a tool result no engine
served, and the episode then leaves the batch instead of raising.

`max_completion_length` is no knob here: the script pins it to `rollout_max_tokens` (TRL reads it only as the
`dr_grpo` normalizer) and raises on another explicit value.

### Training-row cap

A per-turn row over `max_train_row_tokens` (prompt plus completion) leaves the batch while the episode's other turns
train; a whole-trajectory row trains at zero weight. Setting the cap logs `sampling/rows_over_cap_frac`, the share of
rows left out.

The episode budget leaves tool output between turns unbounded, so it does not replace this cap.

On an EP group spanning NVLink domains the cap is also the row length the load-time dispatch gate sizes, with `per_device_train_batch_size` rows (and `per_device_eval_batch_size` where an evaluation runs): unset, the gate judges the context window and refuses above the cross-node 8,192 tokens per rank ([DeepEP](https://whitecircle.com/halo/docs/agent-docs/infrastructure/deepep.md#dispatch-token-ceiling)).

## Tool calls

Native-tool environments send their `tools` schema, so the server needs the matching
`--tool-call-parser` (per-family values: [Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#running-it)).

- A server with no parser is refused at startup ([Startup preflight](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#startup-preflight)).
- A wrong parser leaves calls as text with no `tool_calls`, so each turn reads as a final answer and the episodes tie. The run raises once `tool_parser_watch_rounds` rounds (default `2`) hold the template's call opening in a turn's text with no call parsed anywhere; one parsed call stands the watch down for the run ([Tool parsers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#running-it)).
- [ReAct](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/environments/protocols/react.md) environments send no `tools` and are served without a parser.

`rollout_stop_tokens` (default empty) lists special-token strings sent as `stop_token_ids`. Use it for a tool-call
terminator the model's `eos_token_id` omits: gpt-oss with harmony disabled ends a call with `<|call|>` and otherwise
hallucinates the result (the gpt-oss recipes set `["<|call|>"]`), so gpt-oss on a tool environment without it is
refused at trainer construction. An unknown name raises there too.

## Unproductive turns

A turn is unproductive when it is:

- **cut**: the engine stopped it at its token cap;
- **empty**: the model ended it with neither a tool call nor visible content;
- **rejected**: every tool call named a nonexistent tool or was refused unrun
  ([Untrainable turns](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/environments/protocols/native-tool-use.md#untrainable-turns)).

What follows:

- A cut or empty turn is nudged and retried within `max_turns` and `max_length_cutoff_recoveries` (`null` = no cap
  beyond `max_turns`), a cap the two kinds share. A rejected turn is answered with the refusals and spends a turn, not
  the cap.
- Every cut turn counts under `episode/length_cutoff_turns` and every empty one under `episode/empty_turns`, the one
  that ends the episode included. Under the native protocol, a turn cut at its token cap with a tool call the
  server's parser salvaged also counts under `episode/length_cutoff_in_call_turns`.
- The native protocol charges `length_cutoff_penalty` (default `0`) per recovered cut or empty turn
  ([Native tool-use](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/environments/protocols/native-tool-use.md#reward)).
- A turn past the cap, on the episode's last turn, or on the last turn the output budget affords ends the episode
  truncated, priced like a `max_turns` overflow.
- Their rows train only under a negative advantage ([Untrainable turns](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/objective.md#untrainable-turns)).

The next turn runs under the same caps as any other. Under [carried reasoning](#carried-reasoning) a retry thinks on
from where the cut stopped, so a per-turn cap binds only once cuts are priced.

### Engine aborts

An engine abort never reaches the environment: the actor re-issues the turn up to `max_retries` times (default `3`),
then the episode errors into a masked row.

## Carried reasoning

`carry_reasoning: true` (default off) sends the latest assistant turn's reasoning back, so the next turn continues
that thought. Earlier turns stay visible text, so a request grows by at most one turn's reasoning.

- The trainer refuses it under `rollout_backend: sglang` (unverified there).
- A step whose turns carry no reasoning while carried reasoning or a reasoning term is on warns once: the server most
  likely runs without a reasoning parser.
- The template decides where a carried thought renders. The stock Qwen3.6 template renders it only on turns after
  the last user message, so a recovery nudge hides it, unless `rollout_chat_template_kwargs: {preserve_thinking: true}`
  is set; the stock Qwen3.5 template has no such variable. The shipped Qwen3.6 effort template always renders it.

## Chat template

Two safe setups:

- **Built-in** (default): no `chat_template:` on the trainer and no `--chat-template` on the server, so both load the
  checkpoint's.
- **Custom**: `chat_template: <file>.jinja` plus `force_chat_template: true` on the trainer and
  `--chat-template <same file>` on the server, the file visible inside its container (`/jinja-templates/…` under
  either compose file). A relative path raises at server start, which `restart: unless-stopped` then loops.

Rows train on the engine's prompt ids, so a mismatch shows in no loss. The startup probe asks each server for its render
of a no-tool prompt under two of the run's contracts: a server whose prompt does not move with the level, where the
trainer's does, is refused, as the reasoning terms would charge for a contract the model never read. Any other
difference warns with the first differing token, since an engine also derives a thinking toggle the trainer's render
lacks.

A `chat_template:` that differs from a template the checkpoint ships raises without `force_chat_template: true`. With
it, and no `--chat-template` on the server, the two sides diverge silently. The re-tokenization paths
([Training rows](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/objective.md#training-rows)) then score log-probs against a prompt the policy never generated under;
per-turn rows take the engine's ids and cannot drift.

`rollout_chat_template_kwargs` applies template variables to every request **and** the trainer's own renders. It
refuses `reasoning_effort`, which is per episode ([Effort templates](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/reasoning.md#effort-templates)).
