Episodes
An episode is one multi-turn conversation: the engine samples an assistant turn, the environment answers it, and the
loop runs until a final answer, max_turns, or a spent budget. Knobs are top-level
AsyncRolloutConfig fields except
carry_reasoning, max_length_cutoff_recoveries and length_cutoff_penalty (environment_kwargs), max_turns
(EnvironmentConfig), and max_prompt_length, chat_template and force_chat_template (script arguments).
Reasoning budgets have their own page.
Sampling
rollout_temperature, rollout_top_p, rollout_top_k, rollout_min_p and rollout_repetition_penalty set the
engine's sampling; their ranges, and the cuts refused under a sequence-level ratio
(Trust region masks), are in AsyncRolloutConfig.
Rollout and eval requests send all five, so the model's generation_config.json defaults never apply. TRL's own
sampling fields reach no sampler and are refused when set (Rollout backend).
Trajectory length
rollout_max_tokens caps one turn, reasoning and answer together, and rollout_max_episode_tokens what an
episode's assistant turns sample together, recovery turns included; an episode that burns its max_turns
(per environment) ends truncated. max_prompt_length drops dataset rows whose
rendered prompt exceeds it, and max_train_row_tokens bounds a training row's memory.
Episode output budget
rollout_max_episode_tokens binds per turn and is never stated to the model.
- A turn's
max_tokensis the smaller of its own cap and what the episode has left. Its reasoning cap shrinks by as much, so the turn keeps its answer room. - An episode left with less than that room ends truncated, priced like a
max_turnsoverflow. Watchepisode/output_budget_exhausted. - It travels as
max_tokens, so it holds on both engines.
Context window
The trajectory is never truncated: a row over the model's context window raises on every rank together. The
startup check warns when the worst-case episode overflows the served window
(Rollout backend); size rollout_max_episode_tokens so it fits. The check counts the
environment's system prompt and tool schema as the chat template renders them, and a template that cannot render
that preamble fails the run at startup.
An episode that grows past the served window closes on what it earned, as in an eval, so the longest
failures keep their row and reward; episode/context_overflow counts them. SGLang refuses any request whose prompt plus
max_tokens passes the window where vLLM clamps max_tokens, so there such an episode ends up to rollout_max_tokens
earlier. Under train_on_sampled_tokens: false its whole render can pass the window, ending on a tool result no engine
served, and the episode then leaves the batch instead of raising.
max_completion_length is no knob here: the script pins it to rollout_max_tokens (TRL reads it only as the
dr_grpo normalizer) and raises on another explicit value.
Training-row cap
A per-turn row over max_train_row_tokens (prompt plus completion) leaves the batch while the episode's other turns
train; a whole-trajectory row trains at zero weight. Setting the cap logs sampling/rows_over_cap_frac, the share of
rows left out.
The episode budget leaves tool output between turns unbounded, so it does not replace this cap.
On an EP group spanning NVLink domains the cap is also the row length the load-time dispatch gate sizes, with per_device_train_batch_size rows (and per_device_eval_batch_size where an evaluation runs): unset, the gate judges the context window and refuses above the cross-node 8,192 tokens per rank (DeepEP).
Tool calls
Native-tool environments send their tools schema, so the server needs the matching
--tool-call-parser (per-family values: Rollout Servers).
- A server with no parser is refused at startup (Startup preflight).
- A wrong parser leaves calls as text with no
tool_calls, so each turn reads as a final answer and the episodes tie. The run raises oncetool_parser_watch_roundsrounds (default2) hold the template's call opening in a turn's text with no call parsed anywhere; one parsed call stands the watch down for the run (Tool parsers). - ReAct environments send no
toolsand are served without a parser.
rollout_stop_tokens (default empty) lists special-token strings sent as stop_token_ids. Use it for a tool-call
terminator the model's eos_token_id omits: gpt-oss with harmony disabled ends a call with <|call|> and otherwise
hallucinates the result (the gpt-oss recipes set ["<|call|>"]), so gpt-oss on a tool environment without it is
refused at trainer construction. An unknown name raises there too.
Unproductive turns
A turn is unproductive when it is:
- cut: the engine stopped it at its token cap;
- empty: the model ended it with neither a tool call nor visible content;
- rejected: every tool call named a nonexistent tool or was refused unrun (Untrainable turns).
What follows:
- A cut or empty turn is nudged and retried within
max_turnsandmax_length_cutoff_recoveries(null= no cap beyondmax_turns), a cap the two kinds share. A rejected turn is answered with the refusals and spends a turn, not the cap. - Every cut turn counts under
episode/length_cutoff_turnsand every empty one underepisode/empty_turns, the one that ends the episode included. Under the native protocol, a turn cut at its token cap with a tool call the server's parser salvaged also counts underepisode/length_cutoff_in_call_turns. - The native protocol charges
length_cutoff_penalty(default0) per recovered cut or empty turn (Native tool-use). - A turn past the cap, on the episode's last turn, or on the last turn the output budget affords ends the episode
truncated, priced like a
max_turnsoverflow. - Their rows train only under a negative advantage (Untrainable turns).
The next turn runs under the same caps as any other. Under carried reasoning a retry thinks on from where the cut stopped, so a per-turn cap binds only once cuts are priced.
Engine aborts
An engine abort never reaches the environment: the actor re-issues the turn up to max_retries times (default 3),
then the episode errors into a masked row.
Carried reasoning
carry_reasoning: true (default off) sends the latest assistant turn's reasoning back, so the next turn continues
that thought. Earlier turns stay visible text, so a request grows by at most one turn's reasoning.
- The trainer refuses it under
rollout_backend: sglang(unverified there). - A step whose turns carry no reasoning while carried reasoning or a reasoning term is on warns once: the server most likely runs without a reasoning parser.
- The template decides where a carried thought renders. The stock Qwen3.6 template renders it only on turns after
the last user message, so a recovery nudge hides it, unless
rollout_chat_template_kwargs: {preserve_thinking: true}is set; the stock Qwen3.5 template has no such variable. The shipped Qwen3.6 effort template always renders it.
Chat template
Two safe setups:
- Built-in (default): no
chat_template:on the trainer and no--chat-templateon the server, so both load the checkpoint's. - Custom:
chat_template: <file>.jinjaplusforce_chat_template: trueon the trainer and--chat-template <same file>on the server, the file visible inside its container (/jinja-templates/…under either compose file). A relative path raises at server start, whichrestart: unless-stoppedthen loops.
Rows train on the engine's prompt ids, so a mismatch shows in no loss. The startup probe asks each server for its render of a no-tool prompt under two of the run's contracts: a server whose prompt does not move with the level, where the trainer's does, is refused, as the reasoning terms would charge for a contract the model never read. Any other difference warns with the first differing token, since an engine also derives a thinking toggle the trainer's render lacks.
A chat_template: that differs from a template the checkpoint ships raises without force_chat_template: true. With
it, and no --chat-template on the server, the two sides diverge silently. The re-tokenization paths
(Training rows) then score log-probs against a prompt the policy never generated under;
per-turn rows take the engine's ids and cannot drift.
rollout_chat_template_kwargs applies template variables to every request and the trainer's own renders. It
refuses reasoning_effort, which is per episode (Effort templates).
Source on GitHubUpdated 18 November 2026