Episodes

An episode is one multi-turn conversation: the engine samples an assistant turn, the environment answers it, and the loop runs until a final answer, max_turns, or a spent budget. Knobs are top-level AsyncRolloutConfig fields except carry_reasoning, max_length_cutoff_recoveries and length_cutoff_penalty (environment_kwargs), max_turns (EnvironmentConfig), and max_prompt_length, chat_template and force_chat_template (script arguments). Reasoning budgets have their own page.

Sampling

rollout_temperature, rollout_top_p, rollout_top_k, rollout_min_p and rollout_repetition_penalty set the engine's sampling; their ranges, and the cuts refused under a sequence-level ratio (Trust region masks), are in AsyncRolloutConfig. Rollout and eval requests send all five, so the model's generation_config.json defaults never apply. TRL's own sampling fields reach no sampler and are refused when set (Rollout backend).

Trajectory length

rollout_max_tokens caps one turn, reasoning and answer together, and rollout_max_episode_tokens what an episode's assistant turns sample together, recovery turns included; an episode that burns its max_turns (per environment) ends truncated. max_prompt_length drops dataset rows whose rendered prompt exceeds it, and max_train_row_tokens bounds a training row's memory.

Episode output budget

rollout_max_episode_tokens binds per turn and is never stated to the model.

  • A turn's max_tokens is the smaller of its own cap and what the episode has left. Its reasoning cap shrinks by as much, so the turn keeps its answer room.
  • An episode left with less than that room ends truncated, priced like a max_turns overflow. Watch episode/output_budget_exhausted.
  • It travels as max_tokens, so it holds on both engines.

Context window

The trajectory is never truncated: a row over the model's context window raises on every rank together. The startup check warns when the worst-case episode overflows the served window (Rollout backend); size rollout_max_episode_tokens so it fits. The check counts the environment's system prompt and tool schema as the chat template renders them, and a template that cannot render that preamble fails the run at startup.

An episode that grows past the served window closes on what it earned, as in an eval, so the longest failures keep their row and reward; episode/context_overflow counts them. SGLang refuses any request whose prompt plus max_tokens passes the window where vLLM clamps max_tokens, so there such an episode ends up to rollout_max_tokens earlier. Under train_on_sampled_tokens: false its whole render can pass the window, ending on a tool result no engine served, and the episode then leaves the batch instead of raising.

max_completion_length is no knob here: the script pins it to rollout_max_tokens (TRL reads it only as the dr_grpo normalizer) and raises on another explicit value.

Training-row cap

A per-turn row over max_train_row_tokens (prompt plus completion) leaves the batch while the episode's other turns train; a whole-trajectory row trains at zero weight. Setting the cap logs sampling/rows_over_cap_frac, the share of rows left out.

The episode budget leaves tool output between turns unbounded, so it does not replace this cap.

On an EP group spanning NVLink domains the cap is also the row length the load-time dispatch gate sizes, with per_device_train_batch_size rows (and per_device_eval_batch_size where an evaluation runs): unset, the gate judges the context window and refuses above the cross-node 8,192 tokens per rank (DeepEP).

Tool calls

Native-tool environments send their tools schema, so the server needs the matching --tool-call-parser (per-family values: Rollout Servers).

  • A server with no parser is refused at startup (Startup preflight).
  • A wrong parser leaves calls as text with no tool_calls, so each turn reads as a final answer and the episodes tie. The run raises once tool_parser_watch_rounds rounds (default 2) hold the template's call opening in a turn's text with no call parsed anywhere; one parsed call stands the watch down for the run (Tool parsers).
  • ReAct environments send no tools and are served without a parser.

rollout_stop_tokens (default empty) lists special-token strings sent as stop_token_ids. Use it for a tool-call terminator the model's eos_token_id omits: gpt-oss with harmony disabled ends a call with <|call|> and otherwise hallucinates the result (the gpt-oss recipes set ["<|call|>"]), so gpt-oss on a tool environment without it is refused at trainer construction. An unknown name raises there too.

Unproductive turns

A turn is unproductive when it is:

  • cut: the engine stopped it at its token cap;
  • empty: the model ended it with neither a tool call nor visible content;
  • rejected: every tool call named a nonexistent tool or was refused unrun (Untrainable turns).

What follows:

  • A cut or empty turn is nudged and retried within max_turns and max_length_cutoff_recoveries (null = no cap beyond max_turns), a cap the two kinds share. A rejected turn is answered with the refusals and spends a turn, not the cap.
  • Every cut turn counts under episode/length_cutoff_turns and every empty one under episode/empty_turns, the one that ends the episode included. Under the native protocol, a turn cut at its token cap with a tool call the server's parser salvaged also counts under episode/length_cutoff_in_call_turns.
  • The native protocol charges length_cutoff_penalty (default 0) per recovered cut or empty turn (Native tool-use).
  • A turn past the cap, on the episode's last turn, or on the last turn the output budget affords ends the episode truncated, priced like a max_turns overflow.
  • Their rows train only under a negative advantage (Untrainable turns).

The next turn runs under the same caps as any other. Under carried reasoning a retry thinks on from where the cut stopped, so a per-turn cap binds only once cuts are priced.

Engine aborts

An engine abort never reaches the environment: the actor re-issues the turn up to max_retries times (default 3), then the episode errors into a masked row.

Carried reasoning

carry_reasoning: true (default off) sends the latest assistant turn's reasoning back, so the next turn continues that thought. Earlier turns stay visible text, so a request grows by at most one turn's reasoning.

  • The trainer refuses it under rollout_backend: sglang (unverified there).
  • A step whose turns carry no reasoning while carried reasoning or a reasoning term is on warns once: the server most likely runs without a reasoning parser.
  • The template decides where a carried thought renders. The stock Qwen3.6 template renders it only on turns after the last user message, so a recovery nudge hides it, unless rollout_chat_template_kwargs: {preserve_thinking: true} is set; the stock Qwen3.5 template has no such variable. The shipped Qwen3.6 effort template always renders it.

Chat template

Two safe setups:

  • Built-in (default): no chat_template: on the trainer and no --chat-template on the server, so both load the checkpoint's.
  • Custom: chat_template: <file>.jinja plus force_chat_template: true on the trainer and --chat-template <same file> on the server, the file visible inside its container (/jinja-templates/… under either compose file). A relative path raises at server start, which restart: unless-stopped then loops.

Rows train on the engine's prompt ids, so a mismatch shows in no loss. The startup probe asks each server for its render of a no-tool prompt under two of the run's contracts: a server whose prompt does not move with the level, where the trainer's does, is refused, as the reasoning terms would charge for a contract the model never read. Any other difference warns with the first differing token, since an engine also derives a thinking toggle the trainer's render lacks.

A chat_template: that differs from a template the checkpoint ships raises without force_chat_template: true. With it, and no --chat-template on the server, the two sides diverge silently. The re-tokenization paths (Training rows) then score log-probs against a prompt the policy never generated under; per-turn rows take the engine's ids and cannot drift.

rollout_chat_template_kwargs applies template variables to every request and the trainer's own renders. It refuses reasoning_effort, which is per episode (Effort templates).

Source on GitHubUpdated 18 November 2026