Monitoring
Async GRPO metrics are namespaced by the question they answer. The rollout, sampling/*, reward/* and routing/* metrics and the prefetch counters fold every rank's episodes or counts; timings are the main process's.
Logged metrics
Environments add task-specific outcome/* and reward/* keys through their rollout_metrics hook.
async/*
episode/*
Group and entropy diagnostics
sampling/*
routing/*
Slices
Each episode's resolved effort level slices the metrics as effort/<level>/*, and any string an environment stamps under trajectory.info["slices"] as <slice>/<value>/* (code contests, where the run offers a choice of languages: language/cpp/*). Each slice carries count, reward, generation_tokens, reasoning_tokens, turns, truncation_rate and solve_rate, plus the environment's own episode/* keys without that prefix (effort/high/test_calls).
Step timings
profiling/Time taken: <Trainer>.<name> (main process) includes rollout_acquire (the generation wait, near zero on a prefetch hit), build_training_tensors and weight_sync.
TRL's unnamespaced step_time times only its own training_step body, so it leaves out the clip and the optimizer step and, on the environmental trainer, the weight push, which runs before that body. The whole step is step_time_seconds (enable_efficiency_metrics: true) or the sum of the phase times.
Phase times
log_phase_times: true on the online and environmental trainers logs where each optimizer step's wall time goes, as phase_time/<phase>_s (the mean over the ranks) and phase_time/<phase>_max_s (the slowest rank), seconds per step averaged since the last log. Each phase is its own time, less the phases nested in it, so the keys add up to the step's wall time. The slowest-rank key shows skew the mean hides: an EP rank waiting on a hot peer, or one rank's slow rollout.
Phases run on CUDA events recorded on the current stream, so timing does not drain the queue; the report costs one all-reduce per log.
Reading the metrics
reward_stdis not the learning signal. Between-problem difficulty dominates it. The advantage usesreward/within_group_std(the mean per-group std): near zero means degenerate groups and no gradient (Advantages).episode/natural_termination_rateis not a solve rate. It is the share of episodes that reacheddoneuntruncated;outcome/solve_ratecomes from environments with a solve verdict (code contests).- Turn health is logged apart:
episode/length_cutoff_turns(turns cut at their token cap) andepisode/empty_turns(neither a tool call nor visible content), both unproductive turns, andepisode/reasoning_cjk_rate(reasoning drifted into CJK script). - A rising
rewardis task progress only whenreward/objectivemoves; shaping and external terms move it too. - The headline
reward,reward_stdandreward/within_group_stdcover valid episodes only;sampling/invalid_episode_fracreports the rest in train rounds. Thereward/*components, the reasoning charges and slice rewards also average invalid episodes (Batch construction).
Reward decomposition
The reward/* components separate objective from shaping (Environment arm):
The environment's components (all but the trainer's charges) sum exactly to its settled reward, so a nonzero composition_residue means a channel bypasses them. The headline reward adds the trainer's reasoning charges.
External terms
episode/reward_scoredbelow 1 means a judge or reward-model call reached no verdict;<source>/<name>/scorednames the term. The term scores 0, and itson_errordecides whether the episode stays in the baseline (Outages).judge/<name>/vetois the share of judged episodes a veto check fired on (Veto judge).
Saved trajectories
save_completions (default on) writes the rollouts since the previous log as one part per rank, <output_dir>/completions/completions_<step>/rank<NN>.parquet (the rank zero-padded to the world's width), each holding that rank's own episodes; a step's directory is its record and reads back whole with pd.read_parquet("completions/completions_00010"). Eval rounds write completions_<step>_eval/, whose parts together hold the whole eval set. Under TP or ETP each group's leader writes the group's episodes, once. A completions table is logged when wandb is in report_to.
Both JSON columns are {} when empty. TRL's log_completions controls only the console table, which prints num_completions_to_print rows. The console and wandb tables read a sample gathered to global rank 0, ceil(num_completions_to_print / world size) rows from each rank with the text cut to 8,000 characters a cell; leaving num_completions_to_print unset sends every row, so set it on a large run. The parquet parts keep every row and the full text.
Cut tool calls
A turn the engine cut mid-tool-call shows the partial call after its text, tagged cut by the engine before the turn closed; never run. On vLLM, with the turn's sampled ids captured and a reasoning budget resolving the reasoning close to a single token, it carries the whole decoded text past the close; elsewhere (SGLang, which enforces no reasoning cap, included), what the tool parser salvaged. It never runs, and later turns never see it.
Writer rank
Every rank writes its own part into its node's output_dir, so no rank gathers another's trajectories and a node's directory holds its own ranks' parts on per-node storage; only the tables' sample reaches global rank 0. A failed write (a full disk, a wandb outage) fails the step on every rank.
Halts
Three rules end a run whose rounds train on nothing or on a biased sample, each raising on every rank:
An episode that errored or outran request_timeout trains as a masked row and stays out of the baseline, so a run whose slow episodes time out learns from the fast ones alone, toward short answers; half the episodes failing three rounds running is far past a grader blip. The error rate is logged as episode/error_rate every round, and the startup preflight times a short generation per server and warns when a full rollout_max_tokens turn could not finish inside request_timeout (Startup preflight).
The counters of all three, and the early-stop streaks below, are written beside the pending prefetch rounds in each checkpoint and read back on a resume, so a restart continues the patience budget instead of granting a full one.
Early stop
Off unless a condition is set. A KL-free run drifts in either direction, toward entropy collapse or explosion, and both start slowly enough to stop on. Each condition counts its consecutive breaching readings; when one reaches early_stop_patience (default 3), training ends on a verdict taken across ranks.
- A reading is one training log line (the window mean under
logging_steps> 1). The gap and the breaker (table below) come once per generation round, so a step reusing its round leaves their count unchanged. - The stop skips that step's save and eval (an epoch save included), logs its reason, and exits non-zero on every rank. Resume from a checkpoint taken before the drift began.
TRL's entropy counts a micro-batch whose loss tokens were all dropped (degenerate groups, invalid episodes, padding rows) as 0, so under drops it reads low and moves with the dropped share.
Read the band and the gap off a healthy run; both are model-specific. The gap grows with entropy, and faster than it once the policy flattens, so it shows a drift before entropy does.
- The Qwen3.6 vLLM code-contests recipes stop on a band of
[0.12, 0.40]and a gap of0.005. - Every code-contests recipe stops on skipped updates.
Source on GitHubUpdated 18 November 2026