Monitoring

Async GRPO metrics are namespaced by the question they answer. The rollout, sampling/*, reward/* and routing/* metrics and the prefetch counters fold every rank's episodes or counts; timings are the main process's.

Logged metrics

NamespaceAnswersKeys to watch
async/*rollout throughputmean_rollout_latency; job totals total_rollouts, cumulative_mean_rollout_latency, total_generation_tokens; prefetch_hit_rate, prefetch_hits / prefetch_misses (prefetching runs only); requests_expired_in_sync
episode/*agent behaviorturns, generation_tokens (+_max, _p90), natural_termination_rate, truncation_rate, truncation_alarm, error_rate, length_cutoff_turns, length_cutoff_in_call_turns, empty_turns, context_overflow, thinking_cap_turns, output_budget_exhausted, reasoning_cjk_rate, tool_calls, sandbox_infra_fault, sandbox_agent_fault, reward_scored
outcome/*task successsolve_rate, all_pass_group_frac / all_fail_group_frac, eval success@k, plus per-env keys such as test_pass_frac
reward/*reward and decompositionbare reward / reward_std; within_group_std; the components and composition_residue (Reward decomposition)
judge/*, reward_model/*external reward terms<source>/<name>/scored plus per-source keys: judge, reward model
logps/*policy confidencesampling_mean, advantage_cov; the related kl, entropy, kl_clamp_frac are unnamespaced
advantage/*the round's push on the sampled tokensnet_token_mass, token_mass_scale, negative_only_mass (Token-mass balance)
sampling/*engine/trainer agreementis_correction_active, is_weights_from_loss_forward, logratio_mean, is_ratio_mean / is_ratio_max, is_ratio_extreme_frac, is_ess_frac, is_correction_coverage, is_masked_frac, sampler_certain_frac, update_skipped, degenerate_group_frac, invalid_episode_frac, rows_over_cap_frac, forced_close_frac, untrainable_rows_frac, untrainable_rows_trained_frac, untrainable_turns_rowless_frac, zero_gradient_round
score_centering/*the centered gradient under score_centering_top_kactive_frac, head_mass_mean, tail_ratio_mean, weight_zero_frac, correction_mean (Score centering)
routing/*MoE routing replayreplay_flip_rate; under R3 (rollout replay), the capture shape classes rollout_full_frac, rollout_engine_omits_last_frac, rollout_completion_only_frac, rollout_unresolved_frac, plus rollout_prompt_len_mismatch_frac
unnamespacedTRL loss internalsclip_ratio/*, cispo_clip_ratio (under loss_type: cispo), step_time

Environments add task-specific outcome/* and reward/* keys through their rollout_metrics hook.

async/*

MetricMeaning
prefetch_hits / prefetch_missesTraining rounds, summed over the world's ranks, whose prefetched round was delivered when the round started, against rounds that waited for it, found it failed, or ran cold (a run's first prefetch_depth rounds)
prefetch_input_skipsLogged once non-zero: prefetch submissions (each a round's prompts) skipped because a wedged worker's input queue stayed full; those prompts never train
requests_expired_in_syncPer-round world count of request deadlines that expired across a weight sync (pause credited), each discarding a started turn. Nonzero: raise request_timeout

episode/*

MetricMeaning
thinking_cap_turnsTurns whose counted reasoning reached a vLLM cap (Thinking-cap metric)
output_budget_exhaustedUnder rollout_max_episode_tokens only: episodes whose output budget no longer held a turn's answer room (Trajectory length)
truncation_alarm1 in a round whose truncation_rate exceeds truncation_alarm_rate, which also warns each time the rate crosses it

Group and entropy diagnostics

MetricMeaning
outcome/all_pass_group_frac / all_fail_group_fracGroups every / no valid member solved, by the environment's solve verdict (absent without one). High values: the task is too easy / too hard
outcome/reasoning_terms_waived_fracTraining episodes whose group earned no objective, so neither reasoning term was charged (Failed groups). It counts episodes by the settled objective, which a veto zeroes, so it can exceed all_fail_group_frac. The train reward/reasoning_* means average in the waived zeros
sampling/zero_gradient_round1 in a train round in which no token anywhere carries a policy gradient. Tied rewards with no tool calls usually mean the server's parser leaves them as text; tied rewards with calls mean the environment scores every attempt alike. A launch that has not trained yet raises at zero_gradient_round_limit of them in a row; past its first trained round a streak that long only warns
sampling/degenerate_group_fracGroups whose valid members' settled environment rewards are all equal, or with fewer than two valid members; logged in train rounds under drop_degenerate_groups
logps/advantage_covCovariance of each loss token's pre-update log-prob with its advantage. It pools contexts, so its sign need not match the entropy trend, which advantage/net_token_mass predicts

sampling/*

MetricMeaning
logratio_meanUnclamped mean log-ratio in nats, near 0 when healthy (IS correction)
is_correction_coverageShare of loss tokens that carry a correction: covered by the engine's sampling log-probs and not emitted with probability 1
is_masked_fracShare of those a mask stage zeroed
is_ratio_extreme_fracShare of corrected tokens whose raw ratio lies outside [1/c, c] before any mask, c being vllm_importance_sampling_clip_max
is_ess_fracNormalized effective sample size (Σw)² / (n Σw²) of the applied IS weights; it falls as a few tokens carry the correction
is_ratio_mean / is_ratio_maxThe ratios that survive the mask stages; under score centering the 0/1 keep verdict, 1 on every kept token, as the centered term carries the weight (Score centering)
is_weights_from_loss_forward1 on a round whose IS weights came from the loss forward, 0 on one a no-grad pass weighed (Weight from the loss forward). On a 1 round logratio_mean, is_ratio_* and is_ess_frac read loss tokens only and is_masked_frac is absent (no mask stage can run)
sampler_certain_fracTokens the engine emitted with probability 1, which carry no correction (a forced reasoning close instead gets ratio 0, forced_close_frac)

routing/*

MetricMeaning
rollout_unresolved_fracThe R3 shape class that says replay is degrading; a rank whose routed rows are all unresolved fails the step
rollout_prompt_len_mismatch_fracOutside the shape classes' denominator

Slices

Each episode's resolved effort level slices the metrics as effort/<level>/*, and any string an environment stamps under trajectory.info["slices"] as <slice>/<value>/* (code contests, where the run offers a choice of languages: language/cpp/*). Each slice carries count, reward, generation_tokens, reasoning_tokens, turns, truncation_rate and solve_rate, plus the environment's own episode/* keys without that prefix (effort/high/test_calls).

Step timings

profiling/Time taken: <Trainer>.<name> (main process) includes rollout_acquire (the generation wait, near zero on a prefetch hit), build_training_tensors and weight_sync.

TRL's unnamespaced step_time times only its own training_step body, so it leaves out the clip and the optimizer step and, on the environmental trainer, the weight push, which runs before that body. The whole step is step_time_seconds (enable_efficiency_metrics: true) or the sum of the phase times.

Phase times

log_phase_times: true on the online and environmental trainers logs where each optimizer step's wall time goes, as phase_time/<phase>_s (the mean over the ranks) and phase_time/<phase>_max_s (the slowest rank), seconds per step averaged since the last log. Each phase is its own time, less the phases nested in it, so the keys add up to the step's wall time. The slowest-rank key shows skew the mean hides: an EP rank waiting on a hot peer, or one rank's slow rollout.

PhaseTime between
rolloutthe generation wait: the environmental trainer's round acquisition (near zero on a prefetch hit), online GRPO's engine generation
syncthe engine weight push
buildthe rest of building a round's training tensors, the no-grad forwards aside
no_grad, no_grad_refthe policy's and the KL reference's no-grad forwards on the environmental trainer, which times each of its two passes. Online GRPO reports every no-grad scoring pass, TRL's old-policy and reference forwards, as no_grad
loss_forward_compute_loss, summed over micro-steps
post_backward_finishcompleting FSDP2's post-backward after each micro-step
micro_stepthe rest of each training_step: mostly the backward
reducethe last micro-step's end to the optimizer: the clip, the norm all-reduce, the deferred EP and TP gradient sweeps
optimizerthe optimizer step
log_emitthe log call, completions record included
unaccountedthe step's wall time no phase holds (an untimed phase, an eval round, a checkpoint save); negative where spans overlap that should nest

Phases run on CUDA events recorded on the current stream, so timing does not drain the queue; the report costs one all-reduce per log.

Reading the metrics

  • reward_std is not the learning signal. Between-problem difficulty dominates it. The advantage uses reward/within_group_std (the mean per-group std): near zero means degenerate groups and no gradient (Advantages).
  • episode/natural_termination_rate is not a solve rate. It is the share of episodes that reached done untruncated; outcome/solve_rate comes from environments with a solve verdict (code contests).
  • Turn health is logged apart: episode/length_cutoff_turns (turns cut at their token cap) and episode/empty_turns (neither a tool call nor visible content), both unproductive turns, and episode/reasoning_cjk_rate (reasoning drifted into CJK script).
  • A rising reward is task progress only when reward/objective moves; shaping and external terms move it too.
  • The headline reward, reward_std and reward/within_group_std cover valid episodes only; sampling/invalid_episode_frac reports the rest in train rounds. The reward/* components, the reasoning charges and slice rewards also average invalid episodes (Batch construction).

Reward decomposition

The reward/* components separate objective from shaping (Environment arm):

ComponentWhat it is
reward/objectiveThe environment's grade, priced by its term
reward/turn_shaping, reward/tool_shapingPer-turn and episode-level shaping
reward/<name>The environment's own terms (reward/submission, …) and each judge or reward_model term
reward/reasoning_floor, reward/reasoning_priceThe trainer's charges, each while its term is on (Reasoning length reward)
reward/composition_residueMean absolute gap between the components and the settled reward

The environment's components (all but the trainer's charges) sum exactly to its settled reward, so a nonzero composition_residue means a channel bypasses them. The headline reward adds the trainer's reasoning charges.

External terms

  • episode/reward_scored below 1 means a judge or reward-model call reached no verdict; <source>/<name>/scored names the term. The term scores 0, and its on_error decides whether the episode stays in the baseline (Outages).
  • judge/<name>/veto is the share of judged episodes a veto check fired on (Veto judge).

Saved trajectories

save_completions (default on) writes the rollouts since the previous log as one part per rank, <output_dir>/completions/completions_<step>/rank<NN>.parquet (the rank zero-padded to the world's width), each holding that rank's own episodes; a step's directory is its record and reads back whole with pd.read_parquet("completions/completions_00010"). Eval rounds write completions_<step>_eval/, whose parts together hold the whole eval set. Under TP or ETP each group's leader writes the group's episodes, once. A completions table is logged when wandb is in report_to.

ColumnHolds
completionDetokenized message text, whatever train_on_sampled_tokens
environment_rewardThe episode's reward; less the reward_components sum, it is what the trainer's reasoning terms charged
advantageThe scaled value the trajectory's trainable rows carry
reward_componentsThe settled reward/* components, as JSON
reward_detailsEach scored term's rationale by term name, such as a veto judge's fired checks and quotes

Both JSON columns are {} when empty. TRL's log_completions controls only the console table, which prints num_completions_to_print rows. The console and wandb tables read a sample gathered to global rank 0, ceil(num_completions_to_print / world size) rows from each rank with the text cut to 8,000 characters a cell; leaving num_completions_to_print unset sends every row, so set it on a large run. The parquet parts keep every row and the full text.

Cut tool calls

A turn the engine cut mid-tool-call shows the partial call after its text, tagged cut by the engine before the turn closed; never run. On vLLM, with the turn's sampled ids captured and a reasoning budget resolving the reasoning close to a single token, it carries the whole decoded text past the close; elsewhere (SGLang, which enforces no reasoning cap, included), what the tool parser salvaged. It never runs, and later turns never see it.

Writer rank

Every rank writes its own part into its node's output_dir, so no rank gathers another's trajectories and a node's directory holds its own ranks' parts on per-node storage; only the tables' sample reaches global rank 0. A failed write (a full disk, a wandb outage) fails the step on every rank.

Halts

Three rules end a run whose rounds train on nothing or on a biased sample, each raising on every rank:

RuleHalts when
Empty stepstwo steps in a row hold no valid episode anywhere in the world
zero_gradient_round_limitthat many train rounds in a row carry no policy gradient, in a launch that has not trained yet
episode_error_rate_limitthe world-wide episode/error_rate of a train round stays over it (0.5) for episode_error_round_limit rounds (3) in a row

An episode that errored or outran request_timeout trains as a masked row and stays out of the baseline, so a run whose slow episodes time out learns from the fast ones alone, toward short answers; half the episodes failing three rounds running is far past a grader blip. The error rate is logged as episode/error_rate every round, and the startup preflight times a short generation per server and warns when a full rollout_max_tokens turn could not finish inside request_timeout (Startup preflight).

The counters of all three, and the early-stop streaks below, are written beside the pending prefetch rounds in each checkpoint and read back on a resume, so a restart continues the patience budget instead of granting a full one.

Early stop

Off unless a condition is set. A KL-free run drifts in either direction, toward entropy collapse or explosion, and both start slowly enough to stop on. Each condition counts its consecutive breaching readings; when one reaches early_stop_patience (default 3), training ends on a verdict taken across ranks.

  • A reading is one training log line (the window mean under logging_steps > 1). The gap and the breaker (table below) come once per generation round, so a step reusing its round leaves their count unchanged.
  • The stop skips that step's save and eval (an epoch save included), logs its reason, and exits non-zero on every rank. Resume from a checkpoint taken before the drift began.
KnobStops whenReads
early_stop_entropy_bandentropy stays outside [low, high]TRL's entropy
early_stop_logratio_gapthe trainer-vs-sampler gap stays above itthe magnitude of sampling/logratio_mean; needs the IS correction
early_stop_on_skipped_updatesthe breaker skipped every update, leaving the policy frozen where the rollouts disagree with itsampling/update_skipped; needs skip_update_masked_frac

TRL's entropy counts a micro-batch whose loss tokens were all dropped (degenerate groups, invalid episodes, padding rows) as 0, so under drops it reads low and moves with the dropped share.

Read the band and the gap off a healthy run; both are model-specific. The gap grows with entropy, and faster than it once the policy flattens, so it shows a drift before entropy does.

  • The Qwen3.6 vLLM code-contests recipes stop on a band of [0.12, 0.40] and a gap of 0.005.
  • Every code-contests recipe stops on skipped updates.

Source on GitHubUpdated 18 November 2026