# Online GRPO (RLVR)

RLVR (Reinforcement Learning with Verifiable Rewards) is on-policy GRPO scored by the config's [reward terms](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/rewards.md), `\boxed{}` exact match by default. Trainer `DistributedOnlineGRPOTrainer`, script `scripts/training/grpo/online.py`. A separate vLLM server generates and receives weights over NCCL; the trainer scores rewards.

Parallelism: FSDP2 data parallelism (HSDP across NVLink domains), EP, TP, EP+TP, EP+ETP and pure ETP (`ep_size=1`). CP and PP are refused before the model loads. Which families: [Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#which-families-each-engine-serves).

![Online GRPO cycle: vLLM generates a group per prompt, the reward terms score it, advantages are group-normalized, the loss steps, and weights sync back over NCCL](https://whitecircle.com/halo/docs/assets/agent-docs/assets/diagrams/online_grpo_pipeline.png?ref=872a3e4284ef3a533c18b651f91b76adcbdcf306)

## Dataset

Two columns: `prompt` (a string or a `{"role", "content"}` message list) and `answer` (the ground truth).

```jsonl
{"prompt": "What is 2 + 3? Put your answer in \\boxed{}.", "answer": "5"}
{"prompt": [{"role": "user", "content": "What is 2 + 3?"}], "answer": "5"}
```

- `prompt_field` and `answer_field` must name real columns; a typo raises at load.
- A row with a null or empty prompt is dropped at load.
- `system_prompt` is prepended unless the row already opens with a system turn.
- A rendered prompt over `max_prompt_length` is **dropped**, never truncated.

The recipes pull `openai/gsm8k`, `trl-lib/DeepMath-103K` and `open-r1/DAPO-Math-17k-Processed`.

## Rewards

Each `rewards:` [term](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/rewards.md) becomes one TRL reward function named after it, weighted through `GRPOConfig.reward_weights` and logged as `rewards/<name>/mean`. The default is one `accuracy` term. The list is YAML-only; the CLI cannot set it.

| `source` | Scores `1.0` when | Options |
|---|---|---|
| `accuracy` | the last `\boxed{...}` equals `answer` | none |
| `format` | the completion matches `pattern` | `pattern`, default `<think>.*?</think>\s*<answer>.*?</answer>` under `re.DOTALL` |

Accuracy is strict: the answer keeps only its text after a GSM8K `####`, both sides drop `,` and `$`, braces match by depth, and an empty extraction never scores. The environments' grader also lowercases and accepts `rtol=0.01` numeric matches.

A judge reads the `answer` column as its reference. A veto judge (one listing `checks`) is refused at config time; list `requirements` instead.

A `judge` or `reward_model` term reads the row's `conversation` and `answer` columns, so keep `remove_unused_columns: false` (TRL's GRPO default): at `true` TRL forwards neither column and the term raises at its first scoring call.

## Configuration

Fields: [`OnlineGRPOScriptArguments`](https://whitecircle.com/halo/docs/agent-docs/reference/configuration-reference.md#onlinegrposcriptarguments), TRL's `GRPOConfig`, `ModelArguments` and `ParallelismArguments`.

```yaml
model_name_or_path: Qwen/Qwen3-4B-Instruct-2507
dataset: trl-lib/DeepMath-103K
answer_field: solution

beta: 0.0                     # no KL anchor, no reference model
num_generations: 8            # the GRPO group size
max_prompt_length: 1024       # dataset filter
max_completion_length: 2048   # generation budget; required

use_vllm: true                # server mode is mandatory
vllm_mode: server

learning_rate: 1.0e-06
```

- `max_completion_length` must be a positive int; `null` raises, and leaving it out takes TRL's default, so set it. With both length knobs bounded, their sum becomes `model_max_length`.
- The recipes (`examples/grpo/online/`) run `loss_type: grpo`, `beta: 0`, `epsilon` `0.15`–`0.2`, `num_generations` 4 or 8 and `max_completion_length` 128–4096; a field a config leaves out takes TRL's default ([GRPOConfig](https://whitecircle.com/halo/docs/agent-docs/reference/configuration-reference.md#grpoconfig--distributedonlinegrpotrainer-distributedasyncenvironmentalgrpotrainer-trl-docs)).
- `reasoning_effort` (default unset; `low` / `medium` / `high` / `random`) is applied at dataset-map time and needs a template that reads it.
- Learning rate: `1e-6` to `5e-6` in the recipes, all full fine-tunes; LoRA takes more ([sizing](https://whitecircle.com/halo/docs/agent-docs/training-methods/sft.md#learning-rate-and-global-batch-size)).

### Optional objective changes

All off by default.

| Knob | Effect | Requires |
|---|---|---|
| `scale_rewards_std_floor` | A floor on the std divisor | Not `scale_rewards: none` |
| `drop_degenerate_groups` | Masks all-equal-reward groups out of the loss and normalizer | — |
| `balance_token_mass` | Cancels each generation round's net token-weighted push ([Token-mass balance](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/objective.md#token-mass-balance)) | `loss_type` `cispo`, `dapo` or `dr_grpo`; `top_entropy_quantile: 1.0`; no `off_policy_mask_threshold` |
| `use_rlrr` | Intra-group ranking advantages ([arXiv:2601.23058](https://arxiv.org/abs/2601.23058)) | Neither the std floor nor the drop |
| `use_sdpg` | [Online SDPG](https://whitecircle.com/halo/docs/agent-docs/training-methods/distillation/online-sdpg.md) | — |

The std floor, the drop and RLRR also require `multi_objective_aggregation: sum_then_normalize`.

The `rlrr_*` tunables are range-validated always and refused at a non-default value with `use_rlrr` off. `examples/grpo/online/qwen3/online-grpo-qwen3-8b-rlrr-math.yaml` runs RLRR at these defaults:

```yaml
rlrr_mode: hrr                   # or prr
rlrr_tau: 0.1
rlrr_lambda: 2048.0
rlrr_xi_pos: 1.0e-3              # clip band; xi_neg > xi_pos is refused
rlrr_xi_neg: -1.0e-3
rlrr_std_normalize: false
rlrr_length_rerank: true
rlrr_correctness_clip: true
rlrr_correctness_threshold: 0.5  # the only correctness signal; no gold labels
```

### Importance sampling and the server probe

`vllm_importance_sampling_correction` (TRL default on) weights the loss by `exp(logπ_recompute − logπ_sampling)`. The default `sequence_mask` mode masks a sequence whose ratio exceeds `vllm_importance_sampling_clip_max` (`3.0`) or falls below `vllm_importance_sampling_clip_min` (unset); the `*_truncate` modes clamp instead. With the correction off (and aligned accumulation), the 5-nat k3 KL tail clamp and `kl_clamp_frac` cannot engage, so `beta > 0` warns.

A server restarted between syncs is not recovered here: the push raises on every rank, where async GRPO reconnects and re-sends ([Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#streaming-and-quiesce)).

Rank 0 waits for the server at startup, up to `vllm_server_timeout`, and raises past it. Its probe then refuses a
server off the toolkit's image or, on a MoE, not on `--moe-backend triton` ([Startup preflight](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#startup-preflight)), and one whose logprobs the correction cannot use:

- raw pre-temperature logprobs at `temperature != 1.0`; serve vLLM with `--logprobs-mode processed_logprobs`;
- logprobs renormalized over a sampling cut (`top_p < 1`, a `top_k` or a `min_p`) while the correction sums the per-token log-ratios over each sequence, which stalls the run silently. Keep all three off. A `sequence_*` mode sums them, and so does `loss_type: vespo` in either token mode TRL lets it run.

Under `loss_type: vespo` with `token_mask`, TRL zeroes each ratio outside the clip bounds, which floors that sum, so one such token silences its whole sequence; `token_truncate` clamps instead.

TRL sends `top_k`, `min_p` and `repetition_penalty` on every request, so the model's `generation_config.json` defaults never apply ([Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md)).

### Forced and refused settings

- **Dropout is off**, EP expert-LoRA dropout included, overriding `disable_dropout: false`; a config-level float only warns.
- **MoE balancing is off:** `aux_loss` is inert under a policy-gradient loss, and both bias-update modes downgrade to `none` ([Callbacks](https://whitecircle.com/halo/docs/agent-docs/training-methods/callbacks.md#balancingbiascallback)).
- **TRL's `use_liger_kernel` is off:** it routes the log-probs through TRL's chunked LM head, past the family's head transform. Model-level Liger kernels still apply.
- **`text_only_model` is refused** by the script: the sync forwards names verbatim, and the text-only sibling's `model.layers.*` would miss a multimodal server's `model.language_model.layers.*`.
- **Token budget refused at load:** `reject_unsupported_model` sizes `per_device_train_batch_size × (max_prompt_length + max_completion_length)` against the proxy-GIN ceiling on a group spanning NVLink domains (cross-node EP) before any weight is read ([Dispatch token ceiling](https://whitecircle.com/halo/docs/agent-docs/infrastructure/deepep.md#dispatch-token-ceiling)).
- **Refused at construction** (online and async GRPO): `use_adaptive_entropy` (a sharded resume does not restore its coefficient), a non-zero `entropy_coef` with `use_chunked_grpo_logprobs` (the chunked sweep returns a detached entropy), and an `IterableDataset` on the DP-sharded loader that TP, ETP and pre-sharded datasets use.

## GRPO objective for verifiable rewards

For a shaped reward in `[0, 1]` (a grade plus small shaping terms) over a small group, set these TRL `GRPOConfig` fields (async GRPO too).

| Knob | Set it to | Why |
|---|---|---|
| `loss_type` | `dapo` (TRL's default) | Length-unbiased: normalizes over the global active-token count, unlike `grpo`'s per-sequence mean |
| `epsilon_high` | `0.28` with `epsilon: 0.2` | DAPO clip-higher: only the upper bound loosens |
| `scale_rewards` | `batch` for a shaped reward | `group` divides by the group's own spread, inflating shaping noise to full-scale advantages |
| `mask_truncated_completions` | `true` (default `false`) | Drops a completion cut at the budget instead of scoring it. Async GRPO widens `truncated` to a turn overflow; its recipes leave it off |

- A single binary reward keeps `scale_rewards: group`: the `1e-4` added to the divisor keeps a uniformly-failed group's advantages at zero, not NaN.
- The clip binds only when a generation batch outlives one optimizer step. At the defaults (`num_iterations: 1`, `steps_per_generation` = `gradient_accumulation_steps`) `epsilon` is inert. `epsilon_high` falls back to `epsilon`.

`beta` is the KL anchor to the base model; every online recipe runs `beta: 0`. TRL weights the KL term by the importance-sampling ratio (`use_bias_correction_kl`, on by default), which changes its gradient even on-policy; `use_bias_correction_kl: false` keeps the plain k3 term. Without a PEFT wrapper (a full fine-tune, or expert-only LoRA, which builds no `PeftModel`), `beta != 0` holds a frozen reference:

- The script loads it as the policy loads (`load_reference_model_for_on_policy_grpo`, `halo/model_loading/parallel/frozen_models.py`): run dtype, `model_revision`, the policy's attention request and `reset_sinks`, the non-persistent buffers repaired, the vocabulary grown with the tokenizer. The trainer hands it to TRL in place of the fp32 copy TRL would build from the hub's default revision.
- It is an unsharded dense replica on every rank. Before it loads, the script sizes it from a config-only meta shell at the run dtype and refuses, on every rank, a reference over half the GPU memory free beside the loaded policy or whose copies across the node's ranks (each stages its own in host memory) exceed 80% of the available host RAM; the error names `beta: 0`, an attention LoRA target and offline GRPO's precomputed reference. This covers plain FSDP2 as much as EP, ETP and TP. The check runs twice: before the policy loads, against the free memory then (a run it refuses would fail the later check too), and again after it, against the memory the loaded policy leaves. The half is a heuristic: optimizer state, gradients and activations are not budgeted, so a reference that passes can still run out of memory at the first step. A reference that fits is warned about under EP, ETP and TP when the trainer is built. A resume loads it from `model_name_or_path`, never the checkpoint.
- A trainer constructed with `beta != 0` and no `ref_model` raises, as does a `ref_model` the run never reads (`beta: 0`, or a PEFT policy, whose reference is the policy with its adapter disabled), both before the rollout client connects.
- A reference whose sinks differ from the policy's is rejected in every mode, since the KL would be biased on every token. The script's reference takes the policy's `reset_sinks`, so live sinks (`reset_sinks: false`) score on both sides.

## Chat template handling

The trainer's tokenizer renders each prompt and TRL sends vLLM the **token ids**, so a `chat_template:` override needs no server-side setting ([unlike async GRPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/episodes.md#chat-template)). A `tools_field` column rides through as `tools=`. The leading BOS is stripped **only** when the tokenizer's post-processor adds its own, so a template-emitted BOS (Gemma 4) survives.

## Chunked log-probs

`use_chunked_grpo_logprobs: true` (default off) replaces the full `[B, T, vocab]` logits with a vocab-chunked softmax, for large vocabularies with long completions ([Offline GRPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/offline.md#memory-chunked-log-probs), [limits](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/performance.md#chunked-log-probs)). Here a multimodal batch falls back to the full path, and a PEFT adapter on `lm_head` raises.

## Packed micro-batches

`packed_microbatch_tokens` (default off) groups each rank's rows into token-budget bins that run as one flattened varlen call each, in TRL's old-policy and reference passes and in the loss forward, instead of one call per row at `per_device_train_batch_size: 1`. It needs `use_chunked_grpo_logprobs: true` and is refused with SDPG; mechanism, objective and the other refusals are on [Packed micro-batches](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/performance.md#packed-micro-batches).

## Importance ratio from the loss forward

`loss_forward_importance_weights` (default off) drops TRL's no-grad old-policy pass, one forward over the round's rows, and builds the vLLM importance-sampling ratio and the KL tail clamp from the loss forward's detached log-probs. Refusals, mechanism and metrics: [Weight from the loss forward](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/performance.md#weight-from-the-loss-forward).

## Rank balance

`balance_rank_tokens` (default off) deals each round's rows across ranks so every rank trains a similar token total; the objective is unchanged. Refusals and mechanism: [Rank balance](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/performance.md#rank-balance).

## Server mode only

`DistributedOnlineGRPOTrainer` rejects two generation paths:

- **In-process generation** (`use_vllm: false`, `vllm_mode: colocate`), at init: two NCCL worlds on the same GPUs deadlock ([Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#vllm)).
- **TRL's tool-calling loop** (`tools`, `environment_factory`), which desynchronizes ranks. Tool use trains on [Async GRPO with Environments](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo.md).

TRL generates through `TrlVLLMClient` (`halo/trainers/grpo/rollout/trl_vllm_client.py`): one batched `/v1/completions` request per round, bounded by `HALO_VLLM_GENERATION_TIMEOUT_SECONDS`. The completion ids are the server's sampled ids (`return_token_ids`); a server that omits them is refused rather than re-tokenized. TRL's `vllm_structured_outputs_regex` goes out as vLLM's `structured_outputs.regex`. The route is text-only, so a prompt with an image raises.

Before the trainer is built, every rank raises if `max_prompt_length + max_completion_length` exceeds the served `max_model_len`.

> [!WARNING]
> **GPT-OSS: toolkit vLLM image, sinks on**
>
> Serve GPT-OSS from the toolkit `halo:vllm-0.32.0` image with sinks on (the default) and keep `reset_sinks: false` on the trainer; stock vLLM emits garbage GPT-OSS output on Blackwell ([GPT-OSS](https://whitecircle.com/halo/docs/agent-docs/models/gpt-oss.md#serving-for-grpo-vllm)).

### Weight sync

Weights are pushed before the **next** generation, not at the optimizer step; a resumed run pushes before its first rollout ([mechanics](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#weight-sync)). The script refuses a family no engine sync reaches right after init (`reject_unsyncable_config`), and `reject_unsyncable_model` refuses at construction what the sync cannot carry ([construction gates](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#construction-gates)), including a PEFT layer the sync cannot fold or an adapter on a tensor the dense push does not send ([PEFT](https://whitecircle.com/halo/docs/agent-docs/optimization/peft.md#online-rl--rollout-server-weight-sync)). The first push is checked against the trainer's own log-probs ([first-push parity probe](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#first-push-parity-probe)). `rollout_weight_precision: fp8_block` pushes block-FP8 weights to a server booted from the export tool's checkpoint ([FP8 rollouts](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#fp8-rollouts)).

### Multi-homed nodes (`VLLM_GROUP_HOST`)

Set `VLLM_GROUP_HOST` to the trainer IP a **remote** server should dial when it cannot dial back to the trainer's address on the route to it (NAT, asymmetric routing); this trainer has no per-server `group_host` (TRL builds its client without one). Resolution order and the rendezvous: [Group rendezvous](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#group-rendezvous).

> [!NOTE]
> **Multi-server sync belongs to Async GRPO with Environments**
>
> The list-of-dicts `server_configs` API (per-server `url`/`group_port`/`group_host` via `WeightSyncPool`) belongs to [Async GRPO with Environments](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/setup.md#multiple-servers-and-prefetch) (`AsyncRolloutConfig.rollout_server_configs`), not this single-server RLVR trainer.

## Launch

vLLM must own GPUs **no trainer rank uses**. The compose defaults put the server on GPU 7 (`VLLM_CUDA_DEVICES`) and the trainer on 0–6 (`TRAINER_CUDA_DEVICES`).

```bash
# vLLM on GPU 7 (flags: Rollout Servers)
make build-vllm   # or set VLLM_IMAGE to the public image
export HALO_SCRATCH=/path/to/large/volume   # or set it in .env; compose refuses to start without it
VLLM_MODEL=Qwen/Qwen3-30B-A3B docker compose -f docker-compose.vllm.yml up vllm-server

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6 torchrun --nproc_per_node=7 \
    scripts/training/grpo/online.py examples/grpo/online/rlvr-online-grpo-template.yaml \
    --model_name_or_path=Qwen/Qwen3-30B-A3B
```

`halo launch grpo/online <config> --nproc 7` builds the same line (FSDP2 data parallelism); the device split stays yours. Add `--tensor_parallel_size` for dense TP or `--expert_parallel_size` for MoE.

Under EP, `expert_parallel_size` equal to the training-GPU count forms one DeepEP group, as the recipes do. `expert_parallel_size: 2` also runs as several 2-rank groups, and node-local groups repeat across NVLink domains:

- Within one NVLink domain, a group above `expert_parallel_size: 2` narrower than the domain is rejected ([DeepEP](https://whitecircle.com/halo/docs/agent-docs/infrastructure/deepep.md#ep-grouping-what-is-reliable)).
- `num_experts` must divide by `ep_size`, so most rosters want a power-of-two split: trainer on GPUs 0–3 with `--expert_parallel_size=4`, server on `VLLM_CUDA_DEVICES=4,5,6,7`.
- Under EP+TP, `tp_size` must divide the NVLink domain and `ep_size` must be a multiple of `tp_size`.

### LoRA

Add `use_peft: true` and the `lora_*` fields. LoRA runs under FSDP2 DP, EP, EP+ETP and pure ETP (under ETP attention adapters only: native expert LoRA is refused at `expert_tensor_parallel_size > 1`); under TP and EP+TP every adapter is **rejected**, native expert adapters included.

The weight sync folds each adapter, native expert-LoRA included, into its base weight as it sends it; vLLM loads no adapter and the trainer's frozen base is never written ([PEFT](https://whitecircle.com/halo/docs/agent-docs/optimization/peft.md#online-rl--rollout-server-weight-sync)).

## Data flow and batch construction

Each row becomes a group of `num_generations` completions. Under TP and ETP every rank takes its group leader's completions; TRL's `rollout_func` is refused there. The collator is TRL's row collator (left-padded prompts, right-padded completions), hence no CP. Rows stay padded; `packed_microbatch_tokens` packs them into flattened bins as a separate step ([Packed micro-batches](#packed-micro-batches)).

**Counting.** Rollouts per optimizer step = `per_device_train_batch_size × gradient_accumulation_steps × data_parallel_size`. `per_device_train_batch_size` counts completions, not prompts; with the completion length, it sets per-forward memory.

**Geometry.** Under TP, ETP or a pre-sharded dataset the dataloader enforces both rules when it is built (elsewhere TRL's config check applies the first):

- `per_device_train_batch_size × steps_per_generation × data_parallel_size` must divide by `num_generations`. A pre-sharded dataset drops the DP factor.
- Under TP/ETP the per-rank product must **itself** divide by `num_generations`.

## vLLM on Blackwell (B200)

- **Server:** `VLLM_ATTENTION_BACKEND=FLASH_ATTN` only where FlashInfer JIT-fails on SM 10.0 ([Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#throughput)).
- **Trainer:** all three GRPO scripts give an unset `attn_implementation` the padded-batch default — the loader's own pick on Blackwell (SDPA for a remote-code checkpoint, FA2 for a softcapped policy), **SDPA** elsewhere — and request nothing under `reset_sinks: false` and offline GRPO under CP ([padded workloads](https://whitecircle.com/halo/docs/agent-docs/optimization/flash-attention.md#model-specific-handling)). A pinned value wins.

## Testing a setup

Run a 3-step smoke config against a live server first: `examples/grpo/online/qwen3/online-grpo-qwen3-4b-smoke.yaml` (dense) or `.../gptoss/online-grpo-gptoss-20b-ep-smoke.yaml` (EP).

CPU: `pytest tests/cpu/grpo -m cpu`. GPU: `tests/gpu/trainers/grpo/test_online_grpo_vllm_e2e.py` (the PEFT × parallelism × resume matrix against a live server, one `--mode` per row).

## What to watch

Metric names follow TRL's `GRPOTrainer`. Read every run: `rewards/accuracy/mean`, `frac_reward_zero_std`, `sampling/importance_sampling_ratio/mean` (near 1 when trainer and engine agree), `completions/clipped_ratio`, `entropy`, and each external term's (`judge`, `reward_model`) `<source>/<name>/scored`. Toolkit additions:

- `kl_clamp_frac`: share of loss tokens whose reference log-ratio hits the 5-nat clamp, at `beta > 0`.
- `sampling/degenerate_group_frac`: share of completions `drop_degenerate_groups` masked out.
- `phase_time/<phase>_s` and `phase_time/<phase>_max_s` under `log_phase_times: true`: where each step's wall time goes, rank mean and slowest rank ([Phase times](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/monitoring.md#phase-times)).
- `rank_balance/tokens_max_over_mean_before` and `_after` under `balance_rank_tokens`: the busiest rank's attended tokens over the mean, before and after the deal ([Rank balance](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/performance.md#rank-balance)).
- Under `enable_efficiency_metrics`: `step_tokens_per_second` / `avg_tokens_per_second` (attended tokens, prompt plus completion, which TRL counts while it builds the round; SDPG's teacher forward is not counted) and `step_data_wait_seconds` / `avg_data_wait_seconds` ([EfficiencyCallback](https://whitecircle.com/halo/docs/agent-docs/training-methods/callbacks.md#efficiencycallback)).
- A first-push parity warning in the log when the served model's log-probs on a fixed row stray past the probe's bound from the trainer's; `HALO_WEIGHT_SYNC_PARITY_PROBE=raise` stops the run, `off` skips it ([probe](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#first-push-parity-probe)).
- `advantage/net_token_mass` every round (an early sign of entropy drift) and, under `balance_token_mass`, `advantage/token_mass_scale`; the sign reads as the entropy push only under a token-sum loss such as `dapo`.

`early_stop_entropy_band` and `early_stop_logratio_gap` work as on [Async GRPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/monitoring.md#early-stop); the gap needs `vllm_importance_sampling_correction` and reads TRL's `sampling/sampling_logp_difference/mean` (a mean absolute difference), so a threshold read off an Async GRPO run does not carry over. `save_completions` (default on) writes `<output_dir>/completions/completions_<step>.parquet` (step zero-padded to five digits, `_eval` suffix for evaluation).

**Eval padding.** An eval split's final round is padded with repeats of its first prompts, which TRL counts in generation, loss and metrics, so a split that does not fill whole rounds over-weights them. [Async GRPO with Environments](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo.md) never rolls the padding out: its rollout metrics and completions count the split's own episodes alone ([Padding and round means](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/evaluation.md#padding-and-round-means)).

| Symptom | Cause | Fix |
|---|---|---|
| All rewards zero | The policy emits no `\boxed{}` | Ask for it in `system_prompt`, raise `max_completion_length`, read the parquet |
| IS ratio far from 1 | Trainer and engine disagree (template, sinks, dropout); a small gap on identical weights is [bf16 routing noise](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/objective.md#trainer-and-engine-noise) | Match the served model; `reset_sinks: false` for GPT-OSS |
| Hang at sync or generation | Server down or wedged, a GPU shared with a trainer rank, blocked ports | `curl .../health`; check both containers' `CUDA_VISIBLE_DEVICES`; raise `DIST_NCCL_TIMEOUT_MINUTES` (default `30`); [Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#troubleshooting) |
| OOM | Full-vocab logits or a long completion | `use_chunked_grpo_logprobs: true`, lower the batch or `max_completion_length`, raise EP size |

## Related pages

[Offline GRPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/offline.md) · [Async GRPO with Environments](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo.md) · [Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md) · [Trainer Architecture](https://whitecircle.com/halo/docs/agent-docs/reference/trainer-architecture.md)
