Online GRPO (RLVR)

RLVR (Reinforcement Learning with Verifiable Rewards) is on-policy GRPO scored by the config's reward terms, \boxed{} exact match by default. Trainer DistributedOnlineGRPOTrainer, script scripts/training/grpo/online.py. A separate vLLM server generates and receives weights over NCCL; the trainer scores rewards.

Parallelism: FSDP2 data parallelism (HSDP across NVLink domains), EP, TP, EP+TP, EP+ETP and pure ETP (ep_size=1). CP and PP are refused before the model loads. Which families: Rollout Servers.

Online GRPO cycle: vLLM generates a group per prompt, the reward terms score it, advantages are group-normalized, the loss steps, and weights sync back over NCCL
Online GRPO cycle: vLLM generates a group per prompt, the reward terms score it, advantages are group-normalized, the loss steps, and weights sync back over NCCL

Dataset

Two columns: prompt (a string or a {"role", "content"} message list) and answer (the ground truth).

JSON Lines

{"prompt": "What is 2 + 3? Put your answer in \\boxed{}.", "answer": "5"}
{"prompt": [{"role": "user", "content": "What is 2 + 3?"}], "answer": "5"}
  • prompt_field and answer_field must name real columns; a typo raises at load.
  • A row with a null or empty prompt is dropped at load.
  • system_prompt is prepended unless the row already opens with a system turn.
  • A rendered prompt over max_prompt_length is dropped, never truncated.

The recipes pull openai/gsm8k, trl-lib/DeepMath-103K and open-r1/DAPO-Math-17k-Processed.

Rewards

Each rewards: term becomes one TRL reward function named after it, weighted through GRPOConfig.reward_weights and logged as rewards/<name>/mean. The default is one accuracy term. The list is YAML-only; the CLI cannot set it.

sourceScores 1.0 whenOptions
accuracythe last \boxed{...} equals answernone
formatthe completion matches patternpattern, default <think>.*?</think>\s*<answer>.*?</answer> under re.DOTALL

Accuracy is strict: the answer keeps only its text after a GSM8K ####, both sides drop , and $, braces match by depth, and an empty extraction never scores. The environments' grader also lowercases and accepts rtol=0.01 numeric matches.

A judge reads the answer column as its reference. A veto judge (one listing checks) is refused at config time; list requirements instead.

A judge or reward_model term reads the row's conversation and answer columns, so keep remove_unused_columns: false (TRL's GRPO default): at true TRL forwards neither column and the term raises at its first scoring call.

Configuration

Fields: OnlineGRPOScriptArguments, TRL's GRPOConfig, ModelArguments and ParallelismArguments.

YAML

model_name_or_path: Qwen/Qwen3-4B-Instruct-2507
dataset: trl-lib/DeepMath-103K
answer_field: solution

beta: 0.0                     # no KL anchor, no reference model
num_generations: 8            # the GRPO group size
max_prompt_length: 1024       # dataset filter
max_completion_length: 2048   # generation budget; required

use_vllm: true                # server mode is mandatory
vllm_mode: server

learning_rate: 1.0e-06
  • max_completion_length must be a positive int; null raises, and leaving it out takes TRL's default, so set it. With both length knobs bounded, their sum becomes model_max_length.
  • The recipes (examples/grpo/online/) run loss_type: grpo, beta: 0, epsilon 0.15–0.2, num_generations 4 or 8 and max_completion_length 128–4096; a field a config leaves out takes TRL's default (GRPOConfig).
  • reasoning_effort (default unset; low / medium / high / random) is applied at dataset-map time and needs a template that reads it.
  • Learning rate: 1e-6 to 5e-6 in the recipes, all full fine-tunes; LoRA takes more (sizing).

Optional objective changes

All off by default.

KnobEffectRequires
scale_rewards_std_floorA floor on the std divisorNot scale_rewards: none
drop_degenerate_groupsMasks all-equal-reward groups out of the loss and normalizer—
balance_token_massCancels each generation round's net token-weighted push (Token-mass balance)loss_type cispo, dapo or dr_grpo; top_entropy_quantile: 1.0; no off_policy_mask_threshold
use_rlrrIntra-group ranking advantages (arXiv:2601.23058)Neither the std floor nor the drop
use_sdpgOnline SDPG—

The std floor, the drop and RLRR also require multi_objective_aggregation: sum_then_normalize.

The rlrr_* tunables are range-validated always and refused at a non-default value with use_rlrr off. examples/grpo/online/qwen3/online-grpo-qwen3-8b-rlrr-math.yaml runs RLRR at these defaults:

YAML

rlrr_mode: hrr                   # or prr
rlrr_tau: 0.1
rlrr_lambda: 2048.0
rlrr_xi_pos: 1.0e-3              # clip band; xi_neg > xi_pos is refused
rlrr_xi_neg: -1.0e-3
rlrr_std_normalize: false
rlrr_length_rerank: true
rlrr_correctness_clip: true
rlrr_correctness_threshold: 0.5  # the only correctness signal; no gold labels

Importance sampling and the server probe

vllm_importance_sampling_correction (TRL default on) weights the loss by exp(logπ_recompute − logπ_sampling). The default sequence_mask mode masks a sequence whose ratio exceeds vllm_importance_sampling_clip_max (3.0) or falls below vllm_importance_sampling_clip_min (unset); the *_truncate modes clamp instead. With the correction off (and aligned accumulation), the 5-nat k3 KL tail clamp and kl_clamp_frac cannot engage, so beta > 0 warns.

A server restarted between syncs is not recovered here: the push raises on every rank, where async GRPO reconnects and re-sends (Rollout Servers).

Rank 0 waits for the server at startup, up to vllm_server_timeout, and raises past it. Its probe then refuses a server off the toolkit's image or, on a MoE, not on --moe-backend triton (Startup preflight), and one whose logprobs the correction cannot use:

  • raw pre-temperature logprobs at temperature != 1.0; serve vLLM with --logprobs-mode processed_logprobs;
  • logprobs renormalized over a sampling cut (top_p < 1, a top_k or a min_p) while the correction sums the per-token log-ratios over each sequence, which stalls the run silently. Keep all three off. A sequence_* mode sums them, and so does loss_type: vespo in either token mode TRL lets it run.

Under loss_type: vespo with token_mask, TRL zeroes each ratio outside the clip bounds, which floors that sum, so one such token silences its whole sequence; token_truncate clamps instead.

TRL sends top_k, min_p and repetition_penalty on every request, so the model's generation_config.json defaults never apply (Rollout Servers).

Forced and refused settings

  • Dropout is off, EP expert-LoRA dropout included, overriding disable_dropout: false; a config-level float only warns.
  • MoE balancing is off: aux_loss is inert under a policy-gradient loss, and both bias-update modes downgrade to none (Callbacks).
  • TRL's use_liger_kernel is off: it routes the log-probs through TRL's chunked LM head, past the family's head transform. Model-level Liger kernels still apply.
  • text_only_model is refused by the script: the sync forwards names verbatim, and the text-only sibling's model.layers.* would miss a multimodal server's model.language_model.layers.*.
  • Token budget refused at load: reject_unsupported_model sizes per_device_train_batch_size × (max_prompt_length + max_completion_length) against the proxy-GIN ceiling on a group spanning NVLink domains (cross-node EP) before any weight is read (Dispatch token ceiling).
  • Refused at construction (online and async GRPO): use_adaptive_entropy (a sharded resume does not restore its coefficient), a non-zero entropy_coef with use_chunked_grpo_logprobs (the chunked sweep returns a detached entropy), and an IterableDataset on the DP-sharded loader that TP, ETP and pre-sharded datasets use.

GRPO objective for verifiable rewards

For a shaped reward in [0, 1] (a grade plus small shaping terms) over a small group, set these TRL GRPOConfig fields (async GRPO too).

KnobSet it toWhy
loss_typedapo (TRL's default)Length-unbiased: normalizes over the global active-token count, unlike grpo's per-sequence mean
epsilon_high0.28 with epsilon: 0.2DAPO clip-higher: only the upper bound loosens
scale_rewardsbatch for a shaped rewardgroup divides by the group's own spread, inflating shaping noise to full-scale advantages
mask_truncated_completionstrue (default false)Drops a completion cut at the budget instead of scoring it. Async GRPO widens truncated to a turn overflow; its recipes leave it off
  • A single binary reward keeps scale_rewards: group: the 1e-4 added to the divisor keeps a uniformly-failed group's advantages at zero, not NaN.
  • The clip binds only when a generation batch outlives one optimizer step. At the defaults (num_iterations: 1, steps_per_generation = gradient_accumulation_steps) epsilon is inert. epsilon_high falls back to epsilon.

beta is the KL anchor to the base model; every online recipe runs beta: 0. TRL weights the KL term by the importance-sampling ratio (use_bias_correction_kl, on by default), which changes its gradient even on-policy; use_bias_correction_kl: false keeps the plain k3 term. Without a PEFT wrapper (a full fine-tune, or expert-only LoRA, which builds no PeftModel), beta != 0 holds a frozen reference:

  • The script loads it as the policy loads (load_reference_model_for_on_policy_grpo, halo/model_loading/parallel/frozen_models.py): run dtype, model_revision, the policy's attention request and reset_sinks, the non-persistent buffers repaired, the vocabulary grown with the tokenizer. The trainer hands it to TRL in place of the fp32 copy TRL would build from the hub's default revision.
  • It is an unsharded dense replica on every rank. Before it loads, the script sizes it from a config-only meta shell at the run dtype and refuses, on every rank, a reference over half the GPU memory free beside the loaded policy or whose copies across the node's ranks (each stages its own in host memory) exceed 80% of the available host RAM; the error names beta: 0, an attention LoRA target and offline GRPO's precomputed reference. This covers plain FSDP2 as much as EP, ETP and TP. The check runs twice: before the policy loads, against the free memory then (a run it refuses would fail the later check too), and again after it, against the memory the loaded policy leaves. The half is a heuristic: optimizer state, gradients and activations are not budgeted, so a reference that passes can still run out of memory at the first step. A reference that fits is warned about under EP, ETP and TP when the trainer is built. A resume loads it from model_name_or_path, never the checkpoint.
  • A trainer constructed with beta != 0 and no ref_model raises, as does a ref_model the run never reads (beta: 0, or a PEFT policy, whose reference is the policy with its adapter disabled), both before the rollout client connects.
  • A reference whose sinks differ from the policy's is rejected in every mode, since the KL would be biased on every token. The script's reference takes the policy's reset_sinks, so live sinks (reset_sinks: false) score on both sides.

Chat template handling

The trainer's tokenizer renders each prompt and TRL sends vLLM the token ids, so a chat_template: override needs no server-side setting (unlike async GRPO). A tools_field column rides through as tools=. The leading BOS is stripped only when the tokenizer's post-processor adds its own, so a template-emitted BOS (Gemma 4) survives.

Chunked log-probs

use_chunked_grpo_logprobs: true (default off) replaces the full [B, T, vocab] logits with a vocab-chunked softmax, for large vocabularies with long completions (Offline GRPO, limits). Here a multimodal batch falls back to the full path, and a PEFT adapter on lm_head raises.

Packed micro-batches

packed_microbatch_tokens (default off) groups each rank's rows into token-budget bins that run as one flattened varlen call each, in TRL's old-policy and reference passes and in the loss forward, instead of one call per row at per_device_train_batch_size: 1. It needs use_chunked_grpo_logprobs: true and is refused with SDPG; mechanism, objective and the other refusals are on Packed micro-batches.

Importance ratio from the loss forward

loss_forward_importance_weights (default off) drops TRL's no-grad old-policy pass, one forward over the round's rows, and builds the vLLM importance-sampling ratio and the KL tail clamp from the loss forward's detached log-probs. Refusals, mechanism and metrics: Weight from the loss forward.

Rank balance

balance_rank_tokens (default off) deals each round's rows across ranks so every rank trains a similar token total; the objective is unchanged. Refusals and mechanism: Rank balance.

Server mode only

DistributedOnlineGRPOTrainer rejects two generation paths:

  • In-process generation (use_vllm: false, vllm_mode: colocate), at init: two NCCL worlds on the same GPUs deadlock (Rollout Servers).
  • TRL's tool-calling loop (tools, environment_factory), which desynchronizes ranks. Tool use trains on Async GRPO with Environments.

TRL generates through TrlVLLMClient (halo/trainers/grpo/rollout/trl_vllm_client.py): one batched /v1/completions request per round, bounded by HALO_VLLM_GENERATION_TIMEOUT_SECONDS. The completion ids are the server's sampled ids (return_token_ids); a server that omits them is refused rather than re-tokenized. TRL's vllm_structured_outputs_regex goes out as vLLM's structured_outputs.regex. The route is text-only, so a prompt with an image raises.

Before the trainer is built, every rank raises if max_prompt_length + max_completion_length exceeds the served max_model_len.

Weight sync

Weights are pushed before the next generation, not at the optimizer step; a resumed run pushes before its first rollout (mechanics). The script refuses a family no engine sync reaches right after init (reject_unsyncable_config), and reject_unsyncable_model refuses at construction what the sync cannot carry (construction gates), including a PEFT layer the sync cannot fold or an adapter on a tensor the dense push does not send (PEFT). The first push is checked against the trainer's own log-probs (first-push parity probe). rollout_weight_precision: fp8_block pushes block-FP8 weights to a server booted from the export tool's checkpoint (FP8 rollouts).

Multi-homed nodes (VLLM_GROUP_HOST)

Set VLLM_GROUP_HOST to the trainer IP a remote server should dial when it cannot dial back to the trainer's address on the route to it (NAT, asymmetric routing); this trainer has no per-server group_host (TRL builds its client without one). Resolution order and the rendezvous: Group rendezvous.

Launch

vLLM must own GPUs no trainer rank uses. The compose defaults put the server on GPU 7 (VLLM_CUDA_DEVICES) and the trainer on 0–6 (TRAINER_CUDA_DEVICES).

Bash

# vLLM on GPU 7 (flags: Rollout Servers)
make build-vllm   # or set VLLM_IMAGE to the public image
export HALO_SCRATCH=/path/to/large/volume   # or set it in .env; compose refuses to start without it
VLLM_MODEL=Qwen/Qwen3-30B-A3B docker compose -f docker-compose.vllm.yml up vllm-server

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6 torchrun --nproc_per_node=7 \
    scripts/training/grpo/online.py examples/grpo/online/rlvr-online-grpo-template.yaml \
    --model_name_or_path=Qwen/Qwen3-30B-A3B

halo launch grpo/online <config> --nproc 7 builds the same line (FSDP2 data parallelism); the device split stays yours. Add --tensor_parallel_size for dense TP or --expert_parallel_size for MoE.

Under EP, expert_parallel_size equal to the training-GPU count forms one DeepEP group, as the recipes do. expert_parallel_size: 2 also runs as several 2-rank groups, and node-local groups repeat across NVLink domains:

  • Within one NVLink domain, a group above expert_parallel_size: 2 narrower than the domain is rejected (DeepEP).
  • num_experts must divide by ep_size, so most rosters want a power-of-two split: trainer on GPUs 0–3 with --expert_parallel_size=4, server on VLLM_CUDA_DEVICES=4,5,6,7.
  • Under EP+TP, tp_size must divide the NVLink domain and ep_size must be a multiple of tp_size.

LoRA

Add use_peft: true and the lora_* fields. LoRA runs under FSDP2 DP, EP, EP+ETP and pure ETP (under ETP attention adapters only: native expert LoRA is refused at expert_tensor_parallel_size > 1); under TP and EP+TP every adapter is rejected, native expert adapters included.

The weight sync folds each adapter, native expert-LoRA included, into its base weight as it sends it; vLLM loads no adapter and the trainer's frozen base is never written (PEFT).

Data flow and batch construction

Each row becomes a group of num_generations completions. Under TP and ETP every rank takes its group leader's completions; TRL's rollout_func is refused there. The collator is TRL's row collator (left-padded prompts, right-padded completions), hence no CP. Rows stay padded; packed_microbatch_tokens packs them into flattened bins as a separate step (Packed micro-batches).

Counting. Rollouts per optimizer step = per_device_train_batch_size × gradient_accumulation_steps × data_parallel_size. per_device_train_batch_size counts completions, not prompts; with the completion length, it sets per-forward memory.

Geometry. Under TP, ETP or a pre-sharded dataset the dataloader enforces both rules when it is built (elsewhere TRL's config check applies the first):

  • per_device_train_batch_size × steps_per_generation × data_parallel_size must divide by num_generations. A pre-sharded dataset drops the DP factor.
  • Under TP/ETP the per-rank product must itself divide by num_generations.

vLLM on Blackwell (B200)

  • Server: VLLM_ATTENTION_BACKEND=FLASH_ATTN only where FlashInfer JIT-fails on SM 10.0 (Rollout Servers).
  • Trainer: all three GRPO scripts give an unset attn_implementation the padded-batch default — the loader's own pick on Blackwell (SDPA for a remote-code checkpoint, FA2 for a softcapped policy), SDPA elsewhere — and request nothing under reset_sinks: false and offline GRPO under CP (padded workloads). A pinned value wins.

Testing a setup

Run a 3-step smoke config against a live server first: examples/grpo/online/qwen3/online-grpo-qwen3-4b-smoke.yaml (dense) or .../gptoss/online-grpo-gptoss-20b-ep-smoke.yaml (EP).

CPU: pytest tests/cpu/grpo -m cpu. GPU: tests/gpu/trainers/grpo/test_online_grpo_vllm_e2e.py (the PEFT × parallelism × resume matrix against a live server, one --mode per row).

What to watch

Metric names follow TRL's GRPOTrainer. Read every run: rewards/accuracy/mean, frac_reward_zero_std, sampling/importance_sampling_ratio/mean (near 1 when trainer and engine agree), completions/clipped_ratio, entropy, and each external term's (judge, reward_model) <source>/<name>/scored. Toolkit additions:

  • kl_clamp_frac: share of loss tokens whose reference log-ratio hits the 5-nat clamp, at beta > 0.
  • sampling/degenerate_group_frac: share of completions drop_degenerate_groups masked out.
  • phase_time/<phase>_s and phase_time/<phase>_max_s under log_phase_times: true: where each step's wall time goes, rank mean and slowest rank (Phase times).
  • rank_balance/tokens_max_over_mean_before and _after under balance_rank_tokens: the busiest rank's attended tokens over the mean, before and after the deal (Rank balance).
  • Under enable_efficiency_metrics: step_tokens_per_second / avg_tokens_per_second (attended tokens, prompt plus completion, which TRL counts while it builds the round; SDPG's teacher forward is not counted) and step_data_wait_seconds / avg_data_wait_seconds (EfficiencyCallback).
  • A first-push parity warning in the log when the served model's log-probs on a fixed row stray past the probe's bound from the trainer's; HALO_WEIGHT_SYNC_PARITY_PROBE=raise stops the run, off skips it (probe).
  • advantage/net_token_mass every round (an early sign of entropy drift) and, under balance_token_mass, advantage/token_mass_scale; the sign reads as the entropy push only under a token-sum loss such as dapo.

early_stop_entropy_band and early_stop_logratio_gap work as on Async GRPO; the gap needs vllm_importance_sampling_correction and reads TRL's sampling/sampling_logp_difference/mean (a mean absolute difference), so a threshold read off an Async GRPO run does not carry over. save_completions (default on) writes <output_dir>/completions/completions_<step>.parquet (step zero-padded to five digits, _eval suffix for evaluation).

Eval padding. An eval split's final round is padded with repeats of its first prompts, which TRL counts in generation, loss and metrics, so a split that does not fill whole rounds over-weights them. Async GRPO with Environments never rolls the padding out: its rollout metrics and completions count the split's own episodes alone (Padding and round means).

SymptomCauseFix
All rewards zeroThe policy emits no \boxed{}Ask for it in system_prompt, raise max_completion_length, read the parquet
IS ratio far from 1Trainer and engine disagree (template, sinks, dropout); a small gap on identical weights is bf16 routing noiseMatch the served model; reset_sinks: false for GPT-OSS
Hang at sync or generationServer down or wedged, a GPU shared with a trainer rank, blocked portscurl .../health; check both containers' CUDA_VISIBLE_DEVICES; raise DIST_NCCL_TIMEOUT_MINUTES (default 30); Rollout Servers
OOMFull-vocab logits or a long completionuse_chunked_grpo_logprobs: true, lower the batch or max_completion_length, raise EP size

Offline GRPO · Async GRPO with Environments · Rollout Servers · Trainer Architecture

Source on GitHubUpdated 18 November 2026