Online GRPO (RLVR)
RLVR (Reinforcement Learning with Verifiable Rewards) is on-policy GRPO scored by the config's reward terms, \boxed{} exact match by default. Trainer DistributedOnlineGRPOTrainer, script scripts/training/grpo/online.py. A separate vLLM server generates and receives weights over NCCL; the trainer scores rewards.
Parallelism: FSDP2 data parallelism (HSDP across NVLink domains), EP, TP, EP+TP, EP+ETP and pure ETP (ep_size=1). CP and PP are refused before the model loads. Which families: Rollout Servers.

Dataset
Two columns: prompt (a string or a {"role", "content"} message list) and answer (the ground truth).
JSON Lines
{"prompt": "What is 2 + 3? Put your answer in \\boxed{}.", "answer": "5"}
{"prompt": [{"role": "user", "content": "What is 2 + 3?"}], "answer": "5"}prompt_fieldandanswer_fieldmust name real columns; a typo raises at load.- A row with a null or empty prompt is dropped at load.
system_promptis prepended unless the row already opens with a system turn.- A rendered prompt over
max_prompt_lengthis dropped, never truncated.
The recipes pull openai/gsm8k, trl-lib/DeepMath-103K and open-r1/DAPO-Math-17k-Processed.
Rewards
Each rewards: term becomes one TRL reward function named after it, weighted through GRPOConfig.reward_weights and logged as rewards/<name>/mean. The default is one accuracy term. The list is YAML-only; the CLI cannot set it.
Accuracy is strict: the answer keeps only its text after a GSM8K ####, both sides drop , and $, braces match by depth, and an empty extraction never scores. The environments' grader also lowercases and accepts rtol=0.01 numeric matches.
A judge reads the answer column as its reference. A veto judge (one listing checks) is refused at config time; list requirements instead.
A judge or reward_model term reads the row's conversation and answer columns, so keep remove_unused_columns: false (TRL's GRPO default): at true TRL forwards neither column and the term raises at its first scoring call.
Configuration
Fields: OnlineGRPOScriptArguments, TRL's GRPOConfig, ModelArguments and ParallelismArguments.
YAML
model_name_or_path: Qwen/Qwen3-4B-Instruct-2507
dataset: trl-lib/DeepMath-103K
answer_field: solution
beta: 0.0 # no KL anchor, no reference model
num_generations: 8 # the GRPO group size
max_prompt_length: 1024 # dataset filter
max_completion_length: 2048 # generation budget; required
use_vllm: true # server mode is mandatory
vllm_mode: server
learning_rate: 1.0e-06max_completion_lengthmust be a positive int;nullraises, and leaving it out takes TRL's default, so set it. With both length knobs bounded, their sum becomesmodel_max_length.- The recipes (
examples/grpo/online/) runloss_type: grpo,beta: 0,epsilon0.15–0.2,num_generations4 or 8 andmax_completion_length128–4096; a field a config leaves out takes TRL's default (GRPOConfig). reasoning_effort(default unset;low/medium/high/random) is applied at dataset-map time and needs a template that reads it.- Learning rate:
1e-6to5e-6in the recipes, all full fine-tunes; LoRA takes more (sizing).
Optional objective changes
All off by default.
The std floor, the drop and RLRR also require multi_objective_aggregation: sum_then_normalize.
The rlrr_* tunables are range-validated always and refused at a non-default value with use_rlrr off. examples/grpo/online/qwen3/online-grpo-qwen3-8b-rlrr-math.yaml runs RLRR at these defaults:
YAML
rlrr_mode: hrr # or prr
rlrr_tau: 0.1
rlrr_lambda: 2048.0
rlrr_xi_pos: 1.0e-3 # clip band; xi_neg > xi_pos is refused
rlrr_xi_neg: -1.0e-3
rlrr_std_normalize: false
rlrr_length_rerank: true
rlrr_correctness_clip: true
rlrr_correctness_threshold: 0.5 # the only correctness signal; no gold labelsImportance sampling and the server probe
vllm_importance_sampling_correction (TRL default on) weights the loss by exp(logπ_recompute − logπ_sampling). The default sequence_mask mode masks a sequence whose ratio exceeds vllm_importance_sampling_clip_max (3.0) or falls below vllm_importance_sampling_clip_min (unset); the *_truncate modes clamp instead. With the correction off (and aligned accumulation), the 5-nat k3 KL tail clamp and kl_clamp_frac cannot engage, so beta > 0 warns.
A server restarted between syncs is not recovered here: the push raises on every rank, where async GRPO reconnects and re-sends (Rollout Servers).
Rank 0 waits for the server at startup, up to vllm_server_timeout, and raises past it. Its probe then refuses a
server off the toolkit's image or, on a MoE, not on --moe-backend triton (Startup preflight), and one whose logprobs the correction cannot use:
- raw pre-temperature logprobs at
temperature != 1.0; serve vLLM with--logprobs-mode processed_logprobs; - logprobs renormalized over a sampling cut (
top_p < 1, atop_kor amin_p) while the correction sums the per-token log-ratios over each sequence, which stalls the run silently. Keep all three off. Asequence_*mode sums them, and so doesloss_type: vespoin either token mode TRL lets it run.
Under loss_type: vespo with token_mask, TRL zeroes each ratio outside the clip bounds, which floors that sum, so one such token silences its whole sequence; token_truncate clamps instead.
TRL sends top_k, min_p and repetition_penalty on every request, so the model's generation_config.json defaults never apply (Rollout Servers).
Forced and refused settings
- Dropout is off, EP expert-LoRA dropout included, overriding
disable_dropout: false; a config-level float only warns. - MoE balancing is off:
aux_lossis inert under a policy-gradient loss, and both bias-update modes downgrade tonone(Callbacks). - TRL's
use_liger_kernelis off: it routes the log-probs through TRL's chunked LM head, past the family's head transform. Model-level Liger kernels still apply. text_only_modelis refused by the script: the sync forwards names verbatim, and the text-only sibling'smodel.layers.*would miss a multimodal server'smodel.language_model.layers.*.- Token budget refused at load:
reject_unsupported_modelsizesper_device_train_batch_size × (max_prompt_length + max_completion_length)against the proxy-GIN ceiling on a group spanning NVLink domains (cross-node EP) before any weight is read (Dispatch token ceiling). - Refused at construction (online and async GRPO):
use_adaptive_entropy(a sharded resume does not restore its coefficient), a non-zeroentropy_coefwithuse_chunked_grpo_logprobs(the chunked sweep returns a detached entropy), and anIterableDataseton the DP-sharded loader that TP, ETP and pre-sharded datasets use.
GRPO objective for verifiable rewards
For a shaped reward in [0, 1] (a grade plus small shaping terms) over a small group, set these TRL GRPOConfig fields (async GRPO too).
- A single binary reward keeps
scale_rewards: group: the1e-4added to the divisor keeps a uniformly-failed group's advantages at zero, not NaN. - The clip binds only when a generation batch outlives one optimizer step. At the defaults (
num_iterations: 1,steps_per_generation=gradient_accumulation_steps)epsilonis inert.epsilon_highfalls back toepsilon.
beta is the KL anchor to the base model; every online recipe runs beta: 0. TRL weights the KL term by the importance-sampling ratio (use_bias_correction_kl, on by default), which changes its gradient even on-policy; use_bias_correction_kl: false keeps the plain k3 term. Without a PEFT wrapper (a full fine-tune, or expert-only LoRA, which builds no PeftModel), beta != 0 holds a frozen reference:
- The script loads it as the policy loads (
load_reference_model_for_on_policy_grpo,halo/model_loading/parallel/frozen_models.py): run dtype,model_revision, the policy's attention request andreset_sinks, the non-persistent buffers repaired, the vocabulary grown with the tokenizer. The trainer hands it to TRL in place of the fp32 copy TRL would build from the hub's default revision. - It is an unsharded dense replica on every rank. Before it loads, the script sizes it from a config-only meta shell at the run dtype and refuses, on every rank, a reference over half the GPU memory free beside the loaded policy or whose copies across the node's ranks (each stages its own in host memory) exceed 80% of the available host RAM; the error names
beta: 0, an attention LoRA target and offline GRPO's precomputed reference. This covers plain FSDP2 as much as EP, ETP and TP. The check runs twice: before the policy loads, against the free memory then (a run it refuses would fail the later check too), and again after it, against the memory the loaded policy leaves. The half is a heuristic: optimizer state, gradients and activations are not budgeted, so a reference that passes can still run out of memory at the first step. A reference that fits is warned about under EP, ETP and TP when the trainer is built. A resume loads it frommodel_name_or_path, never the checkpoint. - A trainer constructed with
beta != 0and noref_modelraises, as does aref_modelthe run never reads (beta: 0, or a PEFT policy, whose reference is the policy with its adapter disabled), both before the rollout client connects. - A reference whose sinks differ from the policy's is rejected in every mode, since the KL would be biased on every token. The script's reference takes the policy's
reset_sinks, so live sinks (reset_sinks: false) score on both sides.
Chat template handling
The trainer's tokenizer renders each prompt and TRL sends vLLM the token ids, so a chat_template: override needs no server-side setting (unlike async GRPO). A tools_field column rides through as tools=. The leading BOS is stripped only when the tokenizer's post-processor adds its own, so a template-emitted BOS (Gemma 4) survives.
Chunked log-probs
use_chunked_grpo_logprobs: true (default off) replaces the full [B, T, vocab] logits with a vocab-chunked softmax, for large vocabularies with long completions (Offline GRPO, limits). Here a multimodal batch falls back to the full path, and a PEFT adapter on lm_head raises.
Packed micro-batches
packed_microbatch_tokens (default off) groups each rank's rows into token-budget bins that run as one flattened varlen call each, in TRL's old-policy and reference passes and in the loss forward, instead of one call per row at per_device_train_batch_size: 1. It needs use_chunked_grpo_logprobs: true and is refused with SDPG; mechanism, objective and the other refusals are on Packed micro-batches.
Importance ratio from the loss forward
loss_forward_importance_weights (default off) drops TRL's no-grad old-policy pass, one forward over the round's rows, and builds the vLLM importance-sampling ratio and the KL tail clamp from the loss forward's detached log-probs. Refusals, mechanism and metrics: Weight from the loss forward.
Rank balance
balance_rank_tokens (default off) deals each round's rows across ranks so every rank trains a similar token total; the objective is unchanged. Refusals and mechanism: Rank balance.
Server mode only
DistributedOnlineGRPOTrainer rejects two generation paths:
- In-process generation (
use_vllm: false,vllm_mode: colocate), at init: two NCCL worlds on the same GPUs deadlock (Rollout Servers). - TRL's tool-calling loop (
tools,environment_factory), which desynchronizes ranks. Tool use trains on Async GRPO with Environments.
TRL generates through TrlVLLMClient (halo/trainers/grpo/rollout/trl_vllm_client.py): one batched /v1/completions request per round, bounded by HALO_VLLM_GENERATION_TIMEOUT_SECONDS. The completion ids are the server's sampled ids (return_token_ids); a server that omits them is refused rather than re-tokenized. TRL's vllm_structured_outputs_regex goes out as vLLM's structured_outputs.regex. The route is text-only, so a prompt with an image raises.
Before the trainer is built, every rank raises if max_prompt_length + max_completion_length exceeds the served max_model_len.
Weight sync
Weights are pushed before the next generation, not at the optimizer step; a resumed run pushes before its first rollout (mechanics). The script refuses a family no engine sync reaches right after init (reject_unsyncable_config), and reject_unsyncable_model refuses at construction what the sync cannot carry (construction gates), including a PEFT layer the sync cannot fold or an adapter on a tensor the dense push does not send (PEFT). The first push is checked against the trainer's own log-probs (first-push parity probe). rollout_weight_precision: fp8_block pushes block-FP8 weights to a server booted from the export tool's checkpoint (FP8 rollouts).
Multi-homed nodes (VLLM_GROUP_HOST)
Set VLLM_GROUP_HOST to the trainer IP a remote server should dial when it cannot dial back to the trainer's address on the route to it (NAT, asymmetric routing); this trainer has no per-server group_host (TRL builds its client without one). Resolution order and the rendezvous: Group rendezvous.
Launch
vLLM must own GPUs no trainer rank uses. The compose defaults put the server on GPU 7 (VLLM_CUDA_DEVICES) and the trainer on 0–6 (TRAINER_CUDA_DEVICES).
Bash
# vLLM on GPU 7 (flags: Rollout Servers)
make build-vllm # or set VLLM_IMAGE to the public image
export HALO_SCRATCH=/path/to/large/volume # or set it in .env; compose refuses to start without it
VLLM_MODEL=Qwen/Qwen3-30B-A3B docker compose -f docker-compose.vllm.yml up vllm-server
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6 torchrun --nproc_per_node=7 \
scripts/training/grpo/online.py examples/grpo/online/rlvr-online-grpo-template.yaml \
--model_name_or_path=Qwen/Qwen3-30B-A3Bhalo launch grpo/online <config> --nproc 7 builds the same line (FSDP2 data parallelism); the device split stays yours. Add --tensor_parallel_size for dense TP or --expert_parallel_size for MoE.
Under EP, expert_parallel_size equal to the training-GPU count forms one DeepEP group, as the recipes do. expert_parallel_size: 2 also runs as several 2-rank groups, and node-local groups repeat across NVLink domains:
- Within one NVLink domain, a group above
expert_parallel_size: 2narrower than the domain is rejected (DeepEP). num_expertsmust divide byep_size, so most rosters want a power-of-two split: trainer on GPUs 0–3 with--expert_parallel_size=4, server onVLLM_CUDA_DEVICES=4,5,6,7.- Under EP+TP,
tp_sizemust divide the NVLink domain andep_sizemust be a multiple oftp_size.
LoRA
Add use_peft: true and the lora_* fields. LoRA runs under FSDP2 DP, EP, EP+ETP and pure ETP (under ETP attention adapters only: native expert LoRA is refused at expert_tensor_parallel_size > 1); under TP and EP+TP every adapter is rejected, native expert adapters included.
The weight sync folds each adapter, native expert-LoRA included, into its base weight as it sends it; vLLM loads no adapter and the trainer's frozen base is never written (PEFT).
Data flow and batch construction
Each row becomes a group of num_generations completions. Under TP and ETP every rank takes its group leader's completions; TRL's rollout_func is refused there. The collator is TRL's row collator (left-padded prompts, right-padded completions), hence no CP. Rows stay padded; packed_microbatch_tokens packs them into flattened bins as a separate step (Packed micro-batches).
Counting. Rollouts per optimizer step = per_device_train_batch_size × gradient_accumulation_steps × data_parallel_size. per_device_train_batch_size counts completions, not prompts; with the completion length, it sets per-forward memory.
Geometry. Under TP, ETP or a pre-sharded dataset the dataloader enforces both rules when it is built (elsewhere TRL's config check applies the first):
per_device_train_batch_size × steps_per_generation × data_parallel_sizemust divide bynum_generations. A pre-sharded dataset drops the DP factor.- Under TP/ETP the per-rank product must itself divide by
num_generations.
vLLM on Blackwell (B200)
- Server:
VLLM_ATTENTION_BACKEND=FLASH_ATTNonly where FlashInfer JIT-fails on SM 10.0 (Rollout Servers). - Trainer: all three GRPO scripts give an unset
attn_implementationthe padded-batch default — the loader's own pick on Blackwell (SDPA for a remote-code checkpoint, FA2 for a softcapped policy), SDPA elsewhere — and request nothing underreset_sinks: falseand offline GRPO under CP (padded workloads). A pinned value wins.
Testing a setup
Run a 3-step smoke config against a live server first: examples/grpo/online/qwen3/online-grpo-qwen3-4b-smoke.yaml (dense) or .../gptoss/online-grpo-gptoss-20b-ep-smoke.yaml (EP).
CPU: pytest tests/cpu/grpo -m cpu. GPU: tests/gpu/trainers/grpo/test_online_grpo_vllm_e2e.py (the PEFT × parallelism × resume matrix against a live server, one --mode per row).
What to watch
Metric names follow TRL's GRPOTrainer. Read every run: rewards/accuracy/mean, frac_reward_zero_std, sampling/importance_sampling_ratio/mean (near 1 when trainer and engine agree), completions/clipped_ratio, entropy, and each external term's (judge, reward_model) <source>/<name>/scored. Toolkit additions:
kl_clamp_frac: share of loss tokens whose reference log-ratio hits the 5-nat clamp, atbeta > 0.sampling/degenerate_group_frac: share of completionsdrop_degenerate_groupsmasked out.phase_time/<phase>_sandphase_time/<phase>_max_sunderlog_phase_times: true: where each step's wall time goes, rank mean and slowest rank (Phase times).rank_balance/tokens_max_over_mean_beforeand_afterunderbalance_rank_tokens: the busiest rank's attended tokens over the mean, before and after the deal (Rank balance).- Under
enable_efficiency_metrics:step_tokens_per_second/avg_tokens_per_second(attended tokens, prompt plus completion, which TRL counts while it builds the round; SDPG's teacher forward is not counted) andstep_data_wait_seconds/avg_data_wait_seconds(EfficiencyCallback). - A first-push parity warning in the log when the served model's log-probs on a fixed row stray past the probe's bound from the trainer's;
HALO_WEIGHT_SYNC_PARITY_PROBE=raisestops the run,offskips it (probe). advantage/net_token_massevery round (an early sign of entropy drift) and, underbalance_token_mass,advantage/token_mass_scale; the sign reads as the entropy push only under a token-sum loss such asdapo.
early_stop_entropy_band and early_stop_logratio_gap work as on Async GRPO; the gap needs vllm_importance_sampling_correction and reads TRL's sampling/sampling_logp_difference/mean (a mean absolute difference), so a threshold read off an Async GRPO run does not carry over. save_completions (default on) writes <output_dir>/completions/completions_<step>.parquet (step zero-padded to five digits, _eval suffix for evaluation).
Eval padding. An eval split's final round is padded with repeats of its first prompts, which TRL counts in generation, loss and metrics, so a split that does not fill whole rounds over-weights them. Async GRPO with Environments never rolls the padding out: its rollout metrics and completions count the split's own episodes alone (Padding and round means).
Related pages
Offline GRPO · Async GRPO with Environments · Rollout Servers · Trainer Architecture
Source on GitHubUpdated 18 November 2026