Offline GRPO
Offline GRPO trains on pre-collected, off-policy data: several completions per prompt, each with a pre-computed reward. Trainer OfflineGRPOTrainer, script scripts/training/grpo/offline.py, config OfflineGRPOConfig (field reference).
- Models: decoder-only, text-only. An encoder-decoder model is refused at construction, and the script refuses an image column.
- Parallelism: EP, TP, EP+TP, EP+ETP, pure ETP (
ep_size=1), CP, EP+CP, PP, PP+EP and PP+ETP. - CP: full fine-tuning only. PEFT and native expert LoRA are rejected under CP, even at
kl_beta: 0. - PP: full fine-tuning with both
max_prompt_lengthandmax_completion_lengthset; rows pad to the two caps' sum.use_chunked_grpo_logprobs,compute_metricsand PEFT are refused (Trainer support).

Dataset
Three conversational columns. A group is one row: put every completion of a prompt in one row, since two rows with the same prompt stay two groups.
JSON Lines
{"prompt": [{"role": "user", "content": "What is 2+2?"}], "completions": [[{"role": "assistant", "content": "4"}], [{"role": "assistant", "content": "5"}]], "rewards": [1.0, -0.5]}- A singleton or exactly-tied group gets advantage 0 under every method.
- A
completions/rewardslength mismatch, an empty tokenized prompt, or a non-finite reward raises, naming the row or group. - Strict templates (Qwen3.5) work: the script renders
prompt + completionand strips the prompt. - A
tools_fieldcolumn is passed astools=. - Every split, one passed to
evaluate()included, must be a map-styledatasets.Dataset: the sampler cuts rows by group, and anything else raises.
scripts/inference/reward_model/rejection_sample.py emits this shape with --output_format offline_grpo.
Configuration
YAML
model_name_or_path: Qwen/Qwen3.6-35B-A3B
dataset:
- path/to/gsm8k_<model>_offline_grpo.jsonl
advantage_method: quantile_norm
loss_type: bnpo
kl_beta: 0.0
initial_min_log_prob: -0.5
max_prompt_length: 512
max_completion_length: 6144
expert_parallel_size: 8
learning_rate: 5.0e-06Per token the default policy_gradient_formulation: prob_weighted trains -(π·A), and reinforce trains -(log π·A). min_log_prob floors the log-probs of negative-advantage rows only, below 0, ramped linearly from initial_min_log_prob when that is set (refused with no floor to ramp toward); kl_beta is the k3 KL coefficient (Schulman's KL estimator, Reference model). Types and defaults: OfflineGRPOConfig.
- The shipped recipes run
learning_rate: 5.0e-06, the top of the SFT full-FT band; a wider batch helps (sizing). - Both length knobs truncate, and the tokenizer's window is pinned only when both are bounded. A completion cut at its cap gets no EOS, so it is never taught to stop there.
Advantages
Rewards map to advantages per group at preprocessing, then clip to [-10, 10].
best_completion_emphasistakes0.0(off), a float above1.0(2.0= 2× weight) or"auto"(3.0 + 2.0·std/(1.0 + std)); any other value raises.drop_degenerate_groups: trueraises if it empties the dataset. Near-ties are kept at full scale under the rank methods.
Loss types
All three weight each example by 1/group_size, so every group counts equally.
bnpo(default): global token average over the micro-batch's group-weighted token sum.grpo: per-sequence average, then a weighted mean across groups. Use when completion length varies.dr_grpo: divides by effective group count ×max_completion_length, a constant that removes length bias; anullcap raises at init.
Row positions
DeepSeek-V4 and Mistral 4 read a token's row index in attention: DeepSeek-V4's compressors cut KV windows at fixed indices, and Mistral 4's query scale steps at a position. A left-padded row would score as another row, so these families always take the chunked scorer, which forwards each row alone, trimmed to its real tokens (attention_reads_row_positions on the family spec). At per_device_train_batch_size above 1 that is one forward per row before each backward. PP's stages forward the collated batch whole, so DeepSeek-V4 under PP is refused above one row per batch.
Memory: chunked log-probs
The default path materializes [B, T_completion, vocab] logits, the memory peak on wide vocabularies with long completions (twice per micro-batch where the reference is scored live). use_chunked_grpo_logprobs: true computes the same log-probs from last_hidden_state with a vocab-chunked softmax. CP always takes this path, as do the row-position families. Limits: Chunked log-probs.
MoE balancing
moe_balancing: aux_loss warns and does nothing (a policy-gradient loss adds no router aux term), or raises at construction with output_router_logits on and a positive router_aux_loss_coef. bias_update works on a family whose bias exports (Callbacks). The recipes set none.
Reference model
At kl_beta > 0 the loss adds a k3 KL to a reference policy (log-ratio capped at 5 nats).
A supplied reference is moved to the policy's device and frozen in eval mode. A run without KL rejects an unused ref_model.
The sweep
- It needs finite, unsharded
datasets.Datasetsplits. A pre-sharded dataset or a suppliedref_per_token_logpscolumn is refused by the script before the model loads and by the trainer at construction. With KL off, PEFT or expert LoRA, neither restriction applies and unused reference columns are dropped. - It finishes before step 1 and does not rerun on resume.
- Global rank 0 writes the scores on shared storage, each node's local rank 0 on node-local storage; every rank memory-maps the merged file. A write, merge or mapping failure on any rank raises on every rank.
- DPO and KTO share this persistence and resume lifecycle (Checkpoints).
Resume
Every checkpoint carries the raw completion-token scores in reference_logps.pt, which a resume restores as the original anchor (the min_log_prob clamp is applied live).
- Token digests, row counts, reference settings and scoring precision must match: fp32 on the chunked and CP paths, the compute dtype on the full-logits path. A bf16 or fp16 resume off CP that toggles
use_chunked_grpo_logprobsis refused with the regeneration steps. - Non-finite scores or rank-divergent reference values raise.
- From Python, pass the resolved
resume_checkpointat trainer construction and the same path totrain().
A KL checkpoint without its sidecar cannot resume, since rescoring would anchor to the trained weights. The refusal gives the recovery flags. Either:
- copy
reference_logps.ptfrom a complete checkpoint, or - run one step of the same tokenized splits and configuration from the exact original model and revision into a scratch output, and copy its
checkpoint-1/reference_logps.ptover (to each node, with node-local storage). Supplied scores and pre-sharded KL datasets are refused here too.
Scratch cache
The mapped files live under _reference_cache/<launch>/ in output_dir for the whole run (a resume maps a hard link or copy of the sidecar). Hub uploads, export copies and the fresh-output-dir check skip it.
- Each new cache removes only the other launches' scratch whose
_reference_cache/<launch>.lockit can take: a finished run's goes, a still-running job's stays. A mount without locks removes nothing. - Host-local locks (NFS
nolock, Lustrelocalflock, most FUSE mounts) cannot see a job on another host; give that job its ownoutput_dir.
Evaluating new rows
evaluate(new_tokenized_dataset) reuses the original scores for token-identical rows and keeps new rows' scores for the run. Unseen rows need the exact original frozen policy:
- outside CP, pass the run-start weights as
original_reference_model, fully frozen and in eval mode; - under CP, declare evaluation splits before training.
Launch
Bash
torchrun --nproc_per_node=8 scripts/training/grpo/offline.py \
examples/grpo/offline/qwen3_5/offline-grpo-qwen3.6-35b-a3b-gsm8k.yaml --expert_parallel_size=8halo launch grpo/offline <config> --nproc 8 builds the same line. From Python, load the model through load_distributed_model and pass the same ParallelismConfig to the trainer.
Testing a setup
Run max_steps: 10 with save_strategy: "no" on a slice of the data first. A bad loss_type, advantage_method or policy_gradient_formulation fails when the config is parsed.
- CPU:
pytest tests/cpu/grpo -m cpu. - GPU, under
tests/gpu/:trainers/grpo/test_offline_grpo*.py,parallelism/{cp,combined}/test_*cp_grpo_scoring.py(CP1 vs CP2/CP4),trainers/lora/test_lora_offline_grpo*.py; under pipeline parallelismparallelism/pp/test_pp_offline_grpo*.pyand, for save, exact resume and export,parallelism/pp/test_pp_trainer_checkpoints.py --trainer offline_grpo/offline_grpo_kl.
What to watch
Per-sample diagnostics, split by advantage sign, once per log step:
{positive,negative}/{logps,pg_objective,kl,ref_logps}_{mean,std,min,max,range}, plus the pre-clampnegative/*_unclamped_*.klandref_logpsexist only atkl_beta > 0.min_log_prob_restriction: the live clamp floor.- Evaluation prefixes every key with
eval_.
Read positive/logps_mean (stable or rising), negative/logps_mean (falling) and positive/pg_objective_mean.
Related pages
Online GRPO (RLVR) · Async GRPO with Environments · Pipeline Parallelism · Trainer Architecture
Source on GitHubUpdated 18 November 2026