Offline GRPO

Offline GRPO trains on pre-collected, off-policy data: several completions per prompt, each with a pre-computed reward. Trainer OfflineGRPOTrainer, script scripts/training/grpo/offline.py, config OfflineGRPOConfig (field reference).

  • Models: decoder-only, text-only. An encoder-decoder model is refused at construction, and the script refuses an image column.
  • Parallelism: EP, TP, EP+TP, EP+ETP, pure ETP (ep_size=1), CP, EP+CP, PP, PP+EP and PP+ETP.
  • CP: full fine-tuning only. PEFT and native expert LoRA are rejected under CP, even at kl_beta: 0.
  • PP: full fine-tuning with both max_prompt_length and max_completion_length set; rows pad to the two caps' sum. use_chunked_grpo_logprobs, compute_metrics and PEFT are refused (Trainer support).
Offline GRPO pipeline: per-group advantages at tokenization, then a floored per-token loss with optional k3 KL, normalized by loss_type
Offline GRPO pipeline: per-group advantages at tokenization, then a floored per-token loss with optional k3 KL, normalized by loss_type

Dataset

Three conversational columns. A group is one row: put every completion of a prompt in one row, since two rows with the same prompt stay two groups.

ColumnTypeContents
promptlist[dict]Message list ({"role", "content"})
completionslist[list[dict]]One message list per completion (typically 4–16)
rewardslist[float]One reward per completion

JSON Lines

{"prompt": [{"role": "user", "content": "What is 2+2?"}], "completions": [[{"role": "assistant", "content": "4"}], [{"role": "assistant", "content": "5"}]], "rewards": [1.0, -0.5]}
  • A singleton or exactly-tied group gets advantage 0 under every method.
  • A completions/rewards length mismatch, an empty tokenized prompt, or a non-finite reward raises, naming the row or group.
  • Strict templates (Qwen3.5) work: the script renders prompt + completion and strips the prompt.
  • A tools_field column is passed as tools=.
  • Every split, one passed to evaluate() included, must be a map-style datasets.Dataset: the sampler cuts rows by group, and anything else raises.

scripts/inference/reward_model/rejection_sample.py emits this shape with --output_format offline_grpo.

Configuration

YAML

model_name_or_path: Qwen/Qwen3.6-35B-A3B
dataset:
  - path/to/gsm8k_<model>_offline_grpo.jsonl
advantage_method: quantile_norm
loss_type: bnpo
kl_beta: 0.0
initial_min_log_prob: -0.5
max_prompt_length: 512
max_completion_length: 6144
expert_parallel_size: 8
learning_rate: 5.0e-06

Per token the default policy_gradient_formulation: prob_weighted trains -(π·A), and reinforce trains -(log π·A). min_log_prob floors the log-probs of negative-advantage rows only, below 0, ramped linearly from initial_min_log_prob when that is set (refused with no floor to ramp toward); kl_beta is the k3 KL coefficient (Schulman's KL estimator, Reference model). Types and defaults: OfflineGRPOConfig.

  • The shipped recipes run learning_rate: 5.0e-06, the top of the SFT full-FT band; a wider batch helps (sizing).
  • Both length knobs truncate, and the tokenizer's window is pinned only when both are bounded. A completion cut at its cap gets no EOS, so it is never taught to stop there.

Advantages

Rewards map to advantages per group at preprocessing, then clip to [-10, 10].

advantage_methodFormulaBest for
quantile_norm (default)inverse normal CDF of ranksordinal or outlier-heavy rewards
z_norm(r - mean) / (std + ε)normal reward distributions
minmax2(r - min)/(max - min) - 1bounded [-1, +1] advantages
quantile_uniformuniform from ranksmost outlier-tolerant
robust(r - median) / IQRextreme outliers
  • best_completion_emphasis takes 0.0 (off), a float above 1.0 (2.0 = 2× weight) or "auto" (3.0 + 2.0·std/(1.0 + std)); any other value raises.
  • drop_degenerate_groups: true raises if it empties the dataset. Near-ties are kept at full scale under the rank methods.

Loss types

All three weight each example by 1/group_size, so every group counts equally.

  • bnpo (default): global token average over the micro-batch's group-weighted token sum.
  • grpo: per-sequence average, then a weighted mean across groups. Use when completion length varies.
  • dr_grpo: divides by effective group count × max_completion_length, a constant that removes length bias; a null cap raises at init.

Row positions

DeepSeek-V4 and Mistral 4 read a token's row index in attention: DeepSeek-V4's compressors cut KV windows at fixed indices, and Mistral 4's query scale steps at a position. A left-padded row would score as another row, so these families always take the chunked scorer, which forwards each row alone, trimmed to its real tokens (attention_reads_row_positions on the family spec). At per_device_train_batch_size above 1 that is one forward per row before each backward. PP's stages forward the collated batch whole, so DeepSeek-V4 under PP is refused above one row per batch.

Memory: chunked log-probs

The default path materializes [B, T_completion, vocab] logits, the memory peak on wide vocabularies with long completions (twice per micro-batch where the reference is scored live). use_chunked_grpo_logprobs: true computes the same log-probs from last_hidden_state with a vocab-chunked softmax. CP always takes this path, as do the row-position families. Limits: Chunked log-probs.

MoE balancing

moe_balancing: aux_loss warns and does nothing (a policy-gradient loss adds no router aux term), or raises at construction with output_router_logits on and a positive router_aux_loss_coef. bias_update works on a family whose bias exports (Callbacks). The recipes set none.

Reference model

At kl_beta > 0 the loss adds a k3 KL to a reference policy (log-ratio capped at 5 nats).

PolicyReference scoring
Full fine-tuningOne sweep of the run-start policy over the train and evaluation splits before the first update; no second resident model. Same lifecycle under CP
Full fine-tuning with a frozen ref_model (outside CP)That model is swept once and released
PEFT (outside CP)Live at every step with the adapters disabled; no sweep, no sidecar
Native expert-only LoRALive at every step against a required frozen base ref_model, which the script loads from model_name_or_path (never the resume checkpoint) with the policy's revision, attention and sinks settings

A supplied reference is moved to the policy's device and frozen in eval mode. A run without KL rejects an unused ref_model.

The sweep

  • It needs finite, unsharded datasets.Dataset splits. A pre-sharded dataset or a supplied ref_per_token_logps column is refused by the script before the model loads and by the trainer at construction. With KL off, PEFT or expert LoRA, neither restriction applies and unused reference columns are dropped.
  • It finishes before step 1 and does not rerun on resume.
  • Global rank 0 writes the scores on shared storage, each node's local rank 0 on node-local storage; every rank memory-maps the merged file. A write, merge or mapping failure on any rank raises on every rank.
  • DPO and KTO share this persistence and resume lifecycle (Checkpoints).

Resume

Every checkpoint carries the raw completion-token scores in reference_logps.pt, which a resume restores as the original anchor (the min_log_prob clamp is applied live).

  • Token digests, row counts, reference settings and scoring precision must match: fp32 on the chunked and CP paths, the compute dtype on the full-logits path. A bf16 or fp16 resume off CP that toggles use_chunked_grpo_logprobs is refused with the regeneration steps.
  • Non-finite scores or rank-divergent reference values raise.
  • From Python, pass the resolved resume_checkpoint at trainer construction and the same path to train().

A KL checkpoint without its sidecar cannot resume, since rescoring would anchor to the trained weights. The refusal gives the recovery flags. Either:

  • copy reference_logps.pt from a complete checkpoint, or
  • run one step of the same tokenized splits and configuration from the exact original model and revision into a scratch output, and copy its checkpoint-1/reference_logps.pt over (to each node, with node-local storage). Supplied scores and pre-sharded KL datasets are refused here too.

Scratch cache

The mapped files live under _reference_cache/<launch>/ in output_dir for the whole run (a resume maps a hard link or copy of the sidecar). Hub uploads, export copies and the fresh-output-dir check skip it.

  • Each new cache removes only the other launches' scratch whose _reference_cache/<launch>.lock it can take: a finished run's goes, a still-running job's stays. A mount without locks removes nothing.
  • Host-local locks (NFS nolock, Lustre localflock, most FUSE mounts) cannot see a job on another host; give that job its own output_dir.

Evaluating new rows

evaluate(new_tokenized_dataset) reuses the original scores for token-identical rows and keeps new rows' scores for the run. Unseen rows need the exact original frozen policy:

  • outside CP, pass the run-start weights as original_reference_model, fully frozen and in eval mode;
  • under CP, declare evaluation splits before training.

Launch

Bash

torchrun --nproc_per_node=8 scripts/training/grpo/offline.py \
    examples/grpo/offline/qwen3_5/offline-grpo-qwen3.6-35b-a3b-gsm8k.yaml --expert_parallel_size=8

halo launch grpo/offline <config> --nproc 8 builds the same line. From Python, load the model through load_distributed_model and pass the same ParallelismConfig to the trainer.

Testing a setup

Run max_steps: 10 with save_strategy: "no" on a slice of the data first. A bad loss_type, advantage_method or policy_gradient_formulation fails when the config is parsed.

  • CPU: pytest tests/cpu/grpo -m cpu.
  • GPU, under tests/gpu/: trainers/grpo/test_offline_grpo*.py, parallelism/{cp,combined}/test_*cp_grpo_scoring.py (CP1 vs CP2/CP4), trainers/lora/test_lora_offline_grpo*.py; under pipeline parallelism parallelism/pp/test_pp_offline_grpo*.py and, for save, exact resume and export, parallelism/pp/test_pp_trainer_checkpoints.py --trainer offline_grpo / offline_grpo_kl.

What to watch

Per-sample diagnostics, split by advantage sign, once per log step:

  • {positive,negative}/{logps,pg_objective,kl,ref_logps}_{mean,std,min,max,range}, plus the pre-clamp negative/*_unclamped_*. kl and ref_logps exist only at kl_beta > 0.
  • min_log_prob_restriction: the live clamp floor.
  • Evaluation prefixes every key with eval_.

Read positive/logps_mean (stable or rising), negative/logps_mean (falling) and positive/pg_objective_mean.

SymptomCauseFix
Loss NaN or explodingLow-probability tokens on negative-advantage rowsmin_log_prob: -3.0 and max_grad_norm: 1.0
positive/pg_objective_mean falls through trainingDistribution shift from the offline dataAdd kl_beta: 0.05, lower the learning rate
Batch-count equalization left 0 full batchesThe split is too small for this world sizeLower per_device_train_batch_size, shrink the world
OOMFull-vocab logits over long completionsuse_chunked_grpo_logprobs: true, lower the batch or max_completion_length, or raise EP size

Online GRPO (RLVR) · Async GRPO with Environments · Pipeline Parallelism · Trainer Architecture

Source on GitHubUpdated 18 November 2026