Rollouts

Rollouts are the trajectories a model generates during reinforcement learning. One rollout is a single completion or a whole multi-turn episode with its tool calls, and once it is scored it becomes training data.

What a rollout holds

In reinforcement learning the model is not shown the right answer. It produces text, a grader scores it, and the scored samples drive the update. One rollout holds:

  • the prompt;
  • the tokens the model sampled, plus the tool calls and their results in a multi-turn episode;
  • the reward.

Methods such as GRPO (Group Relative Policy Optimization) collect several rollouts per prompt and compare their rewards.

On-policy and off-policy

A rollout is on-policy when the current weights generated it. It is off-policy when an earlier version of the model generated it, or when it was collected before training, as in offline GRPO.

On-policy training generates inside the training loop, so the weights that generate have to keep up with the weights being trained. That is the job of a rollout server and its weight sync. Rollouts from weights that have fallen behind are stale: they describe a policy the model has already left.

In Halo

Online GRPO generates rollouts on a separate vLLM server, and async GRPO on vLLM or SGLang. num_generations sets how many are sampled per prompt. Every logged step's rollouts land in <output_dir>/completions/ as parquet, which is where to look when rewards are all zero. See Rollout servers.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL