# Rollouts

Rollouts are the trajectories a model generates during reinforcement learning. One rollout is a single completion or a whole multi-turn episode with its tool calls, and once it is scored it becomes training data.

## What a rollout holds

In reinforcement learning the model is not shown the right answer. It produces text, a grader scores it, and the scored samples drive the update. One rollout holds:

- the prompt;
- the tokens the model sampled, plus the tool calls and their results in a multi-turn episode;
- the reward.

Methods such as [GRPO](/glossary/grpo) (Group Relative Policy Optimization) collect several rollouts per prompt and compare their rewards.

## On-policy and off-policy

A rollout is on-policy when the current weights generated it. It is off-policy when an earlier version of the model generated it, or when it was collected before training, as in [offline GRPO](/glossary/offline-grpo).

On-policy training generates inside the training loop, so the weights that generate have to keep up with the weights being trained. That is the job of a [rollout server](/glossary/rollout-server) and its weight sync. Rollouts from weights that have fallen behind are stale: they describe a policy the model has already left.

## In Halo

[Online GRPO](/glossary/online-grpo) generates rollouts on a separate vLLM server, and [async GRPO](/glossary/async-grpo) on vLLM or SGLang. `num_generations` sets how many are sampled per prompt. Every logged step's rollouts land in `<output_dir>/completions/` as parquet, which is where to look when rewards are all zero. See [Rollout servers](/halo/docs/rollout-servers).

