Rollouts are the trajectories a model generates during reinforcement learning. One rollout is a single completion or a whole multi-turn episode with its tool calls, and once it is scored it becomes training data.
What a rollout holds
In reinforcement learning the model is not shown the right answer. It produces text, a grader scores it, and the scored samples drive the update. One rollout holds:
- the prompt;
- the tokens the model sampled, plus the tool calls and their results in a multi-turn episode;
- the reward.
Methods such as GRPO (Group Relative Policy Optimization) collect several rollouts per prompt and compare their rewards.
On-policy and off-policy
A rollout is on-policy when the current weights generated it. It is off-policy when an earlier version of the model generated it, or when it was collected before training, as in offline GRPO.
On-policy training generates inside the training loop, so the weights that generate have to keep up with the weights being trained. That is the job of a rollout server and its weight sync. Rollouts from weights that have fallen behind are stale: they describe a policy the model has already left.
In Halo
Online GRPO generates rollouts on a separate vLLM server, and async GRPO on vLLM or SGLang. num_generations sets how many are sampled per prompt. Every logged step's rollouts land in <output_dir>/completions/ as parquet, which is where to look when rewards are all zero. See Rollout servers.
