Rollout servers are inference servers that generate a model's completions during reinforcement learning while a separate trainer updates the weights. After each update the trainer pushes the new weights to the server.
Why a separate server
On-policy reinforcement learning needs the model to generate while it trains. Training code is built to compute gradients. Inference engines such as vLLM and SGLang are built to generate many sequences at once. With a rollout server each does its own job: the trainer sends prompts, the engine returns rollouts, and the trainer learns from them.
Weight sync
The server has to generate with the weights the trainer just produced, or its rollouts are stale. So after an optimizer step the trainer sends the new weights to the server, and generation continues with them.
In Halo
The server is a separate vLLM or SGLang container that the trainer talks to over HTTP. The trainer streams the whole model into the running engine over NCCL (NVIDIA's collective communications library), with no restart and no checkpoint on disk.
Bash
VLLM_MODEL=Qwen/Qwen3-30B-A3B VLLM_CUDA_DEVICES=6,7 VLLM_TP=2 \
docker compose -f docker-compose.vllm.yml up -d vllm-server
curl -s localhost:8000/health- The server must own GPUs that no trainer process uses.
- Serve bf16 (16-bit) weights, not a quantized checkpoint.
- Serve mixture-of-experts models with
--moe-backend triton(--moe-runner-backend tritonon SGLang). The compose files default to it.
Online GRPO is vLLM-only. Async GRPO can use SGLang with rollout_backend: sglang. See Rollout servers.
