vLLM is an open-source inference and serving engine for large language models. It serves many requests at once with continuous batching and paged memory for the attention cache, behind an OpenAI-compatible API.
How it works
Serving one request at a time wastes a GPU. vLLM is built around two ideas:
- Continuous batching. New requests join the running batch as soon as others finish, so the GPU stays full.
- PagedAttention. The key-value (KV) cache, which holds attention state for tokens already processed, is stored in fixed-size blocks handed out on demand, like pages of virtual memory. Less memory is wasted, so more requests fit.
vLLM loads Hugging Face models and exposes an OpenAI-compatible HTTP API, so existing clients work with it.
When to use it
Use vLLM to serve an open-weight model to many users, or to generate in bulk: evaluation, synthetic data, or rollouts for reinforcement learning (RL).
In Halo
vLLM is Halo's default rollout server. Both online RL methods, online GRPO and async GRPO with environments, can generate against a vLLM 0.26.0 container that runs on GPUs the trainer does not use:
Bash
VLLM_MODEL=Qwen/Qwen3-30B-A3B VLLM_CUDA_DEVICES=6,7 VLLM_TP=2 \
docker compose -f docker-compose.vllm.yml up -d vllm-serverAfter an optimizer step the trainer streams the new weights into the running server, with no restart. Halo writes checkpoints in standard Hugging Face form, so vLLM can also serve the trained model where its loader reads that model family. See rollout servers.
