SGLang is an open-source framework for serving large language models. It reuses cached computation across requests that share a prefix and can constrain output to a format, behind an OpenAI-compatible API.
How it works
SGLang serves a model to many clients at once through an OpenAI-compatible API. Two features mark its design:
- Prefix reuse. Requests often start with the same tokens: a system prompt, a tool list, the earlier turns of a conversation. SGLang keeps the attention cache for those prefixes and reuses it, so shared tokens are computed once. It calls this RadixAttention.
- Structured output. It can constrain generation to a JSON schema, a regular expression or a grammar.
When to use it
SGLang suits workloads with long shared prefixes: multi-turn chat, agents that call tools, few-shot prompts. vLLM covers similar ground, and the choice often comes down to which one serves your model and workload better.
In Halo
Halo can use SGLang as the rollout server for async GRPO with environments. Set rollout_backend: sglang in the training config and start the server on GPUs the trainer does not use:
Bash
SGLANG_MODEL=Qwen/Qwen3-30B-A3B SGLANG_CUDA_DEVICES=7 \
docker compose -f docker-compose.sglang.yml up -dThe supported version is SGLang 0.5.17, served from Halo's own image, which weight sync needs. Online GRPO is vLLM-only. See rollout servers.
