# What is SGLang?

SGLang is an open-source framework for serving large language models. It reuses cached computation across requests that share a prefix and can constrain output to a format, behind an OpenAI-compatible API.

## How it works

SGLang serves a model to many clients at once through an OpenAI-compatible API. Two features mark its design:

- **Prefix reuse.** Requests often start with the same tokens: a system prompt, a tool list, the earlier turns of a conversation. SGLang keeps the attention cache for those prefixes and reuses it, so shared tokens are computed once. It calls this RadixAttention.
- **Structured output.** It can constrain generation to a JSON schema, a regular expression or a grammar.

## When to use it

SGLang suits workloads with long shared prefixes: multi-turn chat, agents that call tools, few-shot prompts. [vLLM](/glossary/vllm) covers similar ground, and the choice often comes down to which one serves your model and workload better.

## In Halo

Halo can use SGLang as the [rollout server](/glossary/rollout-server) for [async GRPO with environments](/glossary/async-grpo). Set `rollout_backend: sglang` in the training config and start the server on GPUs the trainer does not use:

```bash
SGLANG_MODEL=Qwen/Qwen3-30B-A3B SGLANG_CUDA_DEVICES=7 \
    docker compose -f docker-compose.sglang.yml up -d
```

The supported version is SGLang 0.5.17, served from Halo's own image, which weight sync needs. Online GRPO is vLLM-only. See [rollout servers](/halo/docs/rollout-servers#sglang-instead).

