Servers and Launch
GPU split
Trainer GPUs must not overlap the server's: NCCL needs a distinct device for every rank of the sync group. Nothing
checks it up front; CUDA_VISIBLE_DEVICES enforces it, and the compose files put the server on GPU 7
(docker-compose.vllm.yml puts its trainer on 0–6). An overlap fails group formation in init_communicator() within
its 120 s deadline (the INFO line logged just before it names the trainer device); a server that never joins fails
the rendezvous after 120 s on SGLang, 300 s on vLLM.
Rollout backend
rollout_backend: vllm (default) or sglang.
- A model the engine cannot update online is refused at construction (which families each serves).
- SGLang also refuses
rollout_max_thinking_tokens(SGLang) andcarry_reasoning. - TRL's
top_p,top_k,min_p,repetition_penaltyandgeneration_kwargsreach no sampler, so one set away from its TRL default is refused at trainer construction; the first four haverollout_*equivalents (Sampling).temperatureis force-set torollout_temperature, so log-probs are scored at the sampling temperature.
Rank 0 waits for each server at startup, up to rollout_connection_timeout, and raises past it, so a server still
loading is read rather than passed. Then it probes each one:
- It raises when one turn cannot fit the served window:
max_prompt_length+ the environment's measured prompt overhead +rollout_max_tokens. - It warns when the multi-turn worst case does not fit: that prompt budget plus the smaller of
max_turns × rollout_max_tokensandrollout_max_episode_tokens. - It raises on a server off the toolkit's image or launched with a flag the run needs (Startup preflight).
verify_backend()fails the launch on an unreachable external judge.
Starting a server
Bash
make build-vllm
VLLM_MODEL=<hub-id-or-path> VLLM_CUDA_DEVICES=7 \
docker compose -f docker-compose.vllm.yml up -d vllm-server
make build-sglang
SGLANG_MODEL=<hub-id-or-path> SGLANG_CUDA_DEVICES=7 \
docker compose -f docker-compose.sglang.yml up -d sglang-serverCompose refuses to start either file without its host path: HALO_SCRATCH (vLLM, the training service's scratch
volume) and SGLANG_MODEL_DIR (SGLang, the directory it mounts read-only). Export them, or set them once in the
repo-root .env, which compose reads.
Confirm /health (:8000 vLLM, :30000 SGLang) first: rank 0 waits only rollout_connection_timeout (default
120 s, 180 in the code-contests recipes) for a server still loading. Parsers, --moe-backend triton and the rest:
Rollout Servers.
Candidate log-probs
score_centering_top_k reads candidate log-probs from an SGLang server of the toolkit's image, which returns them as flat
arrays; vLLM is refused, and a startup probe refuses a server that does not return them
(Candidate log-probs).
Launching the trainer
Bash
CUDA_VISIBLE_DEVICES=0,1,2,3 DIST_NCCL_TIMEOUT_MINUTES=60 torchrun --nproc_per_node=4 \
scripts/training/grpo/environmental.py <config>- Add
--expert_parallel_size=4for MoE expert distribution. halo launch grpo/environmental <config> --nproc 4builds the sametorchrunline; the device split andDIST_NCCL_TIMEOUT_MINUTESstay yours.accelerate launchworks only for plain data parallelism on a dense model; on a MoE it refuses the EP layers weight sync needs.- Context and Pipeline Parallelism are rejected at config time.
Expert parallelism group size
Dense models and ep1 configs (no expert distribution) skip this. Set expert_parallel_size to the trainer-GPU count to
form one DeepEP dispatch group, as the recipes do. ep_size: 2 also runs as several 2-rank groups, and node-local groups
repeat across NVLink domains. Within one domain, a group above ep_size: 2 narrower than the domain is
rejected at config time: ep4 on 4 trainer GPUs passes, on 8 it does not
(DeepEP).
ep_size: 8needs 9 GPUs or a second node, since the server holds one.- ZeRO-3 (
fsdp_reshard_after_forward: true) is rejected wherever an expert-distribution group exists, since its backward all-gather races the DeepEP combine. It runs only atep_size: 1with no expert TP (ZeRO-2 vs ZeRO-3). beta > 0without PEFT holds a frozen reference: the script loads it like the policy (run dtype,model_revision, attention,reset_sinks), frommodel_name_or_pathand never a resume checkpoint, and hands it to TRL in place of TRL's fp32 copy (Online GRPO). It is an unsharded dense replica per rank, refused before it loads when it exceeds half the free GPU memory or 80% of the node's available host RAM across its ranks (sizing). Usebeta: 0, oruse_peft: truewith an attention target off TP (which refuses adapters), to hold none.
Weight synchronization
A forced sync runs at train-begin, after the resume restore and before the first rollout or eval_on_start; prefetch
starts there too, so no rollout uses pre-restore weights. Later syncs run at the start of a step, before that
round's generation.
sync_weights_every_n_steps(default1) trades freshness for pause time; on slow environments sync every 2–4 steps.rollout_group_port(default51216, either backend) is bound on the trainer host, one listener per server, so two servers sharing a port collide even on distinct hosts. With several servers it is the base: an entry with nogroup_portbindsrollout_group_port + index, or its own kernel-assigned port when the base is0(undersingle_groupit is the one port of the one group). TRL'svllm_group_portis refused here; Online GRPO reads it.- Startup and every push fail fast: an error on any rank raises on all ranks together. An engine restarted between syncs is the exception: the client reconnects and re-sends, with a warning (Streaming and quiesce).
- Each EP layer's expert gather transiently materializes the full expert set on every rank, so its cost scales
with total expert count, not
ep_size.
Multiple servers and prefetch
Prefetch rolls the next rounds out while the current one trains, so each round trains prefetch_depth rounds stale
(one optimizer step each at the default steps_per_generation) and the
importance sampling correction covers the lag. enable_prefetch unset
turns it on with two or more servers and off with a single one (rollout_server_url, default
http://localhost:8000, or a one-entry rollout_server_configs), whose step stays serial with no weight lag. true
prefetches on a single server too; false never prefetches.
YAML
rollout_server_configs:
- {url: "http://localhost:8000", group_port: 51216}
- {url: "http://localhost:8001", group_port: 51217}rollout_sync_topology: single_groupforms one weight-sync group over every server and broadcasts each chunk once, where the defaultper_serversends each chunk to each server over its own group;autopickssingle_groupfor two or more servers that report one model and one set of weight-loading flags (One group over every server).single_grouptakes onerollout_group_portand no per-entrygroup_port. Online GRPO talks to one server and has no such field.rollout_adapter_sync: true(SGLang, LoRA) sends the adapter alone after one base push and pauses nothing: a round in flight finishes on the adapter version it started with, and rounds drawn later name the newest (Adapter sync).- Every push pauses all servers together, so one server overlaps a round with the update the way several do: the rounds in flight wait out the push, then resume under the new weights with the tokens they hold (vLLM freezes them, SGLang requeues them).
- Prefetch runs
prefetch_depthrounds deep (default1): a step pops the oldest submitted round, then submits its own prompts. The worker collects every pending round at once and delivers them in submission order, so a round trainsprefetch_depthrounds after it was drawn. - Until the pipeline is full a step is a cold round: it collects its own prompts synchronously and submits them too.
A run opens with
prefetch_depthcold rounds, each trained twice. - Raise
prefetch_depthwhen every step misses while the servers drain at each round's tail; the next round's episodes fill them. Each round in flight holds its ownmax_concurrent_rollouts, so the servers carry up toprefetch_depthrounds' KV, every sync lands on all of them, and an eval round shares the servers with them. rollout_actor_budgetand an explicitmax_concurrent_rolloutsare refused below1. Aprefetch_depthabove1is refused where prefetch is off, namingenable_prefetch: falseor the opt-in a one-server run left unset.
async/prefetch_hit_rate shows which phase bounds the step, not server health. On a short single-turn environment it
should approach 1; below ~0.8, add servers or raise max_concurrent_rollouts / rollout_actor_budget. On a multi-turn
one the round outlasts the update, so it sits near 0 at prefetch_depth: 1; a deeper pipeline raises it.
Prefetch across a resume
A checkpoint stores each rank's submitted-but-untrained rounds, oldest first (prefetch_pending-<rank>-of-<world>.pt),
and a resume resubmits them in order before its first round.
The same file carries the run's halting counters, so a resume at another world size, or from a checkpoint that has no file, restarts their patience at zero and logs it.
- A save on a generation boundary (every save by default, since
steps_per_generationequals the accumulation steps) resumes onto the batches an uninterrupted run trains. A save inside a round that feeds several optimizer steps trains that round twice. - A resume under a smaller
prefetch_depthresubmits the oldest rounds that depth holds and warns; the newer ones are not trained. Under a larger one it opens with cold rounds until the pipeline is full. - No file for this world size, or one drawn under another group size, round size or DP slice, resumes with cold rounds: those batches are skipped and the next ones train twice.
- Under TP or ETP only each group's leader prefetches: a follower's file holds no rounds, and a resume gives it none to resubmit, whatever the checkpoint's file held.
- A file present on some ranks only (a non-shared filesystem under another rank-to-node placement, or a torn copy)
raises. Resume with the saving placement, or delete the
prefetch_pending-*files for cold rounds, which also resets the halting counters the files carry.


Multi-node
Point ray_address at the cluster head (Ray Cluster) and give each
inference node its own rollout_server_configs entry and group_port, plus group_host where the trainer's routable
address differs per server.
- Use resolvable host names. A loopback URL in a run spanning several nodes is refused at startup: it reaches a
different engine, or none, per node, and only the forwarding rank's node takes weight pushes. A launch on one node
with a
ray_addresshead on it keeps its loopback URL. rollout_actor_budgetis a per-node budget on a local Ray, split over the node's rollout ranks, and the cluster-wide total on a shared one, split over every rollout rank; actors are soft-pinned to the rank's node (pool sizing).- The trainer's NCCL address must be routable from the serving nodes: it defaults to the trainer's address on the route
to each server; set
VLLM_GROUP_HOSTorSGLANG_GROUP_HOSTwhere the server cannot dial that back (NAT, asymmetric routing) (Group rendezvous). - The sync runs over EFA when both sides use their EFA overlays (Servers on other nodes), over sockets otherwise.
- Co-located on one host, run the trainer container with
--network=hosttoo, orlocalhost:8000resolves inside its own namespace.


Shipped recipes
examples/grpo/environmental/<family>/<backend>/: gemma4, gptoss and qwen3_5, each with vllm/ and sglang/
(sglang/ ships ep1 files only). In a filename, -lora- / -full- is the adapter and -ep1 / -ep4 the expert distribution
(none, or a 4-rank DeepEP group). Each header carries its launch line and server flags. One template sits at the root:
environmental-grpo-template.yaml (react_math). code-contests-judge/ holds the audit judge's context_files the
code-contests recipes share (Reward).
Start from gptoss/vllm/gptoss-20b-code-contests-lora-ep1.yaml for a tool-heavy graded environment, or
qwen3_5/vllm/qwen3.6-35b-a3b-react-math-full-ep4.yaml for a light two-tool one.
- Every family recipe runs
beta: 0; the root template runs0.01. - The code-contests recipes drive two servers at
episode_timeout: 2700; launch them withDIST_NCCL_TIMEOUT_MINUTES=60(timeout bounds). The Qwen3.6 curriculum stages runepisode_timeout: 3600and needDIST_NCCL_TIMEOUT_MINUTES=120. The rest are single-server, prefetch off.
Source on GitHubUpdated 18 November 2026