Ray Cluster
Ray runs the rollout side of Async GRPO with Environments.
Each training rank (one per training GPU) drives a pool of EnvironmentActor processes; an actor
plays one episode against the rollout servers and returns the trajectory with its reward. That is
Ray's entire job: weight sync and gradients go over NCCL, generation is HTTP to
Rollout Servers, and Ray carries only episode dispatch and results.
On a single node Ray configures itself. The implementation lives in
halo/environments/rollouts/ray_actors.py and the trainer's _init_async_components; the pin is
ray[default] 2.58.x (pyproject.toml). Terms: glossary.
Lifecycle
Every rollout rank (each TP/ETP group's leader, else every rank) calls ray.init (ignore_reinit_error=True) and
builds its own actor pool at the start of train(), not at trainer construction: there is no shared coordinator rank, and a
constructed-but-untrained trainer starts no Ray. With ray_address: null (the default) each rank
starts a local Ray instance, unless a ray start instance already runs on the host, in which case
ray.init joins it.
Two init settings differ from Ray's defaults:
The dashboard is off (
include_dashboard=False): Ray's dashboard/jobs API carries known RCE/DNS-rebinding CVEs (CVE-2023-48022, CVE-2025-62593), so a Ray instance the toolkit starts never opens port 8265. There is no YAML knob to re-enable it; monitor from the host shell instead (below).It governs only the instances
ray.initstarts: joining a head someone else launched inherits that head's setting, so a manualray startmust pass--include-dashboard=falseitself (below).Temp-dir fallback: Ray's object-store socket is an AF_UNIX path with a ~107-byte cap, so when Ray's temp root (
RAY_TMPDIR, elseTMPDIR) is longer than 40 characters, Ray gets a short link under/tmp(halo-ray-<uid>-<hash>) to<root>/rayas its temp dir: the sockets take the short path, while the session logs and spilled objects stay on the root's volume.
Teardown runs in train()'s finally: the prefetch thread stops, cancelling the rounds still collecting (their
episodes run until the actor shutdown), every actor's shutdown() is
awaited sequentially, weight-sync communicators close. ray.shutdown() is never called; actors die
by reference counting. A truly wedged actor can stall this teardown; kill the trainer container if
exit hangs.
The actor pool
EnvironmentActor is declared @ray.remote(max_restarts=-1) and nothing more:
Unlimited restarts: actors are stateless between episodes, so a restarted actor beats a permanent hole in the round-robin. Episodes routed to a dead actor become masked rows (zero reward, excluded from the loss) rather than a crash.
A wholly dead pool still halts the run: a step in which no episode anywhere in the world survived warns, and the second consecutive one raises (
EMPTY_ROLLOUT_STEP_LIMIT = 2) rather than training on all-masked batches.Default resources: Ray reserves one CPU to place an actor, then releases it: a running actor is charged nothing. So Ray's scheduler never caps what the pool actually consumes; what does depends on the environment (below).
Asyncio actor: episode methods are
async def, so one actor serves up to Ray's default 1000 concurrent calls. The pool size is not the concurrency limit;max_concurrent_rolloutsis.Soft node affinity: each rank pins its actors to its own node (
NodeAffinitySchedulingStrategy(soft=True, _spill_on_unavailable=True)), so a shared cluster does not scatter one rank's pool cluster-wide, and placement never fails over it._spill_on_unavailableis load-bearing:softalone spills only when the node is dead or infeasible, so a live but saturated training node would pin every actor there and stall instead of spilling onto the CPU actor tier.Lazy environment build: the env is constructed inside the actor on its first episode. A broken environment constructor surfaces as per-episode errors, not a startup crash.
The actor and the offline eval driver (run_episode in halo/environments/rollouts/eval_runner.py) play one
episode loop, play_episode in halo/environments/rollouts/episode_loop.py: reset, the per-turn caps, the step
context, the truncated close, the effort stamp and the cleanup. Both read replies with
turn_from_chat_completion (halo/environments/rollouts/engine_wire.py) and retry turns under generate_turn
(backoff, engine-abort re-issue). The actor keeps its own aiohttp transport and retryable-4xx rule, and
turns a fault anywhere in the episode into a masked row.
A turn the engine aborts is re-issued for the
same observation, up to max_retries times (default 3), rather than stepped as a length cut; past
that the episode errors into a masked row.
Dispatch is round-robin over actors and over server URLs, one asyncio task per prompt, gated by a
semaphore of max_concurrent_rollouts (default: the larger of a train round's rows and eval_rollout_batch_size, so a
round runs in one wave; never below the rank's actor count). An explicit cap below a round's rows warns with the wave
count it implies. The aiohttp connector is unlimited (limit=0), so a request never queues inside it where the wait
would count against its deadline.
Each episode carries an episode_timeout deadline and each request a request_timeout deadline, both
counting engine-serving time only (a weight-sync pause is credited back:
Rollout Servers). The actors read the pause off a zero-CPU clock
actor that the rank's RolloutManager feeds at every sync; a request whose deadline reads a dead clock
actor fails its episode into a masked row, and the actor restarts with no pause on record. On episode
expiry the task is canceled and the episode returns as a masked row.
The cancel is ray.cancel(force=False), since Ray forbids force-cancel on async-actor tasks, so a
hard-wedged task is not killable. Any actor failure, RayActorError included, also becomes a masked
row; the in-flight episode is lost, not retried.
Pool sizing
rollout_actor_budget (default 256) is an actor budget, split over the rollout ranks: every rank, or under
TP/ETP only each group's leader (data_parallel_size of them). A rank gets budget // divisor actors, at least 1:
ray_address: null: the budget is per node. Each node's ranks start a local Ray, and the divisor is the rollout ranks on that node (LOCAL_WORLD_SIZE ÷ TP/ETP group size). The default is 32 actors per rank on 8 GPUs; a node never runs more than the budget, whatever the node count.ray_addressset: the budget is the cluster-wide total, and the divisor is every rollout rank in the world. More rollout ranks than budget overshoots through the floor of 1.
A split that leaves a rank fewer than 4 actors (or fewer than its round's rows, if that is less) logs a warning on rank 0 at pool start, naming the budget that gives each rank 4.
The split keys off the config field, not Ray's own RAY_ADDRESS environment variable: an
exported RAY_ADDRESS joins the cluster (Ray substitutes it for the address=None the toolkit
passes) while ray_address: null still has every rank build the full pool. Set the config field.
Three cluster-wide totals bound a large run: actors (above), in-flight episodes
(rollout ranks × max_concurrent_rollouts per round in flight: prefetch_depth rounds plus a cold or eval
round), and per-node sandbox slots (next section).
Under TP/ETP only each group's leader rolls out. A follower skips the acquire, the collection and the prefetch
submit, starts no Ray pool, and takes the leader's results from the round entry's broadcast
(_broadcast_object_from_tp_leader), so every group rank tokenizes identical trajectories while actor and
rollout-server load scale with data_parallel_size, not world_size. A follower still joins every fence and
collective the leader joins, writes its prefetch sidecar with no pending rounds and resubmits none on a resume. Online
GRPO keeps TRL's collective generation, so its followers still take part in it.
Per-server load is max_concurrent_rollouts × rollout ranks / num_servers per round in flight
(Rollout Servers — Throughput).
Actors are async, so size the pool to the env's blocking per-episode cost, not to HTTP
concurrency. Sandbox-backed envs (code_contests, swe) are CPU-bound and bursty; search/MCP envs
are latency-bound, where more actors help only until the remote endpoint rate-limits.
Leave the default unless async/mean_rollout_latency rises while the rollout servers sit idle. The
trainer-side batch knobs live under
Async GRPO with Environments — Batch construction.
What bounds actor concurrency
Not Ray; what binds depends on the environment class:
Sandbox-backed (
code_contests,codeforces,swe, any env with apython/code tool): a gate ofHALO_SANDBOX_MAX_CONCURRENCYslots (default: the CPUs the process may use), per process (local/bubblewraponly; no effect underremote).Each actor is its own process, so N actors on a node can run N × that many concurrent sandbox subprocesses unless the var is set so the slots sum to the core budget. Per-run rlimits and scratch sizing: Sandboxes.
Network-bound (
qa_search,react_search,mcp, open-bookexam_qa): no gate; size against the search/MCP endpoint's rate limit, not against cores.
On a single node the actors inherit the trainer's environment, so HALO_SANDBOX_* set at launch
reaches them; on a shared cluster see the export rule below.
Multi-node
The intended large topology is a training node (or nodes), GPU-less CPU nodes for the actor tier, and inference nodes for the rollout servers; the two deployment scenarios are drawn in Async GRPO with Environments — Multi-node deployment. Start the cluster manually; there are no helper scripts:
Bash
ray start --head --port=6379 --include-dashboard=false # head node
ray start --address="head-node:6379" # each worker nodethen set ray_address: "head-node:6379". --include-dashboard=false is required, not optional:
ray start defaults it to on wherever the ray[default] dashboard deps are installed (they are, it
is the pin), and ray.init joining that head cannot turn it back off, so the head would serve the
jobs API on 8265 that the section above says is never opened.
The toolkit configures no other Ray ports; upstream defaults apply. What must hold across nodes:
Rollout-server URLs must resolve from every actor node: actors call the servers directly over HTTP. The trainer refuses a loopback URL in a run spanning several nodes, launch or Ray cluster: a scattered actor would reach another engine or none, and an engine on each node takes no push but the forwarding rank's.
Actor nodes need the environment's runtime, not GPUs — per env: a sandbox plus a large
TMPDIRfor the coding envs andswe(an SWE session's whole working tree lives inTMPDIRfor its episode); outbound network plus the backend's key forreact_search/qa_search/ open-bookexam_qa;npx/uvxformcp. The per-env list: Environments.Actor processes inherit the
ray startdaemon's environment, not the trainer's. ExportHALO_SANDBOX_*,TMPDIR, search/MCP keys (SERPER_API_KEY/BRAVE_API_KEY/TAVILY_API_KEY/KEENABLE_API_KEY), and any HF/proxy vars beforeray starton each node; nothing forwards the trainer's env to remote actors.
Data through the object store
Episode results are the heavy Ray payload: a full multi-turn trajectory with token ids, per-token logprobs, and, under R3 routing replay, the routing mask, which stays base64-encoded end to end and so remains compact on this path. The trainer gathers lightweight per-episode summaries across ranks, never full trajectories.
The object store itself runs at Ray's defaults (~30% of RAM, capped by /dev/shm); the standard
--shm-size=128g launch covers it, and the toolkit configures no spilling.
Monitoring
The trainer logs three cumulative counters on every logging step (from log(), not per training
step): async/total_rollouts, async/cumulative_mean_rollout_latency, and
async/total_generation_tokens. They are accumulated from the per-episode summaries the trainer
already gathers across ranks each round, so they are job totals, not one rank's pool.
No collective and no call into the actors happens inside log(). A ray.get fan-out there would
block this rank on single-threaded actors busy mid-rollout while its peers move on to the next NCCL
call, and NCCL then times out the job.
From the host: ray status (cluster resources, PENDING actors), ray logs (actor stderr). The
dashboard stays off (above). py-spy note: the toolkit's stack-dump helpers discover torchrun PIDs
only; actor processes are children of the raylet, so attach explicitly with
python scripts/profiling/py_spy_diag.py dump --pid <actor-pid> (the subcommand is required; actor
PIDs show as ray::EnvironmentActor in ps).
Source on GitHubUpdated 18 November 2026