# Ray Cluster

Ray runs the rollout side of [Async GRPO with Environments](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo.md).
Each training rank (one per training GPU) drives a pool of `EnvironmentActor` processes; an actor
plays one episode against the rollout servers and returns the trajectory with its reward. That is
Ray's entire job: weight sync and gradients go over NCCL, generation is HTTP to
[Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md), and Ray carries only episode dispatch and results.

On a single node Ray configures itself. The implementation lives in
`halo/environments/rollouts/ray_actors.py` and the trainer's `_init_async_components`; the pin is
`ray[default] 2.58.x` (`pyproject.toml`). Terms: [glossary](https://whitecircle.com/halo/docs/agent-docs/reference/glossary.md).

## Lifecycle

Every rollout rank (each TP/ETP group's leader, else every rank) calls `ray.init` (`ignore_reinit_error=True`) and
builds its own actor pool at the start of `train()`, not at trainer construction: there is no shared coordinator rank, and a
constructed-but-untrained trainer starts no Ray. With `ray_address: null` (the default) each rank
starts a local Ray instance, unless a `ray start` instance already runs on the host, in which case
`ray.init` joins it.

Two init settings differ from Ray's defaults:

- **The dashboard is off** (`include_dashboard=False`): Ray's dashboard/jobs API carries known
  RCE/DNS-rebinding CVEs (CVE-2023-48022, CVE-2025-62593), so a Ray instance the toolkit starts never
  opens port 8265. There is no YAML knob to re-enable it; monitor from the host shell instead (below).

    It governs only the instances `ray.init` starts: joining a head someone else launched inherits
    that head's setting, so a manual `ray start` must pass `--include-dashboard=false` itself
    ([below](#multi-node)).

- **Temp-dir fallback**: Ray's object-store socket is an AF_UNIX path with a ~107-byte cap, so when
  Ray's temp root (`RAY_TMPDIR`, else `TMPDIR`) is longer than 40 characters, Ray gets a short link
  under `/tmp` (`halo-ray-<uid>-<hash>`) to `<root>/ray` as its temp dir: the sockets take the short
  path, while the session logs and spilled objects stay on the root's volume.

Teardown runs in `train()`'s `finally`: the prefetch thread stops, cancelling the rounds still collecting (their
episodes run until the actor shutdown), every actor's `shutdown()` is
awaited sequentially, weight-sync communicators close. `ray.shutdown()` is never called; actors die
by reference counting. A truly wedged actor can stall this teardown; kill the trainer container if
exit hangs.

## The actor pool

`EnvironmentActor` is declared `@ray.remote(max_restarts=-1)` and nothing more:

- **Unlimited restarts**: actors are stateless between episodes, so a restarted actor beats a
  permanent hole in the round-robin. Episodes routed to a dead actor become masked rows (zero
  reward, excluded from the loss) rather than a crash.

    A *wholly* dead pool still halts the run: a step in which no episode anywhere in the world
    survived warns, and the second consecutive one raises (`EMPTY_ROLLOUT_STEP_LIMIT = 2`) rather
    than training on all-masked batches.

- **Default resources**: Ray reserves one CPU to place an actor, then releases it: a running actor
  is charged nothing. So Ray's scheduler never caps what the pool actually consumes; what does
  depends on the environment ([below](#what-bounds-actor-concurrency)).

- **Asyncio actor**: episode methods are `async def`, so one actor serves up to Ray's default 1000
  concurrent calls. The pool size is *not* the concurrency limit; `max_concurrent_rollouts` is.

- **Soft node affinity**: each rank pins its actors to its own node
  (`NodeAffinitySchedulingStrategy(soft=True, _spill_on_unavailable=True)`), so a shared cluster does
  not scatter one rank's pool cluster-wide, and placement never fails over it.

    `_spill_on_unavailable` is load-bearing: `soft` alone spills only when the node is dead or
    infeasible, so a live but saturated training node would pin every actor there and stall instead
    of spilling onto the CPU actor tier.

- **Lazy environment build**: the env is constructed inside the actor on its first episode. A broken
  environment constructor surfaces as per-episode errors, not a startup crash.

The actor and the offline eval driver (`run_episode` in `halo/environments/rollouts/eval_runner.py`) play one
episode loop, `play_episode` in `halo/environments/rollouts/episode_loop.py`: reset, the per-turn caps, the step
context, the truncated close, the effort stamp and the cleanup. Both read replies with
`turn_from_chat_completion` (`halo/environments/rollouts/engine_wire.py`) and retry turns under `generate_turn`
(backoff, engine-abort re-issue). The actor keeps its own aiohttp transport and retryable-4xx rule, and
turns a fault anywhere in the episode into a masked row.

A turn the engine **aborts** is re-issued for the
same observation, up to `max_retries` times (default `3`), rather than stepped as a length cut; past
that the episode errors into a masked row.

Dispatch is round-robin over actors and over server URLs, one asyncio task per prompt, gated by a
semaphore of `max_concurrent_rollouts` (default: the larger of a train round's rows and `eval_rollout_batch_size`, so a
round runs in one wave; never below the rank's actor count). An explicit cap below a round's rows warns with the wave
count it implies. The aiohttp connector is unlimited (`limit=0`), so a request never queues inside it where the wait
would count against its deadline.

Each episode carries an `episode_timeout` deadline and each request a `request_timeout` deadline, both
counting engine-serving time only (a weight-sync pause is credited back:
[Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#weight-sync)). The actors read the pause off a zero-CPU clock
actor that the rank's `RolloutManager` feeds at every sync; a request whose deadline reads a dead clock
actor fails its episode into a masked row, and the actor restarts with no pause on record. On episode
expiry the task is canceled and the episode returns as a masked row.

The cancel is `ray.cancel(force=False)`, since Ray forbids force-cancel on async-actor tasks, so a
hard-wedged task is not killable. Any actor failure, `RayActorError` included, also becomes a masked
row; the in-flight episode is lost, not retried.

## Pool sizing

`rollout_actor_budget` (default 256) is an actor **budget**, split over the **rollout ranks**: every rank, or under
TP/ETP only each group's leader (`data_parallel_size` of them). A rank gets `budget // divisor` actors, at least 1:

- `ray_address: null`: the budget is **per node**. Each node's ranks start a local Ray, and the divisor is the rollout
  ranks on that node (`LOCAL_WORLD_SIZE ÷ TP/ETP group size`). The default is 32 actors per rank on 8 GPUs; a
  node never runs more than the budget, whatever the node count.
- `ray_address` set: the budget is the **cluster-wide total**, and the divisor is every rollout rank in the world. More
  rollout ranks than budget overshoots through the floor of 1.

A split that leaves a rank fewer than 4 actors (or fewer than its round's rows, if that is less) logs a warning on
rank 0 at pool start, naming the budget that gives each rank 4.

The split keys off the **config field**, not Ray's own `RAY_ADDRESS` environment variable: an
exported `RAY_ADDRESS` joins the cluster (Ray substitutes it for the `address=None` the toolkit
passes) while `ray_address: null` still has every rank build the full pool. Set the config field.

Three cluster-wide totals bound a large run: actors (above), in-flight episodes
(`rollout ranks × max_concurrent_rollouts` per round in flight: `prefetch_depth` rounds plus a cold or eval
round), and per-node sandbox slots (next section).

**Under TP/ETP only each group's leader rolls out.** A follower skips the acquire, the collection and the prefetch
submit, starts no Ray pool, and takes the leader's results from the round entry's broadcast
(`_broadcast_object_from_tp_leader`), so every group rank tokenizes identical trajectories while actor and
rollout-server load scale with `data_parallel_size`, not `world_size`. A follower still joins every fence and
collective the leader joins, writes its prefetch sidecar with no pending rounds and resubmits none on a resume. Online
GRPO keeps TRL's collective generation, so its followers still take part in it.

Per-server load is `max_concurrent_rollouts × rollout ranks / num_servers` per round in flight
([Rollout Servers — Throughput](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#throughput)).

Actors are async, so size the pool to the env's **blocking** per-episode cost, not to HTTP
concurrency. Sandbox-backed envs (`code_contests`, `swe`) are CPU-bound and bursty; search/MCP envs
are latency-bound, where more actors help only until the remote endpoint rate-limits.

Leave the default unless `async/mean_rollout_latency` rises while the rollout servers sit idle. The
trainer-side batch knobs live under
[Async GRPO with Environments — Batch construction](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/objective.md#batch-construction).

## What bounds actor concurrency

Not Ray; what binds depends on the environment class:

- **Sandbox-backed** (`code_contests`, `codeforces`, `swe`, any env with a `python`/code tool): a
  gate of `HALO_SANDBOX_MAX_CONCURRENCY` slots (default: the CPUs the process may use), **per
  process** (`local`/`bubblewrap` only; no effect under `remote`).

    Each actor is its own process, so N actors on a node can run N × that many concurrent sandbox
    subprocesses unless the var is set so the slots **sum** to the core budget. Per-run rlimits and
    scratch sizing: [Sandboxes](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/environments/sandbox.md#concurrency-and-sizing).

- **Network-bound** (`qa_search`, `react_search`, `mcp`, open-book `exam_qa`): no gate; size against
  the search/MCP endpoint's rate limit, not against cores.

On a single node the actors inherit the trainer's environment, so `HALO_SANDBOX_*` set at launch
reaches them; on a shared cluster see the export rule below.

## Multi-node

The intended large topology is a training node (or nodes), GPU-less CPU nodes for the actor tier,
and inference nodes for the rollout servers; the two deployment scenarios are drawn in
[Async GRPO with Environments — Multi-node deployment](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/setup.md#multi-node).
Start the cluster manually; there are no helper scripts:

```bash
ray start --head --port=6379 --include-dashboard=false   # head node
ray start --address="head-node:6379"                     # each worker node
```

then set `ray_address: "head-node:6379"`. `--include-dashboard=false` is required, not optional:
`ray start` defaults it to on wherever the `ray[default]` dashboard deps are installed (they are, it
is the pin), and `ray.init` joining that head cannot turn it back off, so the head would serve the
jobs API on 8265 that the section above says is never opened.

The toolkit configures no other Ray ports; upstream defaults apply. What must hold across nodes:

- **Rollout-server URLs must resolve from every actor node**: actors call the servers directly over
  HTTP. The trainer refuses a loopback URL in a run spanning several nodes, launch or Ray cluster: a scattered actor
  would reach another engine or none, and an engine on each node takes no push but the forwarding rank's.

- **Actor nodes need the environment's runtime**, not GPUs — per env: a sandbox plus a large
  `TMPDIR` for the coding envs and `swe` (an SWE session's whole working tree lives in `TMPDIR` for
  its episode); outbound network plus the backend's key for `react_search` / `qa_search` / open-book
  `exam_qa`; `npx`/`uvx` for `mcp`. The per-env list:
  [Environments](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/environments.md).

- **Actor processes inherit the `ray start` daemon's environment, not the trainer's.** Export
  `HALO_SANDBOX_*`, `TMPDIR`, search/MCP keys (`SERPER_API_KEY` / `BRAVE_API_KEY` /
  `TAVILY_API_KEY` / `KEENABLE_API_KEY`), and any HF/proxy vars *before* `ray start` on each node;
  nothing forwards the trainer's env to remote actors.

## Data through the object store

Episode results are the heavy Ray payload: a full multi-turn trajectory with token ids, per-token
logprobs, and, under
[R3 routing replay](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/objective.md#routing-replay),
the routing mask, which stays base64-encoded end to end and so remains compact on this path. The
trainer gathers lightweight per-episode summaries across ranks, never full trajectories.

The object store itself runs at Ray's defaults (~30% of RAM, capped by `/dev/shm`); the standard
`--shm-size=128g` launch covers it, and the toolkit configures no spilling.

## Monitoring

The trainer logs three cumulative counters on every **logging** step (from `log()`, not per training
step): `async/total_rollouts`, `async/cumulative_mean_rollout_latency`, and
`async/total_generation_tokens`. They are accumulated from the per-episode summaries the trainer
already gathers across ranks each round, so they are **job** totals, not one rank's pool.

No collective and no call into the actors happens inside `log()`. A `ray.get` fan-out there would
block this rank on single-threaded actors busy mid-rollout while its peers move on to the next NCCL
call, and NCCL then times out the job.

From the host: `ray status` (cluster resources, PENDING actors), `ray logs` (actor stderr). The
dashboard stays off (above). `py-spy` note: the toolkit's stack-dump helpers discover torchrun PIDs
only; actor processes are children of the raylet, so attach explicitly with
`python scripts/profiling/py_spy_diag.py dump --pid <actor-pid>` (the subcommand is required; actor
PIDs show as `ray::EnvironmentActor` in `ps`).

| Symptom | Cause → fix |
|---|---|
| Actors `PENDING`, rollouts never start | No free CPU anywhere in the cluster (a saturated local node spills, it does not pin) → free cores or lower `rollout_actor_budget` |
| `ray.exceptions.RayActorError` in logs | Actor process died (OOM, native crash); the episode is masked and the actor restarts — check `ray logs`, reduce per-node concurrency if OOM |
| Rollouts fail with connection refused on some ranks only | Loopback/unresolvable server URL from an actor node → use routable hostnames in `rollout_server_configs` |
| Search/MCP env scores all zero on a shared cluster | Backend key or `npx`/`uvx` missing from the actor node's env → export before `ray start` |
| Sandbox runs queue far beyond core count | Per-process gate × many actors → set `HALO_SANDBOX_MAX_CONCURRENCY` so slots sum to cores |
| `episode_timeout` errors while the env is healthy | Deadline below real episode latency (long SWE sessions hit this first) → raise it; it must stay under the NCCL watchdog ([`episode_timeout`](https://whitecircle.com/halo/docs/agent-docs/reference/configuration-reference.md#asyncrolloutconfig)) |
