# Filesystem Handling

When multiple ranks download a model, load a dataset, or create a directory, one rank goes first and the others read from the result. The toolkit picks the coordination scope from whether nodes share a filesystem, with separate control over the read and write sides.

## Env vars

| Var | Governs |
|-----|---------|
| `DIST_SHARED_FILESYSTEM` | Umbrella both sides fall back to; shared unless set |
| `DIST_INPUT_SHARED_FILESYSTEM` | Read side: model/dataset downloads, dataset map/pack, HF caches |
| `DIST_OUTPUT_SHARED_FILESYSTEM` | Write side: checkpoints, `run.log`, dumped artifacts |

Defaults and the other `DIST_*` knobs: [Environment variables](https://whitecircle.com/halo/docs/agent-docs/reference/configuration-reference.md#environment-variables).
Each is read through `env_flag`: **true** for `1`/`true`/`yes`/`on` and **false** for `0`/`false`/`no`/`off`
(case-insensitive). Any other value, a typo included, warns and yields the default, and empty yields it silently,
so a side var set to `""` still falls back to the umbrella; a recognized value wins over it in either direction.

Shared: global rank 0 goes first, all other ranks wait then read the result — for shared NFS / Lustre
/ GPFS. Non-shared: each node's local rank 0 operates independently, in parallel — for per-node local
NVMe (cloud, RunPod).

All three must be the **same on every rank**: they pick the coordination scope, so a split value would
put participants of one tag under two different scopes where they never see each other's keys.
`init_distributed()` therefore broadcasts global rank 0's *resolved* values for all three
(`resolve_shared_filesystem_consensus()`) and memoizes them, and a rank whose own env disagreed warns.

The **output** declaration is additionally checked against the filesystem itself at startup, on
multi-node runs only (`verify_output_filesystem_sharing`): global rank 0 writes a sentinel under
`output_dir` and every rank looks for it, waiting up to `DIST_FS_PROBE_TIMEOUT_SECONDS` when shared is
declared, not at all when per-node is. Either contradiction
raises and names the var to set; a shared mount whose attribute cache outlasts the budget (a larger
`actimeo`/`acdirmax`, or `lookupcache` keeping the negative lookup) reads as unshared until the budget
is raised, and declaring it per-node instead would race every node's writer. The entry scripts probe
before resume detection, and every trainer probes again at construction, so a Python-API run is
checked too; a directory already probed in the process is not probed twice.

Declared shared but invisible to some rank means only global rank 0 writes `trainer_state.json` and
the save-rank sidecars, so a resume finds every checkpoint incomplete on the other nodes. Declared per-node but
visible everywhere means every node's local rank 0 writes the same checkpoint paths concurrently.

The **input** declaration is checked the same way (`verify_input_filesystem_sharing`), by every training
script right after `init_distributed()` and before its first Hub read: global rank 0 writes a sentinel
under `HF_HUB_CACHE` and `HF_DATASETS_CACHE`, in one round. Declared shared but invisible means the other
nodes read caches nobody filled (their Hub reads fail, and each of their ranks recomputes every dataset
map); declared per-node but visible everywhere means every node's local rank 0 fills the same cache files
at once. A cache global rank 0 cannot write (a read-only, pre-staged mount) is filled by nobody and is
skipped. The one declaration covers both caches, so both must be on the same kind of storage; an explicit
`HF_DATASETS_CACHE` does not follow `HF_HOME`.

The split exists because the two sides can want opposite settings on a multi-node run over a slow or
flaky shared mount (NFS/EFS). Input wants per-node: global rank 0 writing the HF/dataset cache and
remote ranks reading those same inodes is a read-after-cross-node-write that NFS surfaces as
`OSError: [Errno 116] Stale file handle`.

Output is fine shared — checkpoints have a single writer, per-rank dumps are distinct files, and a
shared mount gives one authoritative copy with no post-hoc gather. That combination is
`DIST_INPUT_SHARED_FILESYSTEM=0` with the umbrella left at its shared default.

## Coordination primitives

Owning files: `halo/runtime/filesystem.py` (the shared-FS flags, the writer/reader rank predicates,
main-first ordering, `fs_aware_makedirs`, the output-FS probe), `halo/runtime/consensus.py` (the
c10d-store phase, the store and collective rejection joins, `barrier_on_exit`) and `halo/runtime/load_coordination.py`
(the per-node load throttle and its world join).

- `fs_aware_main_first(tag, timeout=None)` — context manager that orders the body main-rank-first.
  **Every rank runs the body**: the main rank runs it alone first and populates a cache, then the
  others run the same code and hit it.

    Scope follows the **input** flag. Shared: the main rank is global rank 0 and all other ranks
    wait. Non-shared: it is each node's local rank 0 and only that node's ranks wait, nodes
    proceeding in parallel. Wrap any download / cache-writing block in it.

    Waiters block on a c10d key-value store key rather than in a collective, because the body is
    unbounded single-rank work (a 100B+ `snapshot_download`) that would trip the
    NCCL watchdog (`DIST_NCCL_TIMEOUT_MINUTES`) on peers held in a collective. The wait is bounded by
    `DIST_STORE_TIMEOUT_HOURS` instead.

    The bound covers the **main rank's** work only: there is no trailing join, so the main rank's own
    wait for the peers is whatever collective comes next, the watchdog, minutes not hours. A
    **misdeclared input flag** (shared declared over per-node storage) leaves the peers a cache they
    cannot see; the input probe refuses that declaration at startup, and the toolkit's model and dataset
    loads do not rely on this primitive alone either. A model
    load joins every rank's view of the source and raises on every rank when some cannot see it. The
    dataset side runs on `run_load_rank_first` (below), whose second phase is store-joined too: each
    peer that cannot see the load rank's cache redoes the download or map itself, under the store
    bound — the same rows, since the ranks then agree what they loaded, at up to one copy of the work
    per rank.

    Two rules the caller owns. The body must issue **no collective**, directly or through a helper
    (`ensure_cache_dir()` and `fs_aware_makedirs()` both join across ranks): while the main rank runs it, the
    peers are on a store key, not in a matching collective, so the main rank would block alone.

    Every rank must also reach a given `tag` the **same number of times, in the same order**: keep
    one tag per call site and never enter one from a rank-dependent branch.

- `hub_metadata_main_first(tag, fetch)` — `fs_aware_main_first` around a single hub-metadata read
  (`config.json`, a processor config), returning `fetch(local_files_only=...)`'s result. Tags live under
  `hub_meta/`. Off the load rank `fetch` is told `local_files_only=True`: a follower reads the cache its
  scope's main rank just filled, so each scope makes one Hub query rather than one per rank.

    The bound is 30 minutes rather than the hours-scale store default: a metadata fetch that slow is
    a dead endpoint, not slow work. It resolves the shared-filesystem consensus itself if that has not
    happened yet, since these reads can precede `init_distributed()` and a split scope would put the
    tag's participants where they never see each other's keys.

- `throttled_load_within_node(max_concurrent)` — store-coordinated throttle that admits at most
  `max_concurrent` ranks per node at a time. `joined_node_load(what, max_concurrent)` wraps it with an
  exit joined over the store, and backs `max_concurrent_loading` at model load
  ([Multi-Node → Model loading](https://whitecircle.com/halo/docs/agent-docs/parallelism/multi-node.md#model-loading)). Same
  `DIST_STORE_TIMEOUT_HOURS` bound.

- `is_shared_filesystem()` / `is_input_shared_filesystem()` / `is_output_shared_filesystem()` — the
  resolved modes, for picking `is_global_main_process()` vs `is_local_main_process()` directly.

- `is_fs_load_rank()` — whether this rank performs shared read-side work (downloads, cache fill,
  the coordinated dataset map). Follows the **input** flag.

- `is_fs_save_rank()` — whether this rank writes shared files. Follows the **output** flag: shared
  → only global rank 0 writes (avoids the NFS write race); non-shared → each node's local rank 0
  writes its own copy.

    Every gathered/sharded save, the `run.log` tee, and `fs_aware_makedirs` by default route their
    writer choice through it.

- `fs_aware_makedirs(path, writer_rank=is_fs_save_rank)` — creates a directory with the correct
  rank behind `barrier_on_exit`, so a read-only or full volume raises on every rank. Pass
  `writer_rank=is_fs_load_rank` for a read-side cache dir (what `halo/entrypoints/run_setup.py` does
  for `HF_DATASETS_CACHE`).

- `reject_across_ranks(local_reason, what, exc_type=RuntimeError)` — the collective that joins a
  rank-gated body. Every rank calls it with its own reason or `None`; if **any** is non-`None`, every
  rank raises the same `exc_type`, naming how many ranks failed, which ones, and the first one's reason.

    Pass `exc_type` to keep a caller's own error contract (a config gate documented as `ValueError`).
    Use it in place of a barrier wherever one rank does work the others wait on, so a failure there
    aborts the job instead of parking the peers in the barrier until the watchdog.

- `barrier_on_exit(what)` — the context-manager form around a rank-gated write: the body's exception
  is recorded, then joined by `reject_across_ranks` where a barrier would stand (one all-reduce, so
  collective order matches a barrier's). The failing rank re-raises its own exception, every peer a
  `RuntimeError` naming the failing ranks. The checkpoint sidecars, adapter writes, index writes and
  the attention warm-up fence on it.

- `gather_rank_reasons(local_reason)` — the same gather without the raise, for a caller whose answer
  to a failure is not always one: every rank gets the world's reasons, indexed by rank, and
  `gathered_failure_summary(reasons, what)` turns them into the message `reject_across_ranks` raises.
  The optimizer-shard resume decides raise vs warm restart from it.

- `run_on_rank0_raising_everywhere(step, describe, exc_type=RuntimeError)` — the same join for work one rank does
  for the world (an external-backend probe, a rollout-server preflight): `step` runs on global rank 0
  alone and its failure is broadcast, so every rank raises `exc_type(describe(error))` together; on rank 0
  it chains the original exception.

- `store_reject_across_ranks(tag, local_reason, what, exc_type=RuntimeError, timeout=None)` — the
  same contract carried over the c10d store instead of a collective, for joins whose preceding work
  is unbounded single-rank time (a fresh-cache dataset map, a first-run shard download).

    Peers wait on store keys bounded by `DIST_STORE_TIMEOUT_HOURS` (or the explicit `timeout`), not
    the NCCL watchdog. World-scoped and collective-equivalent: every rank, same tags, same order. Each
    rank counts itself in on one arrival counter and the last arrival sets the single key the rest
    wait on; a failing rank first records its reason in a numbered slot, which every rank reads in
    one `multi_get` once all have arrived. A clean join therefore costs each rank a constant number
    of store requests, O(world) in total on the one store server; a failed one adds each rank's read
    of the failing ranks' slots, O(failures × world) in total. The coordinated dataset ops and the
    sharded-dataset loads join through it, and `DeferredRankFailure.store_reject(tag)` closes a
    deferred region with it: the checkpoint savers join that way after the per-rank optimizer and
    sharded-EP shard writes, a gathered save's last part and index, and each pipeline stage's parts.

    The tradeoff: a rank that dies *between* joins parks its peers on the store for the full bound
    rather than the watchdog minutes. The timeout diagnostic names the tag, how many ranks arrived
    and the knob, and the launcher's elastic agent usually reaps the group first.

```python
from halo.runtime.filesystem import fs_aware_main_first

with fs_aware_main_first("teacher_model"):
    snapshot_download(teacher_repo_id)   # one rank fetches; the rest read the populated cache
```

- `run_load_rank_first(work, operation_name)` (`halo/data/pipeline/coordinated_map.py`) — the ordering
  for read-side data work: `work` runs on the `is_fs_load_rank()` ranks, every rank joins on the
  store, then the peers run it against the cache and every rank joins again.

    Unlike `fs_aware_main_first`, a failed first run is never repeated by the peers: the join raises
    its cause on every rank. A per-node failure takes every node down with the cause instead of
    leaving the other nodes in their next collective.

> [!WARNING]
> **The coordinated dataset ops are already ordered — never nest them**
>
> Every coordinated dataset op — `coordinated_map` / `coordinated_filter`, the whole-source load,
> the pack — runs through `run_load_rank_first`, which **is** the rank ordering.

The store joins let either side wait out the other's unbounded work (the writer's fresh-cache
map, or a peer's full recompute under a misdeclared input flag) under `DIST_STORE_TIMEOUT_HOURS`
instead of the NCCL watchdog.

Wrapping a call in a main-first block (`local_main_process_first()`, `fs_aware_main_first`) breaks it.
Those hold peers *outside* the body, so the op's `ensure_cache_dir` barrier and store joins land at a
different sequence position on the main rank than on its peers.

The barrier then pairs with the wrong collective (or blocks alone against a store-key wait), and the
joins go permanently off-by-one on the equal-entry invariant.
`tests/cpu/data/test_coordinated_op_not_main_first.py` fails if any call site nests one.

## Where it is applied

| Area | Side | File | Notes |
|------|------|------|-------|
| Model downloads | input | `halo/model_loading/parallel/model_source.py` | `resolve_model_source()` runs the `snapshot_download` fetch in `fs_aware_main_first`, then joins every rank's view of the source over the store and agrees the commit; a source some ranks cannot see raises on every rank ([Multi-Node → Model loading](https://whitecircle.com/halo/docs/agent-docs/parallelism/multi-node.md#model-loading)) |
| Dataset loading | input | `halo/data/sources/loading.py` | `run_load_rank_first` around the source load (S3, Hub, or local) and the train/test split. A Hub source loads on each scope's load rank alone, which hands the scope its memory-mapped copy (the pickle of its tables, file paths under the probed `HF_DATASETS_CACHE`), so the other ranks make no Hub request; a rank that cannot open those files raises on every rank, so a shared `HF_DATASETS_CACHE` must be mounted at the same path on every node, which the startup probe does not check. The preprocessed-dataset probe runs once per filesystem scope (`agree_input_probe_across_ranks`) and its verdict is shared; a probe that raises raises its cause on every rank. A sharded dataset coordinates per shard with a file lock instead, and joins the world over the store so one rank's shard failure aborts every rank with the cause |
| Dataset map / filter | input | `halo/data/pipeline/coordinated_map.py` | `coordinated_map` / `coordinated_filter` — self-ordering, see the warning above |
| Dataset packing | input | `coordinated_pack` (`coordinated_map.py`) | `coordinated_dataset_operation` around a deterministic `cache_file_name`, on the map worker count. Each worker packs whole 1000-row batches, the unit TRL's `pack_dataset` packs alone, so per-node load ranks with different CPU counts pack the same rows. Unguarded, `pack_dataset` materializes a full corpus copy **per rank** |
| TRL dataset preparation (DPO, KTO, BT reward) | input | `halo/trainers/trl_dataset_prep.py` (`TRLDatasetPrepMixin`) | `coordinated_dataset_transform` (`coordinated_map.py`): the load rank runs TRL's `_prepare_dataset` with its `main_process_first` barriers disabled and the map cache off, then publishes the result as `HF_DATASETS_CACHE/transform-<md5>`, keyed on the rows, the tokenizer and every training argument but the run's identity; the rest load it. Its intermediate maps write a staging directory beside that copy (`transform-<md5>.tmp-<uuid>`, through `scratch_dataset_maps`), removed when the preparation ends, rather than `$TMPDIR`. Unguarded, TRL holds the peers in an NCCL barrier for the whole tokenization and writes a full copy per rank per run |
| EP lazy model loading | input | `halo/model_loading/parallel/lazy_plan.py` | A shared prologue orders the meta-device instantiation main-rank-first; each rank then slices its expert range from the safetensors shards |
| Checkpoint saves | output | `halo/checkpoint/parallel/ep.py`, `halo/checkpoint/parallel/save.py`, `halo/trainers/loop/checkpointing.py` | Writer-rank selection via `is_fs_save_rank()`, with HF's `save_on_each_node` aligned to it ([Checkpoints → Multi-node](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#multi-node)); a non-shared output FS also rejects a multi-node sharded-EP save (the per-rank expert shards are keyed by global rank and would scatter across nodes' local disks with no gather path) |
| Optimizer shards and meta | output | `halo/checkpoint/parallel/optimizer.py` | Every rank writes its own `optimizer_shard_<rank>.pt` and the save rank(s) write `optimizer_meta.pt`, so each node's directory carries its own meta. A resume needs each rank's own shard on its node ([Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#warm-restart-vs-exact-resume-torchrun)). Multi-group EP replica dedup runs on a shared output FS only |
| Trainer sidecars | output | `halo/trainers/reference_logps.py`, `halo/trainers/grpo/environmental/async_rollouts.py` | `reference_logps.pt` from the save rank(s), `prefetch_pending-<rank>-of-<world>.pt` from every rank. Each read is rank-joined (`consensus_read`), so a node without its copy fails the resume everywhere |
| Offline GRPO reference scratch | output | `halo/trainers/grpo/offline/reference_cache.py` | `_reference_cache/<launch>/` under `output_dir`: the save rank(s) append and merge the scores, and every rank memory-maps the merged file from its own node's copy. Needs an output filesystem that appends and memory-maps; lifetime and cleanup: [Offline GRPO → Reference model](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/offline.md#reference-model) |
| `run.log` tee | output | `halo/entrypoints/run_setup.py` | Same `is_fs_save_rank()` predicate as the checkpoint writers, so two nodes can never both hold one `run.log` |
| Checkpointing-plan record | output | `halo/callbacks/gradient_checkpointing_plan.py` | `_halo_run_plan.json` in `output_dir` from the save rank(s); each rank reads its own node's copy and the world takes the highest recompute floor any copy sets ([GradientCheckpointingPlanCallback](https://whitecircle.com/halo/docs/agent-docs/training-methods/callbacks.md#gradientcheckpointingplancallback)) |
| Directory creation | either | `fs_aware_makedirs()` | Output dirs by default; read-side cache dirs pass `writer_rank=is_fs_load_rank` |

## Storage layout

Fresh-run output-dir validation ignores the toolkit's run logs, filesystem probes, the checkpointing-plan
record and `_reference_cache/` scratch, including NFS `.nfs*` remnants inside it. Checkpoints and unrelated
contents still require a resume, `overwrite_output_dir: true`, or a different output directory.

Put read-only models and datasets on shared storage; put the HF cache and temp files on local NVMe.

```bash
export DIST_SHARED_FILESYSTEM=1          # 0 for per-node local storage
export HF_HOME=/local/hf_cache
export HF_DATASETS_CACHE=/local/hf_cache/datasets
export TMPDIR=/local/tmp
```

`HF_DATASETS_CACHE` grows without bound across config sweeps: every distinct
(function, tokenizer, knobs, dataset, library-version) tuple leaves a permanent `cache-*.arrow` set,
plus `packed_*.arrow` per packing shape and a `transform-*` directory per TRL preparation, and
`hf cache prune` reclaims only the hub cache, not these. Reclaim by deleting the directory — every cache name is deterministic, so the next run
rebuilds exactly what it needs.

To avoid shared-FS bottlenecks, pre-stage models and datasets to local NVMe before training —
`snapshot_download(repo_id, local_dir=...)` per node (parallel under `DIST_SHARED_FILESYSTEM=0`), or
`rsync`. With `local_dir` set, `huggingface_hub` writes real files, not symlinks (silencing NFS
symlink warnings otherwise needs `HF_HUB_DISABLE_SYMLINKS_WARNING=1`). Reclaim local space with
`hf cache prune` or `hf cache rm <repo>`.

Multi-node launches: [Launch Recipes](https://whitecircle.com/halo/docs/agent-docs/parallelism/launch-recipes.md).
