Filesystem Handling

When multiple ranks download a model, load a dataset, or create a directory, one rank goes first and the others read from the result. The toolkit picks the coordination scope from whether nodes share a filesystem, with separate control over the read and write sides.

Env vars

VarGoverns
DIST_SHARED_FILESYSTEMUmbrella both sides fall back to; shared unless set
DIST_INPUT_SHARED_FILESYSTEMRead side: model/dataset downloads, dataset map/pack, HF caches
DIST_OUTPUT_SHARED_FILESYSTEMWrite side: checkpoints, run.log, dumped artifacts

Defaults and the other DIST_* knobs: Environment variables. Each is read through env_flag: true for 1/true/yes/on and false for 0/false/no/off (case-insensitive). Any other value, a typo included, warns and yields the default, and empty yields it silently, so a side var set to "" still falls back to the umbrella; a recognized value wins over it in either direction.

Shared: global rank 0 goes first, all other ranks wait then read the result — for shared NFS / Lustre / GPFS. Non-shared: each node's local rank 0 operates independently, in parallel — for per-node local NVMe (cloud, RunPod).

All three must be the same on every rank: they pick the coordination scope, so a split value would put participants of one tag under two different scopes where they never see each other's keys. init_distributed() therefore broadcasts global rank 0's resolved values for all three (resolve_shared_filesystem_consensus()) and memoizes them, and a rank whose own env disagreed warns.

The output declaration is additionally checked against the filesystem itself at startup, on multi-node runs only (verify_output_filesystem_sharing): global rank 0 writes a sentinel under output_dir and every rank looks for it, waiting up to DIST_FS_PROBE_TIMEOUT_SECONDS when shared is declared, not at all when per-node is. Either contradiction raises and names the var to set; a shared mount whose attribute cache outlasts the budget (a larger actimeo/acdirmax, or lookupcache keeping the negative lookup) reads as unshared until the budget is raised, and declaring it per-node instead would race every node's writer. The entry scripts probe before resume detection, and every trainer probes again at construction, so a Python-API run is checked too; a directory already probed in the process is not probed twice.

Declared shared but invisible to some rank means only global rank 0 writes trainer_state.json and the save-rank sidecars, so a resume finds every checkpoint incomplete on the other nodes. Declared per-node but visible everywhere means every node's local rank 0 writes the same checkpoint paths concurrently.

The input declaration is checked the same way (verify_input_filesystem_sharing), by every training script right after init_distributed() and before its first Hub read: global rank 0 writes a sentinel under HF_HUB_CACHE and HF_DATASETS_CACHE, in one round. Declared shared but invisible means the other nodes read caches nobody filled (their Hub reads fail, and each of their ranks recomputes every dataset map); declared per-node but visible everywhere means every node's local rank 0 fills the same cache files at once. A cache global rank 0 cannot write (a read-only, pre-staged mount) is filled by nobody and is skipped. The one declaration covers both caches, so both must be on the same kind of storage; an explicit HF_DATASETS_CACHE does not follow HF_HOME.

The split exists because the two sides can want opposite settings on a multi-node run over a slow or flaky shared mount (NFS/EFS). Input wants per-node: global rank 0 writing the HF/dataset cache and remote ranks reading those same inodes is a read-after-cross-node-write that NFS surfaces as OSError: [Errno 116] Stale file handle.

Output is fine shared — checkpoints have a single writer, per-rank dumps are distinct files, and a shared mount gives one authoritative copy with no post-hoc gather. That combination is DIST_INPUT_SHARED_FILESYSTEM=0 with the umbrella left at its shared default.

Coordination primitives

Owning files: halo/runtime/filesystem.py (the shared-FS flags, the writer/reader rank predicates, main-first ordering, fs_aware_makedirs, the output-FS probe), halo/runtime/consensus.py (the c10d-store phase, the store and collective rejection joins, barrier_on_exit) and halo/runtime/load_coordination.py (the per-node load throttle and its world join).

  • fs_aware_main_first(tag, timeout=None) — context manager that orders the body main-rank-first. Every rank runs the body: the main rank runs it alone first and populates a cache, then the others run the same code and hit it.

    Scope follows the input flag. Shared: the main rank is global rank 0 and all other ranks wait. Non-shared: it is each node's local rank 0 and only that node's ranks wait, nodes proceeding in parallel. Wrap any download / cache-writing block in it.

    Waiters block on a c10d key-value store key rather than in a collective, because the body is unbounded single-rank work (a 100B+ snapshot_download) that would trip the NCCL watchdog (DIST_NCCL_TIMEOUT_MINUTES) on peers held in a collective. The wait is bounded by DIST_STORE_TIMEOUT_HOURS instead.

    The bound covers the main rank's work only: there is no trailing join, so the main rank's own wait for the peers is whatever collective comes next, the watchdog, minutes not hours. A misdeclared input flag (shared declared over per-node storage) leaves the peers a cache they cannot see; the input probe refuses that declaration at startup, and the toolkit's model and dataset loads do not rely on this primitive alone either. A model load joins every rank's view of the source and raises on every rank when some cannot see it. The dataset side runs on run_load_rank_first (below), whose second phase is store-joined too: each peer that cannot see the load rank's cache redoes the download or map itself, under the store bound — the same rows, since the ranks then agree what they loaded, at up to one copy of the work per rank.

    Two rules the caller owns. The body must issue no collective, directly or through a helper (ensure_cache_dir() and fs_aware_makedirs() both join across ranks): while the main rank runs it, the peers are on a store key, not in a matching collective, so the main rank would block alone.

    Every rank must also reach a given tag the same number of times, in the same order: keep one tag per call site and never enter one from a rank-dependent branch.

  • hub_metadata_main_first(tag, fetch) — fs_aware_main_first around a single hub-metadata read (config.json, a processor config), returning fetch(local_files_only=...)'s result. Tags live under hub_meta/. Off the load rank fetch is told local_files_only=True: a follower reads the cache its scope's main rank just filled, so each scope makes one Hub query rather than one per rank.

    The bound is 30 minutes rather than the hours-scale store default: a metadata fetch that slow is a dead endpoint, not slow work. It resolves the shared-filesystem consensus itself if that has not happened yet, since these reads can precede init_distributed() and a split scope would put the tag's participants where they never see each other's keys.

  • throttled_load_within_node(max_concurrent) — store-coordinated throttle that admits at most max_concurrent ranks per node at a time. joined_node_load(what, max_concurrent) wraps it with an exit joined over the store, and backs max_concurrent_loading at model load (Multi-Node → Model loading). Same DIST_STORE_TIMEOUT_HOURS bound.

  • is_shared_filesystem() / is_input_shared_filesystem() / is_output_shared_filesystem() — the resolved modes, for picking is_global_main_process() vs is_local_main_process() directly.

  • is_fs_load_rank() — whether this rank performs shared read-side work (downloads, cache fill, the coordinated dataset map). Follows the input flag.

  • is_fs_save_rank() — whether this rank writes shared files. Follows the output flag: shared → only global rank 0 writes (avoids the NFS write race); non-shared → each node's local rank 0 writes its own copy.

    Every gathered/sharded save, the run.log tee, and fs_aware_makedirs by default route their writer choice through it.

  • fs_aware_makedirs(path, writer_rank=is_fs_save_rank) — creates a directory with the correct rank behind barrier_on_exit, so a read-only or full volume raises on every rank. Pass writer_rank=is_fs_load_rank for a read-side cache dir (what halo/entrypoints/run_setup.py does for HF_DATASETS_CACHE).

  • reject_across_ranks(local_reason, what, exc_type=RuntimeError) — the collective that joins a rank-gated body. Every rank calls it with its own reason or None; if any is non-None, every rank raises the same exc_type, naming how many ranks failed, which ones, and the first one's reason.

    Pass exc_type to keep a caller's own error contract (a config gate documented as ValueError). Use it in place of a barrier wherever one rank does work the others wait on, so a failure there aborts the job instead of parking the peers in the barrier until the watchdog.

  • barrier_on_exit(what) — the context-manager form around a rank-gated write: the body's exception is recorded, then joined by reject_across_ranks where a barrier would stand (one all-reduce, so collective order matches a barrier's). The failing rank re-raises its own exception, every peer a RuntimeError naming the failing ranks. The checkpoint sidecars, adapter writes, index writes and the attention warm-up fence on it.

  • gather_rank_reasons(local_reason) — the same gather without the raise, for a caller whose answer to a failure is not always one: every rank gets the world's reasons, indexed by rank, and gathered_failure_summary(reasons, what) turns them into the message reject_across_ranks raises. The optimizer-shard resume decides raise vs warm restart from it.

  • run_on_rank0_raising_everywhere(step, describe, exc_type=RuntimeError) — the same join for work one rank does for the world (an external-backend probe, a rollout-server preflight): step runs on global rank 0 alone and its failure is broadcast, so every rank raises exc_type(describe(error)) together; on rank 0 it chains the original exception.

  • store_reject_across_ranks(tag, local_reason, what, exc_type=RuntimeError, timeout=None) — the same contract carried over the c10d store instead of a collective, for joins whose preceding work is unbounded single-rank time (a fresh-cache dataset map, a first-run shard download).

    Peers wait on store keys bounded by DIST_STORE_TIMEOUT_HOURS (or the explicit timeout), not the NCCL watchdog. World-scoped and collective-equivalent: every rank, same tags, same order. Each rank counts itself in on one arrival counter and the last arrival sets the single key the rest wait on; a failing rank first records its reason in a numbered slot, which every rank reads in one multi_get once all have arrived. A clean join therefore costs each rank a constant number of store requests, O(world) in total on the one store server; a failed one adds each rank's read of the failing ranks' slots, O(failures × world) in total. The coordinated dataset ops and the sharded-dataset loads join through it, and DeferredRankFailure.store_reject(tag) closes a deferred region with it: the checkpoint savers join that way after the per-rank optimizer and sharded-EP shard writes, a gathered save's last part and index, and each pipeline stage's parts.

    The tradeoff: a rank that dies between joins parks its peers on the store for the full bound rather than the watchdog minutes. The timeout diagnostic names the tag, how many ranks arrived and the knob, and the launcher's elastic agent usually reaps the group first.

Python

from halo.runtime.filesystem import fs_aware_main_first

with fs_aware_main_first("teacher_model"):
    snapshot_download(teacher_repo_id)   # one rank fetches; the rest read the populated cache
  • run_load_rank_first(work, operation_name) (halo/data/pipeline/coordinated_map.py) — the ordering for read-side data work: work runs on the is_fs_load_rank() ranks, every rank joins on the store, then the peers run it against the cache and every rank joins again.

    Unlike fs_aware_main_first, a failed first run is never repeated by the peers: the join raises its cause on every rank. A per-node failure takes every node down with the cause instead of leaving the other nodes in their next collective.

The store joins let either side wait out the other's unbounded work (the writer's fresh-cache map, or a peer's full recompute under a misdeclared input flag) under DIST_STORE_TIMEOUT_HOURS instead of the NCCL watchdog.

Wrapping a call in a main-first block (local_main_process_first(), fs_aware_main_first) breaks it. Those hold peers outside the body, so the op's ensure_cache_dir barrier and store joins land at a different sequence position on the main rank than on its peers.

The barrier then pairs with the wrong collective (or blocks alone against a store-key wait), and the joins go permanently off-by-one on the equal-entry invariant. tests/cpu/data/test_coordinated_op_not_main_first.py fails if any call site nests one.

Where it is applied

AreaSideFileNotes
Model downloadsinputhalo/model_loading/parallel/model_source.pyresolve_model_source() runs the snapshot_download fetch in fs_aware_main_first, then joins every rank's view of the source over the store and agrees the commit; a source some ranks cannot see raises on every rank (Multi-Node → Model loading)
Dataset loadinginputhalo/data/sources/loading.pyrun_load_rank_first around the source load (S3, Hub, or local) and the train/test split. A Hub source loads on each scope's load rank alone, which hands the scope its memory-mapped copy (the pickle of its tables, file paths under the probed HF_DATASETS_CACHE), so the other ranks make no Hub request; a rank that cannot open those files raises on every rank, so a shared HF_DATASETS_CACHE must be mounted at the same path on every node, which the startup probe does not check. The preprocessed-dataset probe runs once per filesystem scope (agree_input_probe_across_ranks) and its verdict is shared; a probe that raises raises its cause on every rank. A sharded dataset coordinates per shard with a file lock instead, and joins the world over the store so one rank's shard failure aborts every rank with the cause
Dataset map / filterinputhalo/data/pipeline/coordinated_map.pycoordinated_map / coordinated_filter — self-ordering, see the warning above
Dataset packinginputcoordinated_pack (coordinated_map.py)coordinated_dataset_operation around a deterministic cache_file_name, on the map worker count. Each worker packs whole 1000-row batches, the unit TRL's pack_dataset packs alone, so per-node load ranks with different CPU counts pack the same rows. Unguarded, pack_dataset materializes a full corpus copy per rank
TRL dataset preparation (DPO, KTO, BT reward)inputhalo/trainers/trl_dataset_prep.py (TRLDatasetPrepMixin)coordinated_dataset_transform (coordinated_map.py): the load rank runs TRL's _prepare_dataset with its main_process_first barriers disabled and the map cache off, then publishes the result as HF_DATASETS_CACHE/transform-<md5>, keyed on the rows, the tokenizer and every training argument but the run's identity; the rest load it. Its intermediate maps write a staging directory beside that copy (transform-<md5>.tmp-<uuid>, through scratch_dataset_maps), removed when the preparation ends, rather than $TMPDIR. Unguarded, TRL holds the peers in an NCCL barrier for the whole tokenization and writes a full copy per rank per run
EP lazy model loadinginputhalo/model_loading/parallel/lazy_plan.pyA shared prologue orders the meta-device instantiation main-rank-first; each rank then slices its expert range from the safetensors shards
Checkpoint savesoutputhalo/checkpoint/parallel/ep.py, halo/checkpoint/parallel/save.py, halo/trainers/loop/checkpointing.pyWriter-rank selection via is_fs_save_rank(), with HF's save_on_each_node aligned to it (Checkpoints → Multi-node); a non-shared output FS also rejects a multi-node sharded-EP save (the per-rank expert shards are keyed by global rank and would scatter across nodes' local disks with no gather path)
Optimizer shards and metaoutputhalo/checkpoint/parallel/optimizer.pyEvery rank writes its own optimizer_shard_<rank>.pt and the save rank(s) write optimizer_meta.pt, so each node's directory carries its own meta. A resume needs each rank's own shard on its node (Checkpoints). Multi-group EP replica dedup runs on a shared output FS only
Trainer sidecarsoutputhalo/trainers/reference_logps.py, halo/trainers/grpo/environmental/async_rollouts.pyreference_logps.pt from the save rank(s), prefetch_pending-<rank>-of-<world>.pt from every rank. Each read is rank-joined (consensus_read), so a node without its copy fails the resume everywhere
Offline GRPO reference scratchoutputhalo/trainers/grpo/offline/reference_cache.py_reference_cache/<launch>/ under output_dir: the save rank(s) append and merge the scores, and every rank memory-maps the merged file from its own node's copy. Needs an output filesystem that appends and memory-maps; lifetime and cleanup: Offline GRPO → Reference model
run.log teeoutputhalo/entrypoints/run_setup.pySame is_fs_save_rank() predicate as the checkpoint writers, so two nodes can never both hold one run.log
Checkpointing-plan recordoutputhalo/callbacks/gradient_checkpointing_plan.py_halo_run_plan.json in output_dir from the save rank(s); each rank reads its own node's copy and the world takes the highest recompute floor any copy sets (GradientCheckpointingPlanCallback)
Directory creationeitherfs_aware_makedirs()Output dirs by default; read-side cache dirs pass writer_rank=is_fs_load_rank

Storage layout

Fresh-run output-dir validation ignores the toolkit's run logs, filesystem probes, the checkpointing-plan record and _reference_cache/ scratch, including NFS .nfs* remnants inside it. Checkpoints and unrelated contents still require a resume, overwrite_output_dir: true, or a different output directory.

Put read-only models and datasets on shared storage; put the HF cache and temp files on local NVMe.

Bash

export DIST_SHARED_FILESYSTEM=1          # 0 for per-node local storage
export HF_HOME=/local/hf_cache
export HF_DATASETS_CACHE=/local/hf_cache/datasets
export TMPDIR=/local/tmp

HF_DATASETS_CACHE grows without bound across config sweeps: every distinct (function, tokenizer, knobs, dataset, library-version) tuple leaves a permanent cache-*.arrow set, plus packed_*.arrow per packing shape and a transform-* directory per TRL preparation, and hf cache prune reclaims only the hub cache, not these. Reclaim by deleting the directory — every cache name is deterministic, so the next run rebuilds exactly what it needs.

To avoid shared-FS bottlenecks, pre-stage models and datasets to local NVMe before training — snapshot_download(repo_id, local_dir=...) per node (parallel under DIST_SHARED_FILESYSTEM=0), or rsync. With local_dir set, huggingface_hub writes real files, not symlinks (silencing NFS symlink warnings otherwise needs HF_HUB_DISABLE_SYMLINKS_WARNING=1). Reclaim local space with hf cache prune or hf cache rm <repo>.

Multi-node launches: Launch Recipes.

Source on GitHubUpdated 18 November 2026