Filesystem Handling
When multiple ranks download a model, load a dataset, or create a directory, one rank goes first and the others read from the result. The toolkit picks the coordination scope from whether nodes share a filesystem, with separate control over the read and write sides.
Env vars
Defaults and the other DIST_* knobs: Environment variables.
Each is read through env_flag: true for 1/true/yes/on and false for 0/false/no/off
(case-insensitive). Any other value, a typo included, warns and yields the default, and empty yields it silently,
so a side var set to "" still falls back to the umbrella; a recognized value wins over it in either direction.
Shared: global rank 0 goes first, all other ranks wait then read the result — for shared NFS / Lustre / GPFS. Non-shared: each node's local rank 0 operates independently, in parallel — for per-node local NVMe (cloud, RunPod).
All three must be the same on every rank: they pick the coordination scope, so a split value would
put participants of one tag under two different scopes where they never see each other's keys.
init_distributed() therefore broadcasts global rank 0's resolved values for all three
(resolve_shared_filesystem_consensus()) and memoizes them, and a rank whose own env disagreed warns.
The output declaration is additionally checked against the filesystem itself at startup, on
multi-node runs only (verify_output_filesystem_sharing): global rank 0 writes a sentinel under
output_dir and every rank looks for it, waiting up to DIST_FS_PROBE_TIMEOUT_SECONDS when shared is
declared, not at all when per-node is. Either contradiction
raises and names the var to set; a shared mount whose attribute cache outlasts the budget (a larger
actimeo/acdirmax, or lookupcache keeping the negative lookup) reads as unshared until the budget
is raised, and declaring it per-node instead would race every node's writer. The entry scripts probe
before resume detection, and every trainer probes again at construction, so a Python-API run is
checked too; a directory already probed in the process is not probed twice.
Declared shared but invisible to some rank means only global rank 0 writes trainer_state.json and
the save-rank sidecars, so a resume finds every checkpoint incomplete on the other nodes. Declared per-node but
visible everywhere means every node's local rank 0 writes the same checkpoint paths concurrently.
The input declaration is checked the same way (verify_input_filesystem_sharing), by every training
script right after init_distributed() and before its first Hub read: global rank 0 writes a sentinel
under HF_HUB_CACHE and HF_DATASETS_CACHE, in one round. Declared shared but invisible means the other
nodes read caches nobody filled (their Hub reads fail, and each of their ranks recomputes every dataset
map); declared per-node but visible everywhere means every node's local rank 0 fills the same cache files
at once. A cache global rank 0 cannot write (a read-only, pre-staged mount) is filled by nobody and is
skipped. The one declaration covers both caches, so both must be on the same kind of storage; an explicit
HF_DATASETS_CACHE does not follow HF_HOME.
The split exists because the two sides can want opposite settings on a multi-node run over a slow or
flaky shared mount (NFS/EFS). Input wants per-node: global rank 0 writing the HF/dataset cache and
remote ranks reading those same inodes is a read-after-cross-node-write that NFS surfaces as
OSError: [Errno 116] Stale file handle.
Output is fine shared — checkpoints have a single writer, per-rank dumps are distinct files, and a
shared mount gives one authoritative copy with no post-hoc gather. That combination is
DIST_INPUT_SHARED_FILESYSTEM=0 with the umbrella left at its shared default.
Coordination primitives
Owning files: halo/runtime/filesystem.py (the shared-FS flags, the writer/reader rank predicates,
main-first ordering, fs_aware_makedirs, the output-FS probe), halo/runtime/consensus.py (the
c10d-store phase, the store and collective rejection joins, barrier_on_exit) and halo/runtime/load_coordination.py
(the per-node load throttle and its world join).
fs_aware_main_first(tag, timeout=None)— context manager that orders the body main-rank-first. Every rank runs the body: the main rank runs it alone first and populates a cache, then the others run the same code and hit it.Scope follows the input flag. Shared: the main rank is global rank 0 and all other ranks wait. Non-shared: it is each node's local rank 0 and only that node's ranks wait, nodes proceeding in parallel. Wrap any download / cache-writing block in it.
Waiters block on a c10d key-value store key rather than in a collective, because the body is unbounded single-rank work (a 100B+
snapshot_download) that would trip the NCCL watchdog (DIST_NCCL_TIMEOUT_MINUTES) on peers held in a collective. The wait is bounded byDIST_STORE_TIMEOUT_HOURSinstead.The bound covers the main rank's work only: there is no trailing join, so the main rank's own wait for the peers is whatever collective comes next, the watchdog, minutes not hours. A misdeclared input flag (shared declared over per-node storage) leaves the peers a cache they cannot see; the input probe refuses that declaration at startup, and the toolkit's model and dataset loads do not rely on this primitive alone either. A model load joins every rank's view of the source and raises on every rank when some cannot see it. The dataset side runs on
run_load_rank_first(below), whose second phase is store-joined too: each peer that cannot see the load rank's cache redoes the download or map itself, under the store bound — the same rows, since the ranks then agree what they loaded, at up to one copy of the work per rank.Two rules the caller owns. The body must issue no collective, directly or through a helper (
ensure_cache_dir()andfs_aware_makedirs()both join across ranks): while the main rank runs it, the peers are on a store key, not in a matching collective, so the main rank would block alone.Every rank must also reach a given
tagthe same number of times, in the same order: keep one tag per call site and never enter one from a rank-dependent branch.hub_metadata_main_first(tag, fetch)—fs_aware_main_firstaround a single hub-metadata read (config.json, a processor config), returningfetch(local_files_only=...)'s result. Tags live underhub_meta/. Off the load rankfetchis toldlocal_files_only=True: a follower reads the cache its scope's main rank just filled, so each scope makes one Hub query rather than one per rank.The bound is 30 minutes rather than the hours-scale store default: a metadata fetch that slow is a dead endpoint, not slow work. It resolves the shared-filesystem consensus itself if that has not happened yet, since these reads can precede
init_distributed()and a split scope would put the tag's participants where they never see each other's keys.throttled_load_within_node(max_concurrent)— store-coordinated throttle that admits at mostmax_concurrentranks per node at a time.joined_node_load(what, max_concurrent)wraps it with an exit joined over the store, and backsmax_concurrent_loadingat model load (Multi-Node → Model loading). SameDIST_STORE_TIMEOUT_HOURSbound.is_shared_filesystem()/is_input_shared_filesystem()/is_output_shared_filesystem()— the resolved modes, for pickingis_global_main_process()vsis_local_main_process()directly.is_fs_load_rank()— whether this rank performs shared read-side work (downloads, cache fill, the coordinated dataset map). Follows the input flag.is_fs_save_rank()— whether this rank writes shared files. Follows the output flag: shared → only global rank 0 writes (avoids the NFS write race); non-shared → each node's local rank 0 writes its own copy.Every gathered/sharded save, the
run.logtee, andfs_aware_makedirsby default route their writer choice through it.fs_aware_makedirs(path, writer_rank=is_fs_save_rank)— creates a directory with the correct rank behindbarrier_on_exit, so a read-only or full volume raises on every rank. Passwriter_rank=is_fs_load_rankfor a read-side cache dir (whathalo/entrypoints/run_setup.pydoes forHF_DATASETS_CACHE).reject_across_ranks(local_reason, what, exc_type=RuntimeError)— the collective that joins a rank-gated body. Every rank calls it with its own reason orNone; if any is non-None, every rank raises the sameexc_type, naming how many ranks failed, which ones, and the first one's reason.Pass
exc_typeto keep a caller's own error contract (a config gate documented asValueError). Use it in place of a barrier wherever one rank does work the others wait on, so a failure there aborts the job instead of parking the peers in the barrier until the watchdog.barrier_on_exit(what)— the context-manager form around a rank-gated write: the body's exception is recorded, then joined byreject_across_rankswhere a barrier would stand (one all-reduce, so collective order matches a barrier's). The failing rank re-raises its own exception, every peer aRuntimeErrornaming the failing ranks. The checkpoint sidecars, adapter writes, index writes and the attention warm-up fence on it.gather_rank_reasons(local_reason)— the same gather without the raise, for a caller whose answer to a failure is not always one: every rank gets the world's reasons, indexed by rank, andgathered_failure_summary(reasons, what)turns them into the messagereject_across_ranksraises. The optimizer-shard resume decides raise vs warm restart from it.run_on_rank0_raising_everywhere(step, describe, exc_type=RuntimeError)— the same join for work one rank does for the world (an external-backend probe, a rollout-server preflight):stepruns on global rank 0 alone and its failure is broadcast, so every rank raisesexc_type(describe(error))together; on rank 0 it chains the original exception.store_reject_across_ranks(tag, local_reason, what, exc_type=RuntimeError, timeout=None)— the same contract carried over the c10d store instead of a collective, for joins whose preceding work is unbounded single-rank time (a fresh-cache dataset map, a first-run shard download).Peers wait on store keys bounded by
DIST_STORE_TIMEOUT_HOURS(or the explicittimeout), not the NCCL watchdog. World-scoped and collective-equivalent: every rank, same tags, same order. Each rank counts itself in on one arrival counter and the last arrival sets the single key the rest wait on; a failing rank first records its reason in a numbered slot, which every rank reads in onemulti_getonce all have arrived. A clean join therefore costs each rank a constant number of store requests, O(world) in total on the one store server; a failed one adds each rank's read of the failing ranks' slots, O(failures × world) in total. The coordinated dataset ops and the sharded-dataset loads join through it, andDeferredRankFailure.store_reject(tag)closes a deferred region with it: the checkpoint savers join that way after the per-rank optimizer and sharded-EP shard writes, a gathered save's last part and index, and each pipeline stage's parts.The tradeoff: a rank that dies between joins parks its peers on the store for the full bound rather than the watchdog minutes. The timeout diagnostic names the tag, how many ranks arrived and the knob, and the launcher's elastic agent usually reaps the group first.
Python
from halo.runtime.filesystem import fs_aware_main_first
with fs_aware_main_first("teacher_model"):
snapshot_download(teacher_repo_id) # one rank fetches; the rest read the populated cacherun_load_rank_first(work, operation_name)(halo/data/pipeline/coordinated_map.py) — the ordering for read-side data work:workruns on theis_fs_load_rank()ranks, every rank joins on the store, then the peers run it against the cache and every rank joins again.Unlike
fs_aware_main_first, a failed first run is never repeated by the peers: the join raises its cause on every rank. A per-node failure takes every node down with the cause instead of leaving the other nodes in their next collective.
The store joins let either side wait out the other's unbounded work (the writer's fresh-cache
map, or a peer's full recompute under a misdeclared input flag) under DIST_STORE_TIMEOUT_HOURS
instead of the NCCL watchdog.
Wrapping a call in a main-first block (local_main_process_first(), fs_aware_main_first) breaks it.
Those hold peers outside the body, so the op's ensure_cache_dir barrier and store joins land at a
different sequence position on the main rank than on its peers.
The barrier then pairs with the wrong collective (or blocks alone against a store-key wait), and the
joins go permanently off-by-one on the equal-entry invariant.
tests/cpu/data/test_coordinated_op_not_main_first.py fails if any call site nests one.
Where it is applied
Storage layout
Fresh-run output-dir validation ignores the toolkit's run logs, filesystem probes, the checkpointing-plan
record and _reference_cache/ scratch, including NFS .nfs* remnants inside it. Checkpoints and unrelated
contents still require a resume, overwrite_output_dir: true, or a different output directory.
Put read-only models and datasets on shared storage; put the HF cache and temp files on local NVMe.
Bash
export DIST_SHARED_FILESYSTEM=1 # 0 for per-node local storage
export HF_HOME=/local/hf_cache
export HF_DATASETS_CACHE=/local/hf_cache/datasets
export TMPDIR=/local/tmpHF_DATASETS_CACHE grows without bound across config sweeps: every distinct
(function, tokenizer, knobs, dataset, library-version) tuple leaves a permanent cache-*.arrow set,
plus packed_*.arrow per packing shape and a transform-* directory per TRL preparation, and
hf cache prune reclaims only the hub cache, not these. Reclaim by deleting the directory — every cache name is deterministic, so the next run
rebuilds exactly what it needs.
To avoid shared-FS bottlenecks, pre-stage models and datasets to local NVMe before training —
snapshot_download(repo_id, local_dir=...) per node (parallel under DIST_SHARED_FILESYSTEM=0), or
rsync. With local_dir set, huggingface_hub writes real files, not symlinks (silencing NFS
symlink warnings otherwise needs HF_HUB_DISABLE_SYMLINKS_WARNING=1). Reclaim local space with
hf cache prune or hf cache rm <repo>.
Multi-node launches: Launch Recipes.
Source on GitHubUpdated 18 November 2026