S3 Utilities
Reading and writing HuggingFace Dataset/DatasetDict objects and arbitrary folders in S3, with a
local cache for distributed training and a CLI. The default bucket comes from
HALO_S3_DEFAULT_BUCKET — set it to your own bucket. Four modules plus the CLI and its commands, split so a consumer
takes on only the dependency it needs:
Dataset path formats
parse_dataset_source (halo/data/sources/paths.py — string rules, no boto3, so the
loader and the scripts classify a path without an S3 stack) sorts a path into
('s3' | 'hf_hub' | 'local', bucket, key):
- S3 URI (recommended):
"s3://my-bucket/datasets/my_dataset/train". - HuggingFace Hub:
"HuggingFaceH4/ultrachat_200k"— exactly one slash, no part starting with a dot. Optional suffixes::configpicks a dataset config ("openai/gsm8k:main"),@splitpicks one split when the repo has no plaintrain("HuggingFaceH4/ultrachat_200k@train_sft"). The key is returned verbatim including the suffixes; they are stripped later, at load. - Local: everything else — an absolute,
./or../path, afile://URI (prefix stripped), any path ending in.jsonl/.json/.parquet/.arrow/.csv, and a relative path with no slash or more than one.
A Hub-shaped path that also names a local directory (preprocessed/my_dataset beside a directory of
that name) is refused by every reader, training, prepare_dataset.py --input, the eval runner and the
deduplication tool alike, rather than guessed: write ./preprocessed/my_dataset or an absolute path for the directory, without a
:config / @split suffix (a saved split is its own subdirectory).
reject_ambiguous_hub_source is the check; on training it runs inside the rank-joined load and probe, so
a directory present on some nodes only fails every rank with the same cause.
parse_dataset_destination is its write-side twin and classifies a bare org/name as local:
guessing the Hub is acceptable for a read, but a write only targets the Hub on an explicit
hf://org/name. Combining sources and required schemas: Dataset Formats.
Caching
S3 datasets are cached locally to avoid re-downloading. Cache directory:
$HALO_DATA_ROOT/s3_datasets (default ~/.cache/halo/s3_datasets); point HALO_DATA_ROOT at a
large mount, or override this cache alone with HALO_S3_DATASET_CACHE_DIR:
Bash
HALO_S3_DATASET_CACHE_DIR="$LARGE_VOLUME"/s3_datasets torchrun --nproc_per_node=8 \
scripts/training/sft.py examples/sft/qwen3/qwen3-4b-ultrachat.yamlHow it works:
Cache key: MD5 of
bucket/key→ one subdirectory per dataset.File locking:
filelock.FileLockprevents duplicate downloads across processes.Completion marker: a
.download_completefile marks a valid cache; it records the source URI and an ETag content fingerprint of the S3 prefix. Downloads stage into a unique temp dir and publish with an atomic rename, so a crashed writer never leaves a half-valid cache. A stale tree that cannot be renamed aside (a permission error, say) raises and discards the fresh download, rather than serving the stale rows.One copy on disk: the tree downloads straight into that temp dir through the client's parallel transfer (
download_dataset_tree), so no second copy waits underTMPDIRfor the process to exit. A tree missing its load gates (a torn promote) raises there and its partial download is removed, so it is never published.Staleness check: on a cache hit the marker's fingerprint is re-validated against live S3 — an in-place re-push to the same URI changes the ETags, and a mismatch triggers a re-download. A marker carrying no fingerprint is trusted and upgraded in place.
Offline-resilient hit: a complete cache skips the existence probe and the download, and the staleness probe fails soft.
An unreachable S3 (no credentials, expired SSO, outage) reads as "cannot validate" and the cache is served. A cold cache with S3 unreachable raises instead, naming the cache path and the way out.
Control-file mirror:
metadata.jsonandshard_index.jsonreads (read_control_json_with_cache) write a local mirror under$HALO_S3_DATASET_CACHE_DIR/control/on every live read and serve it when S3 is unreachable.Preprocessed/sharded classification therefore survives an outage too, not just the dataset bytes. Live wins; a live 404 drops the mirror, so a re-pushed raw dataset cannot keep its old classification offline.
A training script reads them once per launch: global rank 0 reads the stamp and both shard indexes (
share_s3_control_files), every rank's later probes and loads use that read, and each node's local rank 0 writes the mirror. A file rank 0 cannot read is left to every rank's own read.
The per-shard cache used by sharded pre-processed datasets (halo/data/sources/sharded_dataset.py) uses the same marker + fingerprint scheme. The first probe that finds S3 unreachable logs a warning and skips the remaining shard probes, so a fully cached offline run pays a few control-read timeouts (the sharded probe and per-split index each try live first), not one per shard.
An empty prefix listing is a real absence, not a transport fault, and does not disarm the probes.
Each rank's shard loads join across the world on the c10d store (DIST_STORE_TIMEOUT_HOURS), so one rank's failed or slow first-run download surfaces its real cause on every rank instead of an NCCL watchdog dump.
Force a re-download with load_dataset_from_s3_uri(..., use_cache=False). Clear it by removing the resolved cache directory ($HALO_DATA_ROOT/s3_datasets unless overridden above).
On multi-node with a shared filesystem (NFS/Lustre) the cache is shared — only one node downloads. With per-node local storage each node caches independently; set DIST_SHARED_FILESYSTEM=0 (see Filesystem Handling). Since a node can serve its own stale cache while S3 is unreachable from it, the load agrees the rows across ranks and fails the launch on a disagreement (Distributed Data Loading).
Authentication
S3Client uses boto3's default credential chain (env vars, ~/.aws/credentials, SSO) — bring your
own account and bucket; setup and credential failures are in AWS Auth. For an S3-compatible
service or non-default bucket, pass aws_access_key_id / aws_secret_access_key / endpoint_url /
region_name directly to S3Client(...).
Retries follow the AWS configuration: AWS_MAX_ATTEMPTS / AWS_RETRY_MODE, or the profile's
max_attempts / retry_mode. A key neither sets takes the toolkit default, 4 total attempts in standard
mode; raise both for a many-node fetch that meets S3 throttling (SlowDown).
Python API
Dataset trees go through halo.data.sources.s3_dataset_store; everything else is halo.data.sources.s3_client
(see the modules for full signatures and defaults):
subfolder is prepended to a bare key; key paths may be nested. Everything else — folder transfers, listings,
deletes — is an S3Client method, called on a client built for the bucket in question.
S3Client
For a non-default bucket or an S3-compatible service:
Python
from halo.data.sources.s3_client import S3Client
from halo.data.sources.s3_dataset_store import push_dataset
client = S3Client(bucket="<bucket>")
push_dataset(client, dataset, "data")
# MinIO / S3-compatible
client = S3Client(
bucket="<bucket>",
endpoint_url="http://localhost:9000",
aws_access_key_id="minioadmin",
aws_secret_access_key="minioadmin",
)S3Client carries the transport surface: exists / object_exists, list_objects, delete, and the
folder transfers:
Both transfer files concurrently (a many-shard dataset or checkpoint moves in parallel, not
file-by-file); HALO_S3_MAX_FOLDER_CONCURRENCY sets the files-in-flight bound (default 16).
Training scripts call load_datasets_auto(), which probes for a pre-processed dataset first, then
delegates to load_datasets(): detect the source, download to cache if needed, file-lock against
duplicate downloads, memory-map the Arrow files.
CLI
Bash
python scripts/before_training/s3.py push ./data project/experiment/data
python scripts/before_training/s3.py download project/experiment/data ./data
python scripts/before_training/s3.py list my_project/ --recursive
python scripts/before_training/s3.py delete my_folder/data.json --yes
python scripts/before_training/s3.py delete my_folder --recursive --yes
python scripts/before_training/s3.py download s3://my-bucket/project/experiment/data ./dataCommands: push, download, list, exists, delete. Every command takes --subfolder/-s,
--bucket/-b (default HALO_S3_DEFAULT_BUCKET; a bare key needs one of the two) and --verbose/-v. A
key (or list's prefix) given as a full s3://bucket/key URI names its own bucket, so it refuses
--subfolder and a different --bucket. The other flags are per command:
CLI delete removes ONE object; a non-recursive delete aimed at a prefix is refused rather than
reported as a no-op success. That refusal, a --recursive delete aimed at a single object, and a
failed delete all exit non-zero; a key with nothing behind it is reported and exits 0.
Source on GitHubUpdated 18 November 2026