Data
Datasets load from the HuggingFace Hub, local paths, or S3 (optional — bring your own bucket), with offline pre-processing, packing, and sharding for large-scale training.
- Dataset formats — Required columns and conversation structure for every method — SFT, DPO/SMPO, GRPO, classification, reward, distillation.
- SFT dataset pre-processing — Offline tokenize, pack, and Megatron-style shard — skip tokenization at training time.
- Collators — Padding, packing, and padding-free modes, with completion-only masking and an auto-selecting factory.
- Filesystem handling — Shared vs. local filesystem coordination for multi-node — the read/write
DIST_*_SHARED_FILESYSTEMknobs andfs_aware_main_first(). - S3 utilities — Upload, download, and
s3://URIs for datasets and arbitrary folders.
Source on GitHubUpdated 18 November 2026