Capabilities & Limitations at Scale
What the toolkit can and cannot do for very large models, long sequences, multi-week runs, huge corpora, and many nodes. Multi-node mechanics: Multi-Node Training. Debugging: Debugging & Profiling. What has run on real machines: Verified topologies and Release validation.
Which axis combinations may run at all is an allowlist (Supported combinations); anything else is rejected at config time, before any rank math.
What works at scale
Known limitations
RL weight sync stages one chunk on the forwarding rank's GPU. For Online GRPO and Async GRPO with Environments the NCCL
weight-sync client (halo/weight_sync/) stages each un-sharded policy weight on the sync GPU and streams it to the engine in
byte-bounded chunks (HALO_WEIGHT_SYNC_CHUNK_MB, 1 GiB), never the model and never host RAM.
The forwarding rank's peak is the assembled EP layer being sent plus the chunk, the largest tensor's snapshot and the engine path's buffers. Each is at least the chunk budget, grown past it only when a tensor exceeds it (a fused expert pair on a 100B+ MoE) (Rollout Servers). The ceiling is the fabric behind one GPU (its NICs over EFA) and the single-producer fan-out below.
Checkpoint save is synchronous. Training pauses during each gathered save; there is no async save.
Every gathered save streams (EP layer by layer, FSDP2/CP/TP chunk by chunk through the shared writer),
so all of them are bounded by write throughput rather than by host RAM. The EP save funnels through one
rank per node (HALO_CHECKPOINT_WRITERS_PER_NODE), so a multi-node shared-filesystem save scales with
the node count; the FSDP2, CP and TP saves keep one writer. HALO_CHECKPOINT_WRITE_THREADS (opt-in) overlaps the save
rank's part writes with the next chunk's gather inside that pause (Checkpoints).
The parallel-write escape hatch covers the flagship pure-EP layouts only. save_sharded_ep requires
a single EP group spanning all ranks (ep_group_size == world_size, exactly what
ep16/ep32/ep64 --ep_scope=global run) and a shared output filesystem on multi-node.
reject_unmergeable_ep_sharded_save refuses replicated-EP layouts (multiple EP groups), ETP, CP and expert
LoRA. Dense, FSDP2, CP and TP have no parallel-write mode at all.
merge_ep_shards.py reassembles the artifact, streaming group by group: peak host RAM is one merged
MoE layer plus the output shard being filled and one per write thread, not the model.
--delete_input_shards frees the per-rank shards once the merged checkpoint is complete.
Optimizer state escapes the funnel (per-rank shards), but each rank's state passes through host RAM:
~60 GB/rank at 120B/EP8 with AdamWBF16. A save holds its copy from the cpu_offload to the end of its
shard write, inside the node's max_concurrent_loading throttle: ~240 GB per 8-GPU node at the default
4, ~480 GB at 0. A resume reads under the same throttle, but every rank keeps its copy until the
collective apply, so a node holds all of them at any setting: ~480 GB. Preflight free host RAM for the
resume.
Where save_sharded_ep does not qualify, tune save_steps — at 400B a gathered save is one rank
writing ~800 GB per checkpoint, so budget it against your write bandwidth. save_only_model: true
drops the optimizer shards but not the funnel.
Choosing a large-scale layout
- Dense model: plain
torchrunDP while it fits on one GPU, TP node-local past that (the load limit above). From-scratch viainit_from_scratch. - MoE model: EP (node-local or cross-node / NVL72-wide), optionally + TP for
attention, or + ETP to also shard each expert's FFN (EP+ETP; each ETP group stays in one NVLink domain), or
pure ETP (
ep_size=1) when experts stay replicated. Continued pre-training or fine-tuning. - Beyond one NVLink domain: PP across domains for the PP-registered families, a MoE with EP (or experimental pure ETP) inside each stage; each stage is a whole number of domains.
- Huge corpus: pre-tokenize with
scripts/before_training/prepare_dataset.py --num_shards N(N >= data_parallel_size;k×world_sizeis safe). Loads per-rank with bounded RAM. - NVL72 rack: set
NVLINK_DOMAIN_SIZEto run TP/CP/EP NVLink-wide; see the NVL72 section.
See also: Why This Framework · Pre-training · Checkpoints & Resume
Source on GitHubUpdated 18 November 2026