Architecture
Halo extends HuggingFace — Transformers, Accelerate, TRL — with Expert, Context, Tensor,
Expert-Tensor, and Pipeline parallelism plus alignment methods TRL does not ship (SMPO, Offline GRPO) and Async
GRPO with Environments. Every trainer subclasses a TRL, Transformers, or SentenceTransformers trainer
and adds DistributedTrainerMixin.
The default save is a standard HuggingFace checkpoint and there is no Megatron conversion step; the
one opt-in per-rank format (save_sharded_ep) needs a merge script before reload.
Everything runs inside the prebuilt image (halo:blackwell / halo:hopper); the
host has no usable Python (Docker).
Component map
Leaf modules keep those imports one-way, each holding a contract several layers share:
halo/runtime/ therefore holds only what needs a process group or the launcher: rank and node math
(ranks.py), init_distributed and the subgroup timeout pin (process_group.py), barriers,
broadcasts, fixed-order group creation and the rank-0 object gather (collectives.py; a ring reduce, since a
per-peer send connects every peer to rank 0 and NCCL keeps those buffers, about 0.3 GiB per NVLink peer within
one NVLink domain, for the run, 2.94 GiB on rank 0 of 8 GPUs against 0.75 GiB over the ring), the cross-rank rejection/consensus seams
and the c10d-store joins (consensus.py), the shared-filesystem scope, FS-aware writer ranks,
main-first ordering and the output-FS probe (filesystem.py), the per-node load throttle
(load_coordination.py) and DTensor access (dtensor.py); anything a single rank can compute alone
lives in the leaves above. It imports only halo.env, halo.log and halo.hardware, and the
parallelism strategies in halo/distributed/ import it, never the reverse.
Layer table
A unit imports only units of strictly lower rows; tests/cpu/conventions/test_import_layering.py
enforces it and asserts this block equals its _LAYERS.
Text
L0 env
L1 log
L2 hardware, config_validation, subclass_registry, cli
L3 runtime, kernels, inference, checkpoint.model_free
L4 models, diagnostics, rewards
L5 model_loading, optimizers, weight_sync
L6 checkpoint.model_aware, data
L7 distributed, environments
L8 model_loading.parallel, checkpoint.parallel, callbacks, configs
L9 trainers
L10 entrypointscheckpoint.model_free is the halo/checkpoint/ modules that read no model (format, atomic,
reading, model_card); the rest of halo/checkpoint/ outside parallel/, tools/ included, is
checkpoint.model_aware.
Trainers
Every distributed trainer uses multiple inheritance: a base trainer (trl.SFTTrainer,
transformers.Trainer, trl.GRPOTrainer, …) plus DistributedTrainerMixin
(halo/trainers/distributed_trainer.py), which composes the concern mixins listed in
Trainer Architecture. The mixin overrides the parallelism-sensitive
methods (accelerator creation, dataloader sharding, gradient clipping, model saving) and delegates
the rest.
Thirteen trainers share this shape; which parallelism axes each supports, and why, is in Trainer Architecture.
Offline GRPO's frozen-reference scores span three modules: halo/trainers/grpo/offline/reference_cache.py
(bounded, filesystem-aware score storage and the memory-mapped token buffers),
halo/trainers/grpo/offline/reference.py (the run-start sweep, the ragged payload, the training-checkpoint
contract and evaluation's original-reference scores), and the shared halo/trainers/reference_logps.py (checkpoint identity and
persistence, which DPO/KTO precompute also use). Storage, resume and recovery:
Offline GRPO → Reference model.
Distributed layer
ParallelismConfig (halo/distributed/parallelism_config.py) is the single source of truth: it
validates the requested combination against the allowlist and refusals in
halo/distributed/parallelism_rules.py, computes data_parallel_size, and builds the EPConfig /
CPConfig process groups, whose membership comes from the layouts in halo/distributed/group_layout.py; the DeviceMeshes come from
halo/distributed/mesh.py. The trainer reads mode flags (has_ep_group, is_cp_mode, …) off it and
never re-derives them.
What each mode shards and its backend: Parallelism → Parallelism modes; its data-parallel size: Supported combinations.
Four subpackages own the mechanics: expert_parallel/ (DeepEP dispatch/combine, per-family EP
layers, grouped-GEMM expert compute, gradient-sync hooks, expert gather), context_parallel/
(UlyssesModel, sequence splitting), tensor_parallel/ (DTensor weight sharding), and
pipeline_parallel/ (layer split, stage module, stage-aware loading, P2P groups, the
torch.distributed.pipelining seam). Inside expert_parallel/, moe_layer/ composes EPMoELayerBase from
one mixin per concern; its compute, routing, grad_sync and balancing_bias modules are imported only
from inside expert_parallel/, and everything else reaches them through layer instances
(named_ep_layers).
halo/model_loading/parallel/ sits above all four: load_distributed_model picks the per-mode loader
off a ParallelismConfig, so it is the one place that reaches into every implementation, and the rest
of halo/model_loading/ stays sharding-agnostic. The collective checkpoint half, halo/checkpoint/parallel/
(the saver ladder in save.py, the EP saves in ep.py, the PP saves in pp.py, the adapter saves in
adapters.py, and the resume beside them), sits above them too: it gathers through each implementation.
These packages import each other acyclically
(tests/cpu/conventions/test_import_layering.py).
The lazy load sits whole in the dispatch package, since only its loaders use it: lazy_plan.py holds the
expert-key vocabulary, the lazy gate, the per-key weight planner and the prologue the EP and PP loaders share
(a dense PP load runs it too); expert_fuser.py stacks a per-expert checkpoint's local experts; beside them
sit the loaders, ep.py and pp.py, the weight load both run once their plans are final
(lazy_load.py), and fp32_masters.py, which selects the configured FP32 parameter masters off the EP
layer registry and streams their stored values back for eager construction
(Load precision).
The machinery both lazy loaders share — the conversion-op vocabulary and the key plan translated into it
(conversion.py), which transformers conversions a load plans with (conversion_selection.py), per-key
weight plans and the shared stream tail (materialization.py), meta-shell instantiation (meta_shell.py) —
lives in halo/model_loading/lazy_safetensors/, with no parallelism knowledge.
The FSDP2, HSDP, and TP axes ride a torch DeviceMesh (halo/distributed/mesh.py); EP and CP use
hand-built groups (build_groups) whose all-to-all patterns do not map to a mesh. The trainer reads
the mesh groups (DP, TP) through the mesh accessors (dp_group, tp_group) and the expert
groups (dispatch, expert-TP, expert-replica) off EPConfig. The bucketed
gradient all-reduce and the grad-presence agreement every post-backward sweep shares — deferred EP
cross-replica, TP replicated, QLoRA — live in a leaf (halo/distributed/grad_reduce.py) that
imports torch, halo.env and halo.runtime, and no parallelism implementation.
Which axis combinations may run is an allowlist, not a denylist — see Parallelism.
A training step
Text
YAML config
│ HaloArgumentParser: toolkit defaults (use_liger_kernel, bf16, logging_nan_inf_filter, non_blocking)
▼
ParallelismConfig ── validate mode, build process groups
│
▼
load_distributed_model halo/model_loading/parallel/loader.py
│ • resolve_attn_implementation → FA4 (SM100+) / FA3 (SM90) / FA2; per-model SDPA/eager overrides
│ • apply_ep_layers → EP layers (when ep_group_size > 1 — EP or ETP — or grouped GEMM on an MoE)
│ • UlyssesModel (when cp_size > 1)
│ • load_pp_stage_model: this rank's stage only (when pp_size > 1)
│ • attach_mtp_head: the release's head and its forward hooks (when mtp_loss_weight > 0)
▼
DistributedTrainerMixin._configure_distributed_modes
│ • FSDP2 fully_shard (EP modules in ignored_params, except at ep_group_size == 1)
│ • router grad-sync hooks already attached at EP-layer construction (load);
│ PEFT modules_to_save router copies re-hooked here
▼
trainer.train() ── HF training loop, DDP wrapping skipped
│ • forward → loss → backward
│ • EP-aware global grad-norm clip (DTensor + replica groups)
│ • AdamWBF16 step (bf16 weights, stochastic rounding — no fp32 master)
│ • callbacks: EfficiencyCallback (tokens/s/GPU), MoEMetricsCallback,
│ BalancingBiasCallback
▼
trainer.save_model ── save_checkpoint (halo/checkpoint/parallel/) gathers shards → HF checkpointThe backend choice and its per-model redirects are on Flash Attention; the EP layers and their FSDP2 treatment on Expert Parallelism; the gathered save on Checkpoints & Resume.
RL: the rollout engine as a separate container
Online and async GRPO generate on a rollout engine (vLLM 0.32.0, or SGLang 0.5.22 under async GRPO's
rollout_backend: sglang) that runs in its own container and is never imported into the training
environment, since it pins its own torch/transformers stack. The trainer talks to it over HTTP for
generation and through an NCCL weight-sync client (halo/weight_sync/) for weights, which it gathers
from every parallel layout before each generation round that follows an update. Online GRPO hands TRL
TrlVLLMClient (halo/trainers/grpo/rollout/) in place of TRL's VLLMClient, which would import the vLLM
package. Images, patches, the
families each engine serves and the sync itself: Rollout Servers;
the trainers: Online GRPO, Async GRPO.
Related pages
- Trainer Architecture — the mixin lifecycle, gradient sync, per-trainer notes
- Why This Framework — comparison to TRL, Unsloth, Megatron-LM, veRL, MS-SWIFT
- Parallelism — EP, CP, TP, ETP, PP and valid combinations
- Configuration — the YAML system and toolkit defaults
- Checkpoints & Resume — checkpoint formats and the two resume paths
Source on GitHubUpdated 18 November 2026