Trainer Architecture
All distributed trainers inherit DistributedTrainerMixin (halo/trainers/distributed_trainer.py), which
sits alongside a base trainer class (SFTTrainer, Trainer, GRPOTrainer) and overrides the
parallelism-sensitive methods while delegating the rest. The spine itself holds construction
(_init_distributed_config), the post-wrap orchestration (_configure_distributed_modes), the HF
overrides (training_step, prediction_step, log, create_optimizer, get_total_train_batch_size,
_move_model_to_device) and the reference-model hooks. Everything else sits in a module named for
its concern:
Each mixin is a class because it reads live trainer state; the methods a test or a caller reaches as
DistributedTrainerMixin.<name> resolve through the MRO unchanged. The HF overrides' zero-arg
super() calls (the checkpoint methods, evaluate, create_accelerator_and_postprocess,
_inner_training_loop) must reach the base Trainer — tests/cpu/trainers/test_mixin_composition.py
fails if a sibling base ever intercepts one.
Three mixins sit outside that composition, in halo/trainers/. StoredMetricsMixin
(metrics/stored_metrics.py) is mixed in directly by SMPO, teacher and self
distillation, and SDPG for buffered per-step metric logging. log reports
each metric's row-weighted world mean through one WorldMetrics flush per log call, and ranks that stored
different keys raise on every rank. store_metrics(..., rows=n) weights
a value by the real rows it averages. Every eval store passes rows=eval_split_rows(n) (pairs for
SMPO) and computes its values over those rows — the distillation and SDPG trainers through
store_batch_metrics, which weighs a train micro-batch 1; the eval losses read
the same rows, bar SDPG's GRPO term, which TRL computes over the whole batch.
TRL's DPO, KTO and reward trainers log through their own _metrics and accelerate's gather instead,
so their eval metrics keep the final round's padding. Their loss call is TRL's, and so are its metric reductions:
every micro-batch gathers each metric across the world and reads it back to the host (about 15 gathers for DPO,
14 for KTO, 4 for reward; TRL's GRPO loss, which online and async GRPO and SDPG run, about 8 reductions). The
cost is latency that grows with the rank count; SFT and the trainers above fold theirs once per log.
Offline GRPO, the embedding trainer, and online and async GRPO keep their own log instead:
offline reads the train/eval bucket off model.training rather than the mixin's "loss" in logs,
the embedding trainer's eval metrics go into output.metrics for best-model tracking, and the two
on-policy trainers decouple the completions table from the metric drain. The embedding trainer's
eval_embed/* values are rank 0's per-batch means and count a padded final round's repeated rows
(Embedding).
WorldMetrics (metrics/world_metrics.py) is the one cross-rank metric accumulator: StoredMetricsMixin, SFT's loss
metrics, offline GRPO's sign metrics and the GRPO step and rollout metrics record rank-local sums into it. A
flush folds each column where it was recorded (a tensor's on its device, with the arithmetic Python's sum,
max and min apply to a list), then reduces over the (key, kind) layout the ranks agreed on last: one
float64 SUM all-reduce carries each rank's fit verdict, the keys it holds and the summed columns, one MAX
all-reduce the maxes and negated mins, and one host read brings both back. Their count, width and host work do
not grow with the rank count. A key outside the layout (the first flush, or a key behind a data-dependent gate the
first time a rank reaches it), ranks holding different keys under a key check, or a percentile's raw values take
one object gather instead, whose host work does grow with it on NCCL, and the next flush reuses the union; the
async GRPO rollout metrics pay that gather every round for episode/generation_tokens_p90. world_sums is the
one-shot form for a value read outside a window (the trust-region breaker, the KL clamp fraction, the prefetch and
routing-replay counters, the token counters).
ModeSetupMixin dispatches by method, not by mode string: _apply_ep_fsdp2,
_apply_cp_fsdp2 and _apply_ep_tp_fsdp2 are called directly from the mode ladder
(_wrap_for_parallel_mode), and the EP one derives the FSDP ignored-module set in a single
module-tree walk it hands to _apply_ep_aware_dp_fsdp2.
TRLDatasetPrepMixin (trl_dataset_prep.py) is mixed in directly by DPO, KTO and the
reward trainer, ahead of TRL: it runs TRL's dataset preparation once per filesystem scope
(Filesystem handling).
ReferenceLogpsMixin (reference_logps.py) is mixed in by DPO/KTO precompute and offline GRPO. It
owns reference identity checks, resume attachment and atomic sidecar persistence; each trainer owns
its score payload and sweep (Checkpoints). Both record the log-prob
precision each split was scored in (LOGPROB_PRECISION_KEY;
DPO,
offline GRPO).
DPO and KTO share one TRLPreferenceMixin (preference/trl_preference.py), first in their bases: it
runs their construction (TRL's Liger path off, reference resume, distributed config, reference gates,
TRL's construction-time reference hash skipped), rebinds TRL's log-prob functions to fp32 ones, and runs
the reference sweep on the DP axis. Ahead of DistributedTrainerMixin its sidecar hook wins over the
checkpointing default; ahead of the TRL base its rebinds wrap TRL's loss and reference pass. A subclass
declares REFERENCE_COLUMNS and its Liger reason; CTOR_POSITIONS is derived from the TRL base's
signature per subclass. SMPO, DPO and KTO construction refusals are pure functions in
preference/preference_rules.py.
Offline GRPO's reference_cache.py streams current-batch scores to filesystem-aware writers and
maps the ordered ragged token values; its reference mixin attaches those buffers without repacking
them (Reference model).
metrics/token_metrics.py::effective_loss_mask (completion_mask ∧ tool_mask where a tool_mask exists, else
completion_mask) is imported as a plain function.
Trainer compatibility
No trainer refuses EP, TP or ETP. GPU rows run all three under SFT, DPO, offline, online and async GRPO,
SDPG, reward and classification; SMPO, KTO, teacher distillation and embedding are verified under EP and TP,
and self-distillation under EP only. A No PP trainer is refused at config time
(<Trainer> does not support Pipeline Parallelism (PP)); see
Pipeline Parallelism.
DistributedSDPGTrainer overrides the online trainer's PACKED_ROWS_REFUSAL and declares its teacher prompt as a second entry of ROUND_PROMPT_ROWS (Online SDPG).
A trainer declares its capabilities as public class constants on DistributedTrainerMixin.
SUPPORTS_CP and PP_REFUSAL are read by one classmethod, reject_undeclared_axes(cp_size=, pp_size=), which parallelism_config_from_args runs before any model loads and
_init_distributed_config runs again at construction:
The refusal texts are <Trainer> does not support Context Parallelism (CP). Use Expert Parallelism (EP) and/or Tensor Parallelism (TP) instead, or remove context_parallel_size. and <Trainer> does not support Pipeline Parallelism (PP): <PP_REFUSAL>. Use ... or remove pipeline_parallel_size.
Every PP-refusing trainer declares its own PP_REFUSAL text.
CP works only where the loss is computable from a sequence chunk. The rest keep the default
SUPPORTS_CP = False.
The reasons: the trainer uses logits_to_keep (Async GRPO with Environments), needs global
log-probability sums (DPO, KTO), needs full-sequence pooling (classification, reward, embedding),
wraps two models (distillation), or runs a separate-length privileged-teacher or rollout sequence
(self distillation, SDPG, online GRPO).
Offline GRPO scores boundary-aligned local hidden states through the chunked head and reduces its token objective across CP with the shared autograd SUM. Its CP path rejects adapters; its full-FT reference is persisted, not recomputed from trained weights on resume.
PP needs a single-forward objective on one stage's logits, and the conditional rows above are constructor-time gates rather than class attributes. Rejections land in three places.
PipelineTrainerMixin._split_pipeline_stage rejects the generic blockers on every trainer
(halo/trainers/parallelism/pipeline_rules.py):
an unset max_length, save_sharded_ep, reentrant gradient checkpointing, peft_config or a model
already PEFT-wrapped (a PeftModel, or one carrying PEFT tuner layers), TRL's activation_offloading,
torch_compile, image-bearing data, and an eval batch size differing from train. Padding-free
collation is rejected at the collator wrap (its flattened width varies every step); packing passes
at one packed row per microbatch, except on a model whose sequence mixers read no document boundary
(PP packing).
pipeline_rules.py also holds the shared vocabulary the preference and offline-GRPO
wrappers add on top, through the _reject_unsupported_pp_options hook or their own gate — a live ref_model, missing precomputed reference columns, and
compute_metrics (no PP reduction reproduces their non-PP prediction convention). A tied-embedding
checkpoint, live MTP tail layers, a family that is refused or not registered in PP_FAMILY_SPECS,
and aux-loss MoE balancing are rejected later, at the model split
(halo/distributed/pipeline_parallel/model_rules.py). See
Pipeline Parallelism.
Modality support
Modality routing differs by method. SFT, SMPO, DPO, reward modeling and the distillation scripts key
on the run (is_vlm_run_across_ranks): the VLM data path needs a multimodal checkpoint and declared image
data, so text-only rows on a natively-multimodal model (Qwen3.5/3.6, Gemma 4, Inkling) train through
the text pipeline with packing available (declaration rules).
KTO routes its data path off the dataset (TRL's own column probe), as
scripts/training/preference/kto.py states. The model class always follows the checkpoint. SFT,
DPO, KTO, SMPO and distillation load the processing class by the rule in
SFT — VLMs; reward modeling loads the processor
for an image run only.
The data contract is the same across methods: conversations carry typed content parts
({"type": "text" | "image", ...}), a per-modality top-level column is injected into the
conversation at map time, placeholder expansion belongs to the model's processor, and media
extraction is driven by part type. A new modality is an additive part type plus a column field.
Lifecycle
Python
# Slots of the base the *args are forwarded to, read off its installed signature.
_CTOR_POSITIONS = resolve_ctor_positions(SomeBaseTrainer, "model", "args")
class MyDistributedTrainer(DistributedTrainerMixin, SomeBaseTrainer):
SUPPORTS_CP = True # declare only what differs from the mixin's defaults
PP_REFUSAL = "the loss couples the whole batch, which no pipeline microbatch sees"
def __init__(self, *args, **kwargs):
# Extract the parallelism kwargs; a positional model/config is read through the table.
kwargs = self._init_distributed_config(kwargs, ctor_args=args, ctor_positions=_CTOR_POSITIONS)
super().__init__(*args, **kwargs) # base init: accelerator, model, optimizer
self._configure_distributed_modes() # apply parallelism after the model exists_init_distributed_config(kwargs, training_args=None, ctor_args=(), ctor_positions=None, **explicit) pops the toolkit-only kwargs
before the base trainer sees them: parallelism_config (a ParallelismConfig; passing None
raises ValueError), the checkpoint knobs save_sharded_ep (default False), save_max_shard_size
and allow_optimizer_warm_restart, max_consecutive_nonfinite_steps (the step guard's
limit), moe_balancing, dataset_presharded, and bf16_optimizer.
Its _apply_training_args_overrides step sets fsdp = "" on the training args (the mixin owns sharding),
hands TRL its router aux-loss coefficient, and
reconciles save_on_each_node, the Liger config and the GC use_reentrant kwarg with the requested
mode and the model (reentrant rule). It then runs the construction refusals
(reject_undeclared_axes, reject_pp_named_optimizer, reject_custom_parallelism_under_accelerate, the determinism
gates) and the PP split.
distributed_trainer_kwargs (halo/entrypoints/script.py) is the bundle every entry script
passes.
A trainer that forwards **kwargs calls it as above. One whose __init__ names those parameters
passes them through **explicit instead (SMPO, Classification, offline GRPO, teacher distillation);
kwargs-style values win over explicit ones. The model and training_args default to the ctor's
model / args, read from kwargs or, for positionals, from their slots in ctor_positions
(positionals without the table raise). Online and async GRPO pass training_args= as well, having
resolved it first, and EmbeddingTrainer passes a synthetic {"model": model, "args": args} (a
SentenceTransformer's config is its backbone's).
ParallelismConfig validates the combination and exposes mode-flag properties (has_ep_group,
is_cp_mode, is_tp_mode, is_expert_tp_mode, is_pp_mode); has_ep_group is ep_group_size > 1,
covering both EP and pure ETP.
create_accelerator_and_postprocess() (AcceleratorMixin) is overridden during base init, on the
custom path only (_needs_custom_accelerator(); otherwise it delegates to the base):
build_plain_accelerator applies no DDP wrapping (manual gradient sync), sets
gradient_accumulation_steps=1 (the Trainer drives accumulation) and bf16/fp16 autocast.
On either path the trainer clears accelerator.native_amp, dropping the fp32 logits upcast that can
cost many GB at long sequence. Under fp16 it stays: native_amp also gates GradScaler unscaling
there, so clearing it would clip and step on scaled gradients.
_configure_distributed_modes() takes back TRL's construction-time writes
(_restore_trl_overrides), runs the pre-wrap gates (adapters under TP, inert router logits,
load_best_model_at_end), aligns the PEFT and fp32-master dtypes, wraps the model, then runs the
post-wrap gates (reject_unsynced_trainable_params, the expert-LoRA rules) and installs
(_configure_wrapped_model). The wrap, ModeSetupMixin._wrap_for_parallel_mode(), dispatches on the
mode flags in this order:
Text
is_pp_mode -> _configure_pipeline_parallel() # stage-scoped (EP-aware) FSDP, runner, clip
has_ep_group + TP -> EP clip, loader mesh, EP+TP FSDP2
has_ep_group + CP -> adopt the CP wrapper, EP clip, EP FSDP2
is_cp_mode -> adopt the CP wrapper, CP FSDP2
is_tp_mode -> _apply_tp_fsdp2() + TP clip # FSDP2 on the DP axis (TP applied at load)
has_ep_group, or EP layers -> EP clip, EP FSDP2 # EP, pure ETP, EP+ETP, grouped-GEMM ep1
accelerate-managed -> accelerate FSDP / DDP # refuses EP layers
else -> plain DP FSDP2 (QLoRA: the DP sweep)TP/CP precede the EP branch because use_grouped_gemm defaults to True even for dense models —
without this ordering a TP-only dense model would enter the EP path with the wrong DP size. The EP
branch takes a model whose EP layers the config does not declare only off an accelerate launch, so a
dense accelerate launch is never hijacked onto the mixin's FSDP2, and an MoE whose experts are
already wrapped under accelerate launch raises.
The EP branch patches gradient clipping and installs the EP gradient sync (which applies FSDP2 with
EP modules in ignored_params, except at ep_group_size == 1 — below).
CP setup requires the model to be a UlyssesModel already (wrapped at load time,
require_ulysses_model). FSDP2 (fully_shard,
reshard_after_forward from fsdp_reshard_after_forward, default false) carries non-EP gradient
sync wherever DP > 1 — pure TP and EP+TP at DP=1 skip the wrap entirely. The wrap leaves out
frozen parameters of a dtype no trainable parameter shares, and raises if it would leave out a
trainable one nothing else syncs (Data Parallelism).
QLoRA skips FSDP2 on both the plain-DP and the CP path (fully_shard cannot wrap bnb's non-float
Params4bit). _install_qlora_grad_sync sets a flag rather than per-parameter hooks, whose
rank-local firing would hang a job whose microbatch touches different adapters per rank, and gives
every floating-point parameter rank 0's copy: the sync averages gradients only, so adapters the ranks
drew from their own RNG streams would otherwise stay apart for the whole run.
The sync is one bucketed all-reduce over every trainable grad per optimizer step, with membership agreed by a grad-presence mask so no rank reduces alone. It is ordered ahead of clipping (otherwise each rank clips by its own coefficient) and backstopped by a step-pre-hook for a step that misses the clip.
Parallelism modes
Which combinations may run, and the domain-locality rules each one carries, are the
ParallelismConfig allowlist's business — see
Supported combinations. One consequence lands in
this table: multi-domain multi-group EP+TP is rejected because the deferred cross-replica average
assumes FSDP shards over the EP group, while EP+TP shards over the (dp, tp) mesh.
Gradient synchronization
The mixin bypasses HuggingFace's DDP and manages sync directly, per the table above. The one
exception to the expert/non-expert split is experts_fsdp_managed — fsdp_shard_ep1_experts
(default true) at ep_group_size == 1, the default shape of every ep_size: 1 MoE run: the
ignored_params list is emptied, FSDP2 shards the experts, its reduce-scatter becomes their sole
sync, and the EP layer skips its own hooks and the sweep its expert leg.
In CP-only mode each rank computes partial gradients from its chunk and FSDP averaging yields the
correct global mean (effective_grad = (1/cp_size) * sum(partial_grad_i)). In EP+TP with DP=1,
DTensor and the EP sync alone; with DP>1, FSDP2 syncs non-EP params across nodes.
Clipping and sync read the mesh groups (TP, DP) through the mesh accessors (tp_group, dp_group
in halo/distributed/mesh.py) and the expert groups (dispatch-EP, expert-TP, expert-replica) off
EPConfig, rather than re-deriving them per call site.
EP-aware gradient clipping
A local norm is wrong when expert params are sharded, so the mixin patches
accelerator.clip_grad_norm_ to reconstruct the global norm from separate expert and non-expert
norms. The EP and TP clips share one norm, _compute_global_grad_norm:
- DTensor params (TP, EP+TP): take the local shard via
._local_tensorand all-reduce shard norms over the group that tiles them into the non-expert norm. - ETP: all-reduce expert shard norms within
expert_tp_groupfirst. - All EP topologies: all-reduce expert norms within the dispatch (sub-EP) group.
- Multiple EP groups: no replica reduce. Every multi-group topology defers, and the clip runs the deferred sweep before the norm, so the replicas already hold identical expert grads.
Then global_norm = sqrt(expert_norm_sq + other_norm_sq) sets the clip. Every clip derives one
device-resident coefficient,
clip_coefficient (halo/trainers/parallelism/grad_clipping.py), and refuses any norm_type but 2 (require_l2).
The EP clip refuses to install on a model with no EP layers.
The TP clip, and the EP clip under any other optimizer, scale every local gradient by it
in place (scale_shards_to_max_norm_). Under EP with AdamWBF16 stepping exactly the clipped parameters
that hold gradients, the EP clip hands the coefficient to the optimizer instead (defer_grad_scale),
which multiplies it into each gradient inside its fused step; the gradients stay unscaled until then.
Optimizer construction
Each optimizer module under halo/optimizers/ owns its own builder, and every builder is pure: it
takes the model, args and decay parameters and returns an optimizer without touching trainer state.
create_optimizer picks one and owns the self.optimizer assignment:
build_optimizer (halo/trainers/construction/optimizer_selection.py) returns None for that last row, and the
stock build is gated, not a fallthrough: reject_stock_adamw_on_mixed_params raises on a
stock fused/foreach AdamW whenever ep_group_size > 1, or the EP layers hold the experts and
FSDP2 does not (experts_fsdp_managed false). aten._fused_adamw_ raises "mixed torch.Tensor and
DTensor" over that parameter set at the first step; the message names AdamWBF16 and
fp32_non_ep_params as the two supported shapes.
Each grad sweep runs inside the patched clip_grad_norm_. At max_grad_norm <= 0 transformers skips
the clip but still calls it on every step for an unclipped norm, at an infinite threshold that
is_clipping_enabled treats as no clipping, so no gradient is rescaled; the non-finite step guard reads
that norm. create_optimizer also registers three step-pre-hooks (_register_grad_sync_step_hooks) as
backstops for a step that misses the call: the TP replicated-grad sync, the multi-group-EP cross-replica
sweep, and the QLoRA DP sweep. Each sweep is marked once per step, so a hook acts only when the clip did
not run it: the TP hook no-ops when max_grad_norm > 0, and the EP hook gates on whether the sweep
already ran this step, never on max_grad_norm. Every hook sits out the zero-LR step torch takes to
materialize an empty optimizer's state, which a shard restore takes and a shard save takes for an optimizer
that keeps none (plain SGD): a sweep there would claim the step the next one owns. See
BF16 Optimizer.
Non-finite steps
An optimizer step whose global gradient norm is NaN or inf is never applied (halo/trainers/loop/step_guard.py).
Without the guard one such step writes NaN into every parameter and both Adam moments (the toolkit's clips
pass a NaN norm through as coefficient 1.0, torch's multiplies it into every gradient), and the model is NaN
from the next forward on. The norm transformers computes right before optimizer.step() is all-reduced by
every clip path, so every rank reaches the same verdict. The optimizer picks how the verdict is applied:
On the device path the warning and both raises come at the late read, at the latest before the next step's update, so no later step is applied and a final non-finite step is reported before training ends. Against no guard on 8× B300 at s4096 (1.2.0) that path reads TP2 +0.88%, ep1 +0.31%, ep2 +0.69% and CP2 −0.96% (six rounds, ranges overlapping), where a host read before every step costs TP2 3.26% and ep2 42 ms of sync wait per step.
The skip is installed in create_optimizer on the optimizer that steps (the one inside accelerate's wrapper), so
every trainer gets it: before the LR scheduler wraps its step, or over the scheduler's wrapper when the caller
passes a pre-built pair (optimizers=(optimizer, scheduler)), in which case a skipped step still counts as taken
for the scheduler. The zero-LR step that materializes optimizer state for a shard restore or save always steps.
A skipped step's loss is discarded with its gradients: transformers' loss accumulator returns to its value
before the step's first micro-batch (by a device-side select on the device path), so none of the step's
micro-batches, finite ones included, adds to its window's logged loss or to the run's train_loss, whatever
the logging strategy, and later windows stay finite. Both divide by the applied steps alone, so each is the mean
over the steps that were applied; a window none of whose steps was applied logs loss: nan, with
logging_nan_inf_filter on too, and so does train_loss for a run none of whose steps was applied.
A step that saves reads the unread verdict first, so a checkpoint's optimizer state counts the applied steps
only, and a train() call settles one an interrupted run left before it loads any checkpoint (its warning or
raise lands there). Nothing else of the guard is checkpointed: a resumed run starts with no streak.
Router balancing biases (the bias-update modes'
BalancingBiasCallback) still update on a skipped
step. The update reads no gradient: a sign step of at most router_balancing_rate per bias, taken from the
expert-load counts the step's forward recorded, which are finite token counts whatever the gradients held. A
skipped step therefore moves a bias no further than an applied one and cannot make it non-finite.
Under fp16, where accelerate's optimizer wrapper steps through the loss scaler, the scaler owns overflow
steps and the guard takes no verdict; the toolkit's no-wrap prepare keeps the raw optimizer, so there no scaler
steps it and the guard stays armed. Async GRPO's trust-region
breaker is the other skip path: it discards the
gradients in on_pre_optimizer_step, after the guard's verdict, and either one discarding them skips the
step.
DataLoader and data parallelism
DataParallelLoaderMixin overrides get_train_dataloader() / get_eval_dataloader() /
get_test_dataloader() (the eval body, for predict()) to build
dataloaders with a custom DP size/rank so ranks in the same TP/CP/ETP group receive identical data.
The custom path fires when _needs_custom_dataloader() is True: TP, CP or ETP active, or the dataset is pre-sharded per DP rank. EP alone does not trigger
it (EP is orthogonal to DP) unless the dataset is pre-sharded.
Some trainers diverge: SMPO sets custom tokenized signature columns; Classification defaults
remove_unused_columns=False; online and async GRPO scale the batch by steps_per_generation
and rebuild TRL's RepeatSampler at the DP consumption rate
(batch geometry);
Offline GRPO uses MultiGroupSampler.
ParallelismConfig computes both: data_parallel_size
(see Stacking the dimensions) and, through
get_data_parallel_rank(), each rank's shard index per mode
(per-mode derivation).
_prepare_dataloader() passes the computed size/rank to
accelerate.prepare_data_loader() as num_processes / process_index. A dataset already sharded
per DP rank passes 1 / 0 instead, so accelerate places batches on the device without re-sharding
away (N-1)/N of each slice; offline GRPO passes the same pair, its MultiGroupSampler having
already sharded. accelerate's dispatching loader (its default for an iterable dataset) ignores both
and slices rank 0's batches by global rank, so a loader sharded fewer ways than the world never
dispatches: an iterable dataset is sharded by DP rank instead, and an explicit
accelerator_config.dispatch_batches: true raises. Sharded that way, an iterable eval split's final
round repeats its first rows with no length to count them by, so its metrics count them (the eval
loader warns).
accelerate's even-batches shard fills every rank's last eval batch with the split's first rows, so
all ranks run the same rounds; with even_batches off the last batch is short instead, while HF
still repeats each rank's scalar loss eval_batch_size times. The split's own rows fill that final
round in rank order, and the loader's BatchSamplerShard says how many: the rows its sampler draws
(num_generations per prompt under TRL's RepeatSampler) modulo one round
(final_round_split_rows). That count is trusted only
for torch's BatchSampler iteration over a sampler whose length is its rows (sequential, random,
subset, group-by-length, RepeatSampler), which sentence-transformers' default batch sampler is.
A batch sampler with its own iteration counts off its dataset only where its trainer declares it
draws every row once (DATASET_DRAWING_BATCH_SAMPLERS: the embedding trainer's no-duplicates
sampler, exact while only its last batch is short). Any other is not counted, so nothing is cut:
group-by-label draws fewer rows than its dataset holds.
eval_split_rows(n) gives a rank its share of a batch of n, identical across TP/CP/ETP siblings.
The evaluation gather installed on every trainer cuts the gathered final round by the same
geometry, per rank: each rank's chunk keeps the head standing for its own real rows, rounded up, so
a rank with any real row keeps its loss even where HF's loss repeat is narrower than the loader
batch. It runs after keeping one chunk per DP rank wherever the loader is DP-sharded; a loader
sharded any other number of ways (one built around get_eval_dataloader / get_test_dataloader)
raises on its first batch. The object gather (eval_use_gather_object, or a non-tensor input) keeps
the same ranks and cuts each one's entries the same way. accelerate's own remainder counts the
dataset instead, which miscounts a RepeatSampler and trims real rows from a one-process loader
(pre-sharded data, offline GRPO). A half-padded rank's scalar loss still averages over its padding,
except where the eval loss reads eval_split_rows (SMPO, classification, teacher and self
distillation). SFT's eval loss is TRL's
model-forward mean over rows that packing and padding-free collation flatten across examples, so no
row cut reaches it.
When the eval dataloader yields a different number of batches per rank (or has no length),
evaluate() replaces the per-step metric gather with the identity to avoid a deadlock, and warns:
the eval loss and compute_metrics values it logs are then rank 0's shard, not the global value. A dict of
named splits gets this verdict split by split.
Training loop integration
HuggingFace's _inner_training_loop calls accelerator.prepare(model), which normally wraps in
DDP. AcceleratorMixin replaces prepare() with no_wrap_prepare for model/optimizer/scheduler when
_fsdp_wrapped or _qlora_grad_sync or _mixin_manages_ep_layers or is_tp_mode or is_expert_tp_mode or is_pp_mode.
_mixin_manages_ep_layers is needs_ep_layers and not accelerate-managed; needs_ep_layers alone
defaults True even for dense models, so this guards against hijacking an accelerate-managed DDP/FSDP
launch.
In CP mode DistributedSFTTrainer disables HuggingFace's num_items_in_batch loss scaling
(model_accepts_loss_kwargs = False): CP ranks process chunks of the same sequence and FSDP
averaging already gives correct gradients, so without this both loss and grad_norm are inflated by
cp_size.
HuggingFace infers that flag from the model's forward signature alone — any **kwargs makes it True
— and then drops its own /gradient_accumulation_steps division. A trainer whose compute_loss
returns the batch's own mean therefore declares LOSS_IS_OWN_MEAN, which _configure_distributed_modes
applies once for every trainer that declares it. It also sets loss_is_scaled_for_ga = False, since a
compute_loss_func drops the same division.
SFT loss and metrics
DistributedSFTTrainer calls base Trainer.compute_loss, not SFTTrainer.compute_loss, which gathers and reads
its metrics on every micro-batch and assumes the logits cover the whole sequence. It forwards the inputs TRL's
version prepares (_add_trl_forward_inputs; tests/cpu/trainers/test_sft_trl_loss_inputs.py pins them to the
installed TRL) and records accuracy and entropy over the supervised tokens, Liger's fused token accuracy, the
attended tokens and the router aux loss into a per-mode WorldMetrics window (_record_loss_metrics) that
flushes once per log. Under CP, UlyssesModel.forward() owns the boundary handling and the globally-normalized
sum loss (the two corrections), and the recorder mirrors the boundary
handling on the rank's chunk; under its fused head loss it
weighs the chunk's token_accuracy by the chunk's own supervised targets. A run with an MTP head also records ntp_ce and mtp_ce, and at an evaluation its acceptance. Every rank records its whole batch's attended tokens, and the flush divides the
world sum by the ranks sharing a batch (non_dp_replication_factor). A subclass whose own loss records none of
these declares RECORDS_LOSS_METRICS = False and skips the flush (self-distillation, which stores its terms through
StoredMetricsMixin).
Trainer-specific notes
- SMPOTrainer — reference-free preference optimization; loss types
sigmoid,hinge,ipo,smooth_lower_bound; margin scheduling viaVariableSchedulerCallback; CP via cross-rank all-reduce of log-prob sums and token counts. See SMPO. - OfflineGRPOTrainer — advantage methods
z_norm,minmax,quantile_norm,quantile_uniform,robust; PG formulationsprob_weightedorreinforce.LogprobScoringMixin(logprob_scoring.py) owns its log-prob path. - DistributedDPOTrainer — the reference model is never parallelized: under EP/TP a frozen copy is a
whole dense replica per rank (warned);
precompute_ref_log_probs=True, or PEFT/LoRA withref_model=None(LoRA works under EP, not TP), avoids it. - DistributedTeacherDistillationTrainer — losses
kl_divergence,reverse_kl,mse,soft_cross_entropy,cosine_similarity,jensen_shannon,slim; the teacher forward runs undertorch.no_grad()ineval()mode. - DistributedAsyncEnvironmentalGRPOTrainer — async multi-turn RL with Ray actors against vLLM or
SGLang servers (
rollout_backend); rollout generation overlaps training; environments resolved byenvironment_typethroughhalo/environments/registry.py. See Async GRPO.
The halo/trainers/grpo/ package keeps the online trainer (online.py), the offline/ and environmental/
packages, and these shared leaves at the top level: on_policy.py, the constructor spine
(OnPolicyGRPOInitMixin) and generation-round geometry (GenerationRoundMixin) both on-policy trainers
compose; grpo_rules.py, their construction refusals as pure reject_* functions called ahead of TRL's
constructor; early_stop.py; phase_timing.py (PhaseTimedMixin, the phase spans and their report at each log) and
row_span_stash.py (RowSpanStashMixin, the round's row spans), both composed by the on-policy trainers ahead of
DistributedTrainerMixin; rounds/packed_rounds.py (PackedRoundMixin, the packed micro-batches' plan, slot layout and bin-by-bin
scoring), rounds/packed_rules.py (their refusals), rounds/microbatch_plan.py and rounds/row_order.py (the row permutation and the loss types
that admit a regrouping of rows), which both on-policy trainers share; and logprob_scoring.py, the log-prob scoring every GRPO trainer composes (the chunked sweep over
halo/kernels/head_logprobs.py, the full-logits memory guard, the protected-token entropy mask, and
_reference_policy, the model the KL reference forward runs: a held ref_model, else the PEFT policy with
its adapters disabled). Support code lives in objective/ (pure loss-side functions) and rollout/
(weight_push.py, the push and its construction gate; engine_preflight.py, the COLLECTIVE rollout-server
preflights; trl_vllm_client.py; and completions_logging.py, whose DecoupledCompletionsLogMixin both
on-policy trainers compose to write the completions record under save_completions and the console table
under TRL's log_completions). offline/ holds trainer.py, offline_rules.py (its construction refusals),
group_rows.py (one tokenized row per stored completion, carrying its advantage), group_sampler.py,
pipeline_loss.py, reference.py (the reference mode, the run-start
sweep, the checkpoint payload and evaluation reuse) and reference_cache.py (the storage of its
frozen-reference scores).
environmental/ holds the async GRPO trainer, one concern per module: trainer.py
(DistributedAsyncEnvironmentalGRPOTrainer: construction, the round entry, compute_loss, log),
environmental_rules.py (the pure reject_* construction refusals), step_batch.py (StepBatchMixin:
_build_training_tensors as named phases in their collective order, TrajectoryRowMap, StepRows, StepScores, the
BatchBuildFence), loss/importance_sampling.py (ImportanceSamplingMixin: the engine-to-trainer ratio, its mask
stages, the engine re-score and the trust-region breaker), reasoning.py (ReasoningBudgetMixin: the per-level
thinking contract and the rollout rewards with the reasoning length terms charged), routing_replay.py (the
routing-replay injector and RoutingReplayMixin), rollout_routing.py (the round's ragged per-row routing, R2's captured or R3's engine), async_rollouts.py (AsyncRolloutMixin: Ray actors, the prefetch
thread, engine weight sync), trajectory_tokenize.py and trajectory_spans.py (TrajectoryTokenizeMixin:
trajectory → training rows), and rollout_metrics.py (RolloutMetricsMixin: completion logs and per-episode
diagnostics).
Online GRPO vLLM weight sync
DistributedOnlineGRPOTrainer._install_weight_sync replaces TRL's
VLLMGeneration.sync_weights with _distributed_sync_weights at construction, and raises when TRL
built no vllm_generation (server-mode vLLM is required). Before it, the init spine both on-policy
trainers close through (OnPolicyGRPOInitMixin._finish_on_policy_init) runs
reject_unsyncable_model against the trainer's engine.
TRL builds TrlVLLMClient (halo/trainers/grpo/rollout/trl_vllm_client.py), TRL's generate
contract over the vLLM weight-sync client, in place of its own VLLMClient, which would import the
vLLM package (generation).
TRL's default only unfolds DTensors when its DistributedBackend.is_fsdp flag is set (an accelerate
fsdp_plugin). FSDP2 is applied manually via fully_shard with accelerate in MULTI_GPU mode, so
the default forwards DTensors verbatim, and the torch.cat in pack_tensors triggers
a DTensor dispatch that deadlocks against the trainer↔vLLM NCCL group.
_generate_single_turn broadcasts the per-rank rollout result from the TP-group leader; without it
TRL slices the broadcast by process_index, each TP rank lands on different completions, and the
first forward deadlocks on its first all-reduce.
_distributed_sync_weights calls sync_trainer_weights
(halo/trainers/grpo/rollout/weight_push.py, also used by the async GRPO trainer), which gathers,
sends, and closes the engine's update with finish_weight_update — see
the gather for what it collects.
EP layers are found by isinstance(module, EPMoELayerBase) rather than an ep_config probe (a PEFT
modules_to_save wrapper forwards __getattr__ and would match the wrapper too), and PEFT/LoRA is
folded out of place into each base weight as it is sent, under base-model param names, without writing
the frozen base (PEFT).
FSDP2 output capturing
Transformers models capture auxiliary outputs (router logits, hidden states, attention weights) via
a capture_outputs decorator keyed on _CAN_RECORD_REGISTRY (str(class) → capturable flags).
FSDP2 fully_shard creates dynamic subclasses at wrap time (GptOssModel → FSDPGptOssModel) that
are not in the registry, so capture silently fails.
_register_output_capturing_for_fsdp() walks the wrapped model, finds modules with _can_record_outputs not yet registered, and adds them. It
runs from _configure_distributed_modes() only when _fsdp_wrapped=True (the torchrun path);
accelerate-managed FSDP v1 does not create dynamic subclasses and is unaffected.
Checkpoint save and load
The save flow, the saver ladder, the on-disk formats and every resume path are on
Checkpoints & Resume. On the trainer side, _checkpoint_context is the one place a
saver's trainer state is read, and the ladder reads rank-uniform state only, so every rank picks the
identical saver (None = fall through to Trainer.save_model).
Weight resume goes through load_resume_weights() over _weights_load_context(), and the per-rank
optimizer shards through OptimizerShardStore (halo/checkpoint/parallel/) over
_checkpoint_load_context(); where load_resume_weights() declines, _load_weights_through_base runs
the base Trainer's own load, and _load_best_model routes through _load_from_checkpoint too. A trainer
whose saves write a sub-module's names overrides the two hooks (the embedding trainer: the backbone,
and sentence-transformers' own reload).
Around it the mixin orchestrates the non-weight parts of a checkpoint: _save_checkpoint (which
defers save_total_limit rotation until the new checkpoint is complete, and keeps the base's
optimizer.pt out of a save whose shard store writes per-rank shards), and the LR-scheduler and
router_balancing_biases.pt sidecars of halo/checkpoint/parallel/sidecars.py.
_persist_trainer_sidecars is a trainer's own hook, called on every rank before rotation, and
_restore_trainer_sidecars its read-back, called on every rank of a resume (never a best-model
load). The shared reference mixin overrides the write for DPO/KTO and offline GRPO's reference_logps.pt.
DPO/KTO read it back during TRL's __init__, and offline GRPO restores it during construction,
not through the resume hook. The async GRPO rollout mixin overrides both for its pending prefetch
rounds. Each trainer lists the overriding mixin ahead of DistributedTrainerMixin in its bases,
so the empty defaults do not shadow it.
load_best_model_at_end is refused at construction for every shape whose end-of-run reload is
guaranteed to be refused: cp_size > 1, a MoE carrying EP layers (ep_size: 1
plain FSDP2 included, since use_grouped_gemm defaults on), and tp_size > 1 with
data_parallel_size > 1. The reload would otherwise fail after the whole run and export the LAST
weights.
Adapter-only runs reload in place and are exempt — unless merge_expert_lora_on_save folds the
adapter into a full base checkpoint, which the loader refuses like any other.
Related pages
- Checkpoints & Resume · Pipeline Parallelism
- Expert · Tensor · Context Parallelism
- SMPO · Offline GRPO · RLVR Online GRPO · Async GRPO
Source on GitHubUpdated 18 November 2026