Training Callbacks

Callbacks extend transformers.TrainerCallback to log metrics, generate samples, schedule hyperparameters and balance MoE routers. Source: halo/callbacks/.

Scripts assemble them in configure_balancing_and_build_callbacks(...) (halo/entrypoints/script.py): configure_moe_balancing (halo/callbacks/balancing.py, collective) resolves, agrees and applies moe_balancing, then build_run_callbacks (halo/callbacks/factory.py) builds the roster for that mode. run_trainer then reorders HF's report-to integrations last: they fire before user callbacks, so otherwise every key added via logs.update(...) is lost.

Built-in callbacks

CallbackWhat it doesEnable
ParameterStatsCallbackParam counts, modules, dtypes at train beginAlways, on toolkit scripts
GenerateExamplesCallbackEval-time sample generationsgenerate_eval_examples: true
EfficiencyCallbacktokens/s/GPU, step time, memoryenable_efficiency_metrics: true
MoEMetricsCallbackmoe/* per-expert loadenable_moe_metrics: true (default)
BalancingBiasCallbackDeepSeek-V3 bias update, plus moe/*a bias_update mode
TorchProfilerCallbackPer-rank traces, memory timelineenable_torch_profiler: true (Debugging)
PhaseTimerCallbackThe reduce and optimizer phases and the step boundary of the on-policy GRPO phase timeslog_phase_times: true, registered by the trainer
GradientCheckpointingPlanCallbackRelaxes gradient checkpointing to what the memory budget needsgradient_checkpointing_plan: auto
VariableSchedulerCallbackSchedules a model attributeAdded by SMPO (use_margin_schedule) and offline GRPO (initial_min_log_prob)
MtpAcceptanceGateFails a head fit whose held-out acceptance fellmtp_train_scope: head_only (MTP heads)

ParameterStatsCallback

Prints parameter counts, a per-module breakdown (numeric layer indices collapsed to X, so identical layers group) and a dtype / memory table on on_train_begin, main process only. Every count is this rank's local share (local_numel): the callback runs after wrapping, where a plain numel() reports FSDP2/TP DTensors globally and FSDP-ignored EP expert tensors locally — two scopes silently summed.

GenerateExamplesCallback

Generates text samples on on_evaluate and logs them as a table — Prompt, Completion, and Chosen / Rejected on preference rows — to the first backend in report_to (wandb or clearml).

Scripts build it through GenerateExamplesCallback.from_config(...), which returns None unless generate_eval_examples is set and the generate split carries tokenized input_ids — the VLM path has none. That flag defaults to true on DPO / SMPO / offline GRPO, false on SFT.

DDP splits the samples across ranks and gathers on rank 0; FSDP2 and EP/ETP generate on every rank together, forced greedy, since divergent EOS timing would hang the collective. TP, CP and PP are skipped — sharded DTensors, Ulysses attention and partial stages cannot decode. Gradient checkpointing is restored module by module.

EfficiencyCallback

Tracks token throughput, step time and GPU memory; tokens/s/GPU is the headline. MFU, S-MFU and achieved TFLOPS are computed every step but reach the log only under report_mfu_diagnostics: true, staying readable off efficiency_cb.mfu / .smfu / .tps / .memory, with .measured_step_times holding every measured step's unrounded time in seconds. The first two steps are warmup.

Key (each as step_* and avg_*)Meaning
tokens_per_secondcluster_tokens_per_second / world size — the headline, comparable across every parallel layout
cluster_tokens_per_secondUnique cluster tokens over step time: each data-parallel replica's batch once, however many TP, expert-TP, CP or pipeline ranks share it
step_time_secondsOptimizer-step time from the previous step's end to this one's, between CUDA events on the current stream: the host time between two steps (the log, the next batch's fetch) counts, so the steps sum to the wall clock they span, and nothing drains the device for it. The run's first step, and the first of each epoch or after an evaluation or a save, start at their own begin
data_wait_secondsSeconds the step spent blocked fetching its micro-batches from the train loader (host perf_counter around each next, no device sync), the mean over ranks from the step's one existing all-reduce, logged as step_data_wait_seconds and avg_data_wait_seconds. Near zero when the loader keeps ahead of the GPU; with dataloader_num_workers: 0 the collation itself shows here
mfu_percent, tflops_per_secAgainst the device peak in halo/hardware.py; 0.0, with a startup warning, for an undetected GPU or a GPU/precision pair it lacks
smfu_percent, smfu_tflops_per_secMoE-aware twin, crediting only the experts that fire; sparse models only

Memory keys take no prefix: allocated_gb, reserved_gb, peak_allocated_gb, training_peak_allocated_gb; num_full_model_params adds distributed_efficiency.

Off by default: utilization misleads for multi-sequence trainers — DPO / SMPO / reward forward both chosen and rejected, distillation also the teacher. Turning it on sets include_num_input_tokens_seen="all": each micro-batch adds its attended tokens on the device, and the callback's step end sums the step over the world in the same host read as the step's attention work and data wait (HF's own count gathers and syncs on every micro-batch). Without the callback the count is summed at each log and before each save, so it is exact in the logs and in trainer_state.json but lags between them in the state a callback sees.

The peak is keyed on lowp_precision only on the native path; the simulated path runs bf16 matmuls and scores against the bf16 peak. S-MFU scales the expert term by (top_k / num_experts) × ep_size; expert_tp_size already divides the local expert params.

Throughput counts attended tokens, never padding: per token group (input_ids and every <group>_input_ids, as SMPO's prompt / chosen / rejected and the embedding columns) the attention_mask sum, else the row_lengths a packing collator emits for rows it keeps apart under pipeline parallelism, else the last cu_seq_lens_q offset, else the element count of a batch with nothing to tell (count_attended_tokens, halo/trainers/metrics/token_metrics.py). SMPO counts a prompt once although its forward runs it with both completions. Online GRPO's loader yields prompts, not token tensors: TRL adds a round's attended prompt and completion tokens to num_input_tokens_seen in the step that builds the round, and SDPG's teacher forward is not counted. A step with no usable count (no counter, a step that builds no round when steps_per_generation exceeds gradient_accumulation_steps, or an implausible delta) falls back to a padded-length estimate (per_device_train_batch_size × gradient_accumulation_steps × max_seq × 0.8) behind one warning — a figure that reads as measured but is not.

Three cumulative token counters nest: num_input_tokens_seen (attended tokens under include_num_input_tokens_seen: all, the EfficiencyCallback's mode), num_attended_tokens_seen (SFT, DPO, KTO and reward, each batch counted once across its TP/ETP/CP group) and num_unmasked_output_tokens_seen (labels != -100, or the completion mask). Only the first survives a resume, via trainer_state.json; it also drives throughput. A run resumed from a trainer_state.json written when the counter included padding continues from that saved value.

MoEMetricsCallback

Tracks per-expert load for any MoE model with a declared router. A forward hook on each router counts the experts it selected, accumulates across micro-batches, all-reduces and injects moe/* keys. On by default, and a no-op for dense models.

Metric (moe/)Meaning
load_maxMax per-expert share, normalized to uniform routing (>1 = hot expert)
load_minMin per-expert share (→0 = dead or cold expert)
load_cvCoefficient of variation across experts; 0 = perfect balance
dead_fracFraction of experts that got zero tokens this step
load_max_first / load_max_lastMax-load ratio for the first / last MoE layer

Counted is the router's own selection, not topk(router_logits): a DeepSeek-style router selects over biased scores inside an n_group mask while publishing raw logits. Routers come from transformers' _can_record_outputs["router_logits"] declaration; one returning logits alone re-ranks them and warns once. Counters zero every step, and log_every_n_steps follows logging_steps.

It emits nothing in three cases:

  • Under either bias-update mode — BalancingBiasCallback emits the same keys from its load counter.
  • Under pipeline parallelism, where it is not wired: the stage forwards return bare tensors without router_logits, so bias_update, on a family whose EP layers export the bias, is the route to load metrics there.
  • Where output_router_logits is off, stamped forced-off by the balancing strategy or just unset. The flag costs a [B*S, num_experts] tensor per MoE layer per forward and adds the aux loss, so metrics never enable it; opt in via model_init_kwargs.

Two live paths bypass the hooks even when wired: an EP layer routing without the HF router module (SEVERS_AUX_LOSS — Bailing/Ling, Inkling, DeepSeek-V4, Nemotron-H), and a GRPO trainer taking log-probs from the backbone alone. Both warn once after the first step; use bias_update there.

BalancingBiasCallback

DeepSeek-V3 auxiliary-loss-free router balancing (halo/callbacks/balancing.py). The bias is added to the detached top-k scores, so gradients flow through un-biased gating weights and quality matches no-balancing.

The rule is b_i ← b_i + γ · sign(mean_count − count_i) over tokens routed to expert i in the last step, summed across ranks. It runs every step; only the moe/* summary is periodic, since it ends in a host sync. Defaults: update_rate=1e-3, log_every_n_steps=1, exclude_last_slot off the router class attribute _has_discard_expert_slot.

The mode holds an export contract: bias_update is accepted only where the trained bias lands in checkpoint-exported state a serving engine loads, so trainer and server route on the same bias. A family satisfies it with a native buffer, an adopted native slot (an nn.Parameter re-registered as a persistent buffer under the same key, frozen out of gradient training), or a materialized zero buffer plus its config flag.

FamilyExported balancing slot
ZayaNative persistent balancing_biases buffer, in every parallelism mode
GPT-OSSrouter.bias, the hub router's own logit-space bias
GLM-4, Laguna, GLM-5 Next, Inkling, DeepSeek-V4, Step-3.7, Nemotron-Hgate.e_score_correction_bias (Laguna's hub spelling: experts.e_score_correction_bias)
Bailing / LingThe hub gate's persistent expert_bias, adopted whole — no side-buffer fallback
LFM-2The block's expert_bias, materialized with use_expert_bias flipped on for the export
Qwen3, Qwen3.5/3.6, Qwen3-Next, Qwen3.8-Flash-Next, Mistral4, Cohere2 MoENone — bias_update_transient only
Gemma 4None, and no balancing route at all (below)

The contract is checked on the enabled tree, not the class declaration: a renamed upstream slot, or one already FSDP-sharded as a DTensor, falls back to the side-buffer and bias_update then raises. Bailing has no fallback at all (_supports_transient_balancing_bias = False). A family on a weight-sync client's DROPPED_BALANCING_BIAS_MODEL_TYPES (Laguna on vLLM) warns, naming that client's ENGINE_RELEASE, that the engine's loader skips the exported key, so a copy served there never routes on the trained bias.

Requires EP patching — ep_group_size > 1 or use_grouped_gemm, so expert_parallel_size=1 still qualifies with grouped GEMM on. Without those EP layers, and on a dense model, an explicit bias_update raises rather than balancing nothing, except on routers with a native buffer (Zaya). bias_update_transient is refused under pipeline parallelism.

GradientCheckpointingPlanCallback

gradient_checkpointing_plan: auto (halo/callbacks/gradient_checkpointing_plan.py, policy math in halo/models/gradient_checkpointing_plan.py) starts every launch fully checkpointed and measures its step 2, the first at the steady-state footprint (step 1 allocates the optimizer moments and runs the JIT warm-ups). From that one step it predicts the peak of every lighter policy (no checkpointing, then every_n_layers 8, 4, 3 and 2, each checkpointing more layers than the last) and applies the lightest whose prediction fits 85% of device memory (75% under EP, whose dropless routing moves the hot rank's receive count between steps).

  • The layers are the text decoder list. A multimodal checkpoint's vision blocks checkpoint too, but a text run never calls them, so they keep their flags and are not counted.
  • The readings: the peak before each micro-step's first backward recompute, the peak before the optimizer step, the step peak, the allocation one layer's recompute leaves (its uncheckpointed activations) and a checkpoint's input, all host-side allocator stats. Early stop is off for that step, so a non-reentrant recompute runs to its layer's end. The optimizer phase is the same under every policy, so only the forward and backward phases grow.
  • The budget is taken on the world's smallest device, net of the memory outside the allocator (the CUDA context, DeepEP and NCCL buffers), and scaled by the allocated-to-reserved ratio of the most fragmented rank at the measured step, since a device holds what its caching allocator reserves.
  • One plan for the world: the ranks' fingerprints must agree at startup, the readings are reduced in one all-reduce (MAX of the peaks, the recompute, the reserved-to-allocated ratio and the non-allocator bytes, MIN of the input) before the choice, and the resolved plan is checked for agreement, raising on every rank on a divergence.
  • The switch: only the decoder layers' gradient_checkpointing flags move, by transformers' index % k == 0 rule; the installed checkpoint function (the scoped, reentrant one under EP/CP/MoE) stays. A policy replays the same forward, so losses and gradients match full checkpointing bitwise where the kernels are deterministic.
  • The record and its floor: _halo_run_plan.json in output_dir (from the save rank or ranks, like run.log) holds the applied plan, whether the run finished training on it, and its fingerprint: model type, decoder-layer count, hidden and vocab size, attention backend, max_length, batch size and accumulation, world size and parallel layout, GPU model and memory, and the torch and transformers versions. It is written unfinished when the plan is applied and marked finished at train end. The next launch into that output_dir, a resume or a relaunch after a crash before the first checkpoint, still measures, but checkpoints at least one step more than an unfinished plan (an OOM, a crash or a preemption), so a plan never replays into the same OOM; full checkpointing stays full. A finished plan, or one made for other inputs, sets no floor. On a per-node filesystem each rank reads its own node's copy and the world takes the highest floor. Delete the file on every node to plan without a floor; a timestamped output_dir (strftime directives) starts a new directory, so it gets no floor. Exports and Hub pushes leave every top-level _* entry behind, this one included.
  • Rows: only packed rows qualify, where every micro-batch carries up to per_device_train_batch_size × max_length tokens: SFT with packing: true, or a corpus pre-processed packed (the plan reads the rows the collator builds, not the flag). The measured step must hold the run's largest micro-batch: on rows short against max_length nearly every pack is full, but packs that fill unevenly (documents near max_length under bfd) can measure low over few micro-batches, and an OOM then floors the next launch.
  • Refused at startup under pipeline parallelism, torch_compile, auto_find_batch_size (it answers an OOM by halving the batch), trainers that generate rollouts (TRL re-enables checkpointing around generation), rows the collator does not pack (every trainer but SFT), multimodal rows, activation offload, an explicit every_n_layers, text decoder layers that are not GradientCheckpointingLayer (model-level checkpointing), or with checkpointing off.

On 8× B300 (1.2.0), packed dense SFT at s8192 resolves to no checkpointing and runs 25.5% faster than full checkpointing; packed dense SFT at s32768 and Qwen3.5-35B-A3B ep8 match the best fixed policy within 0.4%.

VariableSchedulerCallback

Schedules any numeric attribute on the model over training (halo/callbacks/variable_scheduler.py) — min_log_prob for offline GRPO, target_margin for SMPO.

Python

VariableSchedulerCallback("min_log_prob", -2.0, -5.0)

The attribute lands on the model the Trainer passes in, unwrapped past a DDP/FSDP .module, so the value the loss reads is the scheduled one. It moves linearly from initial_value to final_value over max_steps, then holds; a run with no training steps raises at train begin.

MoE balancing modes

YAML

moe_balancing: auto              # auto | none | aux_loss | bias_update | bias_update_transient
router_balancing_rate: 1.0e-3    # gamma; only used when a bias-update mode is active

auto (the default) resolves per model, below; the four explicit modes are:

moe_balancingEffectrouter_aux_loss_coefoutput_router_logitsCallback
bias_updateSign update into the family's own checkpoint slot; raises on any router left on the side-bufferforced 0forced FalseBalancingBiasCallback
bias_update_transientThe same update in a trainer-only side-buffer no export carries, so a served copy routes without it (near-tied top-k picks flip); raises where every router exports nativelyforced 0forced FalseBalancingBiasCallback
aux_lossThe model's own switch-style aux losspreservedTrue only where the term reaches the lossnone
noneNo intervention; a self-balancing model keeps doing sopreservedpreservednone

The coefficient is the model config's: a router_aux_loss_coef set on a TRL trainer config raises at construction (set it through model_init_kwargs), and TRL's own loss receives the mode's coefficient only in DPO and KTO (Where aux_loss is inert).

Both bias modes force output_router_logits=False, overriding model_init_kwargs: the EP bias path bypasses the HF router module the recorder hooks, so leaving it on returns an empty router_logits tuple and load_balancing_loss_func raises IndexError. These writes are run-scoped — every exported config.json carries the hub's coefficient and flag.

aux_loss sets output_router_logits=True only where router_aux_loss_coef > 0; without a usable coef, or where the EP layers sever the aux path (DeepSeek-V4), it leaves the flag off and warns rather than letting TRL read an outputs.aux_loss those models never populate. With a usable coef it raises where the forward declares no output_router_logits parameter — HF's config fallback lives on that parameter, so a forward that reads the flag from kwargs only never reaches the aux term. The probe reads the model under a PEFT or CP wrapper, both of which pass the flag through.

Under pipeline parallelism no stage runs the *ForCausalLM.forward that adds the term, so a mode resolving to aux_loss (auto included) at a positive router_aux_loss_coef raises at the split; balance with bias_update where the family exports it, or none (PP + EP).

The SFT trainer weighs the term at router_aux_loss_coef once per optimizer step: it averages the window's micro-batch aux losses, and the gradient sync averages them over ranks. In each eval batch it adds the term once at that coefficient, so loss and eval_loss both read as the language-model mean plus router_aux_loss_coef × aux_loss.

HF's num_items_in_batch path multiplies every micro-batch loss, train and eval, by the process count and sums the accumulation window. That is right for the token-summed cross-entropy, but it would weigh the per-micro-batch aux mean world_size × gradient_accumulation_steps times its coefficient (world_size times in eval). DistributedSFTTrainer._weigh_router_aux_loss therefore rescales the term. When HF withholds the labels from the forward, the forward adds no aux term, so the trainer adds it itself. HF does this under a label smoother (label_smoothing_factor) or a TRL compute_loss_func (loss_type: dft with Liger off). Both this rescale and the CP model read the coefficient through require_router_aux_loss_coef (halo/models/moe/config_fields.py), which raises when the config declares none. CP computes its own mean and needs no rescale.

The resolved mode is reconciled world-wide with precedence bias_update > bias_update_transient > aux_loss > none, so every rank runs the same collectives.

aux_loss under gradient checkpointing

Every MoE run with gradient checkpointing is reentrant, and a reentrant checkpoint runs a layer's original forward under no_grad. transformers collects router_logits in that pass, so on its own a checkpointed layer's aux term reaches the loss with no graph and adds no router gradient.

aux_loss therefore installs install_router_aux_gradient (halo/models/moe/router_aux_gradient.py) on the module collecting the logits, the innermost one declaring router_logits in _can_record_outputs. Each tensor collected under a checkpoint's no_grad pass becomes a carrier whose backward keeps the gradient the loss hands it. In the layer's recompute, an identity node on the output of the block owning the router passes that gradient to the router's recomputed logits.

The gradient is the family's own load_balancing_loss_func, differentiated by autograd: the load pooled over all layers, the attention mask, router_aux_loss_coef and the trainer's loss scaling apply exactly as without checkpointing, whichever consumer adds the term (the model forward, a fused-loss forward, the CP wrapper, TRL's KTO). The coefficient stays as configured, so the logged loss and aux_loss read as they do without checkpointing. Collected logits that carry a graph (no checkpointing, the layers every_n_layers leaves unchecked) keep the native path, so no term counts twice. EP needs no collective: each rank computes its aux loss from its own tokens, as HF does.

Every case the hooks cannot route raises instead of dropping the gradient: a kept gradient no recompute took by the end of the backward, one backward carrying the aux loss of two checkpointed forwards, a router run twice in one checkpointed forward or first seen after the model's first forward, recomputed logits of another shape, and an owning block that returns no hidden-state tensor. A model declaring no router_logits capture gets no hooks and a warning: its checkpointed layers' aux term reaches the logged loss without a router gradient.

auto resolution per family

auto wants bias_update on three signals: a native balancing_biases router (Zaya); EP layers that sever the aux-loss path (DeepSeek-V4, Bailing, Inkling, Nemotron-H); or a forward taking no output_router_logits while an EP layer accepts the bias.

It commits only where the bias would land in exported state. GLM-4 MoE Lite, LFM-2 and Step-3.7 qualify on the third signal — none of their modelings has aux-loss machinery. GLM-5 Next does not: its forward honors the flag, so auto gives aux_loss while the exported slot still serves an explicit bias_update, the Laguna shape.

Mistral4 and Cohere2 MoE have no slot and resolve to none, warning that bias_update_transient is the deliberate opt-in.

aux_loss follows for the families whose forward declares the parameter: GPT-OSS, Qwen3, Qwen3.5/3.6 (multimodal and text-only), Qwen3-Next, Qwen3.8-Flash-Next, Laguna, GLM-5 Next. Where it does not and no EP layer accepts a bias either, the model has no balancing route at all, auto resolves to none with the reason, and an explicit aux_loss raises.

That is also what use_grouped_gemm: false at ep_size = 1 does to GLM-4 MoE Lite, LFM-2 and Step-3.7: with no EP layers to carry the bias, the run trains unbalanced.

Gemma 4 has no balancing path

auto resolves Gemma 4 to none with a warning: its EP layer accepts no bias, no router ships a balancing_biases buffer, and its forward declares no output_router_logits parameter. An explicit bias_update raises, because the router lives in the parent decoder layer rather than the EP module. An explicit aux_loss finds no router_aux_loss_coef either, so it stamps the flag forced-off and warns rather than raising.

Either way Gemma 4 gets no balancing and no moe/* metrics, and output_router_logits: true in model_init_kwargs cannot buy them back — no Gemma 4 config level declares the field, so the override is rejected at load. Every shipped Gemma 4 config sets moe_balancing: none.

Where aux_loss is inert

*ForCausalLM.forward adds router_aux_loss_coef * load_balancing_loss_func(...) only when labels is not None, so aux_loss balances experts under SFT and pretraining, with or without gradient checkpointing. DPO and KTO reach it another way: TRL's DPOTrainer and KTOTrainer add the term themselves, at the coefficient the balancing mode put in effect. Everywhere else the term never reaches the loss:

  • SMPO / reward — the forward runs without labels, and neither adds the term.
  • Distillation (teacher and self) — both strip labels, for full-vocab logits.
  • Classification / embedding — non-causal heads; the causal-LM forward never runs.
  • GRPO (offline / online / async) — the loss is per-token log-probs, so the mode warns and leaves the flag off; with output_router_logits already on at a positive router_aux_loss_coef, an explicit aux_loss raises at construction (below).

Inertness is a trainer-class contract: ROUTER_AUX_LOSS_ADDED_BY declares what adds the aux term: the model's labels forward (DistributedSFTTrainer), TRL's own loss (DistributedDPOTrainer, DistributedKTOTrainer), or nothing (every other trainer). An explicit aux_loss on a non-consuming trainer raises; an auto resolution landing there turns output_router_logits back off.

Online and async GRPO go further: syncing weights to a live rollout engine downgrades both bias modes to none, since the sync ships parameters and carries no bias that moves — an adopted native slot is a buffer, a transient bias a plain attribute — and a bias the generator never sees drifts trainer routing off the trajectories it produced. Ling 2.0's gate.expert_bias does ride the push, at its checkpoint value (Partly loaded layers).

With aux_loss inert under the same loss, those runs have no router balancing on any family.

The balancing sidecar

Every bias mode checkpoints its biases to router_balancing_biases.pt on the FS-aware save rank and restores them on resume.

The restore is all-or-nothing across ranks: a missing or torn file on some ranks raises, as does a saved bias whose shape does not match the live router, rather than being copy_-broadcast into it. A checkpoint without the file keeps zero-init biases and warns; a sidecar matching no live router drops every trained bias, loudly — what a weight-sync RL leg does to a bias_update checkpoint.

The PEFT merge (merge_peft_adapters.py) applies the sidecar into the model's native slots, materializing a config-gated slot the base lacks — a merge starts from base weights that never saw the sign updates. A full checkpoint already carries the biases in those slots, so convert_to_bf16.py reads them back from its shards at the trained fp32. Saves keep balancing tensors at their trained fp32; a bf16 round trip would quantize away several 1e-3 steps.

Relevant YAML

YAML

generate_eval_examples: true      # + num_eval_examples: 50
enable_efficiency_metrics: false  # report_mfu_diagnostics and num_full_model_params need it on
enable_moe_metrics: true
moe_balancing: auto
router_balancing_rate: 1.0e-3     # gamma, bias-update modes only
logging_steps: 1                  # also the moe/* summary period

Defaults and full descriptions: Performance & balancing flags.

Writing custom callbacks

Extend transformers.TrainerCallback and override hooks. Each receives args, state, control and **kwargs — HF's CallbackHandler passes model, processing_class, optimizer, lr_scheduler, the two dataloaders, and no trainer.

Python

class LossSpikeCallback(TrainerCallback):
    def on_log(self, args, state, control, logs=None, **kwargs):
        if logs and logs.get("loss", 0) > self.max_loss:
            control.should_training_stop = True   # also: should_evaluate/save/log

Distributed rules: gate prints on is_global_main_process() (halo/runtime/ranks.py); unwrap FSDP/DDP before .generate(); wrap collectives in barrier(), keeping any participation gate rank-uniform. A reusable callback goes in halo/callbacks/ and is imported from its own module — the package init re-exports nothing.

Source on GitHubUpdated 18 November 2026