Configuration Reference

Every training script takes a YAML config file as its first positional argument. Config classes extend HuggingFace TrainingArguments, so all its parameters (learning rate, batch size, scheduler, …) are valid everywhere. Any field can be overridden on the command line; CLI wins, and every config is built from the merged values (CLI overrides). Every numeric knob the toolkit declares refuses a bool and a NaN or infinity at parse time, and an integer knob a fraction (ParallelismArguments checks only the type; ParallelismConfig owns the ranges):

Bash

python scripts/training/sft.py examples/sft/qwen3/qwen3-4b-ultrachat.yaml \
    --learning_rate=0.00001 --num_train_epochs=2 --per_device_train_batch_size=2

A guard's refusal names the config class and the field in one form, <Config>: <field> must be [an integer ]<bounds>[ (<why>)], got <value> (halo/config_validation.py).

Working configs per method and model family live under examples/.


Sequence length: caps vs generation budgets

Three knob names carry the length settings. They mean the same thing everywhere; only the mechanism that enforces them differs per trainer, because some data can be safely truncated and some cannot.

KnobMeaning
max_lengthTotal tokenized sequence budget (prompt + completion). null or any non-positive value resolves to the model's context window at launch (resolve_length_to_context, which reads max_position_embeddings / max_seq_length / n_positions off the text sub-config so composite VLM configs resolve too, then falls back to tokenizer.model_max_length). Also becomes tokenizer.model_max_length for the run.
max_prompt_lengthPrompt share of the budget.
max_completion_lengthCompletion share of the budget, or — under RL — the number of tokens the policy may generate. Async GRPO with Environments' entry script pins it to rollout_max_tokens, the generation cap there, and raises on any value other than TRL's default or rollout_max_tokens.

Omitting a field falls back to the dataclass default in the tables below. Writing max_length: null opts into the model's context window; null on max_prompt_length / max_completion_length means whatever the per-trainer row below says (no cap, no prompt filtering, or a derived share of max_length) — never the context window.

Per-trainer mechanism

Trainer (config)max_lengthmax_prompt_lengthmax_completion_length
SFT (SFTConfig)drop: the chat processor tokenizes untruncated and discards any conversation over the budget (cutting one mid-turn corrupts it). Under packing it is also the fixed pack size, which must be explicit — the script raises on null——
DPO (DPOConfig)TRL truncates the concatenated prompt + completion keep_start— (no training-side cap; see below)—
KTO (KTOConfig)TRL truncates the assembled prompt + completion to it, keeping the start——
SMPO (SMPOConfig)total budget, split into the two shares below (resolve_length_budget)truncate per truncation_mode; null → half of max_lengthtruncate (EOS only within the budget); null → the max_length − max_prompt_length remainder. The two shares must fit inside max_length
Reward (RewardConfig)TRL filters: pairs whose chosen or rejected exceeds it are dropped, not truncated——
Classification (ClassificationConfig)truncate at tokenization on the script path; a dataset handed straight to the trainer as raw text/label is tokenized untruncated and filtered instead——
Distillation (TeacherDistillationConfig)over-length conversations are dropped, not truncated——
Embedding (EmbeddingConfig)installed as SentenceTransformers' max_seq_length (truncates)——
Offline GRPO (OfflineGRPOConfig)not a key: the budget is the two shares, and pipeline parallelism pads every row to their sumleft-truncate (keep the tokens nearest the completion)truncate the stored completion (EOS only within the budget); also the dr_grpo loss normalizer
Online GRPO — RLVR (GRPOConfig + script args)—dataset filter: over-length prompts are dropped, never truncatedgeneration budget (TRL max_new_tokens / vLLM SamplingParams)
Async GRPO (AsyncRolloutConfig + script args)—dataset filter (as above)not a knob — the script pins it to rollout_max_tokens and raises on any other explicit value; TRL reads it only as the dr_grpo normalizer

Two knobs are not a plain cap:

  • DPO has no training-side prompt cap. TRL truncates the concatenated sequence keep_start, so an over-long prompt eats its own completion — filter them in the dataset. DPOScriptArguments.generation_max_prompt_length bounds only the eval-time generation dataset.
  • Async GRPO never truncates a trajectory. rollout_max_tokens is the per-turn generation budget and rollout_max_episode_tokens the episode's (unset by default; an episode that runs it down ends truncated after a whole turn, never mid-row); the multi-turn trajectory accumulates across turns and is bounded by the served model's context window — one that exceeds it fails (trajectory length).

The tokenizer's window

Every training script but embedding routes its tokenizer.model_max_length write through one of three seams in halo/entrypoints/model_setup.py, each pinning its window through apply_tokenizer_window:

  • apply_max_length — every single-budget script (SFT, DPO, KTO, SMPO, reward, classification, distillation). It resolves max_length against the context window, writes the resolved number back onto the config, and pins the same number, so collators, length filters and packers all read one already-resolved value.

  • apply_prompt_completion_window — offline and online GRPO, which have no single max_length. The window is max_prompt_length + max_completion_length, pinned only when both halves are bounded.

    HF resolves any truncation=True, max_length=None call against model_max_length, so pinning a partial sum would turn the unbounded half into a silent cap at the other half's budget. An unbounded half therefore leaves the tokenizer at its own value. On the on-policy scripts the completion half is a generation budget with no unbounded setting; an unset one is rejected here.

  • apply_tokenizer_window with resolve_model_context_window — Async GRPO, whose multi-turn trajectory no configured length bounds (rollout_max_episode_tokens caps the generated share alone). It pins the raw context window, the same hard limit the rollout server enforces, which the trainer fails against rather than truncating.

Boundedness is one predicate everywhere (is_bounded_length): a positive int bounds, while null and any non-positive value mean unset.

The pin is run-scoped and never exported. save_pretrained writes the live model_max_length into tokenizer_config.json, so an unrestored pin would ship the training budget as the served context — a run at max_length: 40000 capping a 262k-context model. configure_run_tokenizer records the tokenizer's own bound and DistributedTrainerMixin.save_model runs every writer under pristine_model_max_length, which serves that bound for the duration of the save and puts the pin back afterwards.

Embedding training pins nothing: its length is installed as SentenceTransformers' max_seq_length. The inference scripts (scripts/inference/) take a raw CLI number and resolve nothing.

The YAML parser migrates no spelling: max_seq_length is a field of no config, so it hits the unknown-key raise (Parsing).

Performance & balancing flags

Every standard training script inherits these from CommonScriptArguments; configure_moe_balancing (halo/callbacks/balancing.py) applies the balancing mode and build_run_callbacks (halo/callbacks/factory.py) builds the observability and balancing callbacks.

ParameterTypeDefaultDescription
enable_efficiency_metricsboolfalseConstruct EfficiencyCallback. Off by default: multi-sequence trainers (DPO/SMPO/Reward forward chosen + rejected; Distillation also forwards a teacher) report a misleading utilization. Sets include_num_input_tokens_seen="all" when on — the HF field is tri-state ("no" | "all" | "non_padding"), not a bool.
gradient_checkpointing_plan"off" | "auto""off"auto measures step 2 fully checkpointed, then checkpoints only the text decoder layers the memory budget needs; packed SFT rows only: GradientCheckpointingPlanCallback.
enable_moe_metricsbooltrueConstruct MoEMetricsCallback: moe/load_{max,min,cv}, moe/dead_frac, moe/load_max_{first,last}, emitted every logging_steps. No-op for dense models. It turns output_router_logits on itself only under moe_balancing: aux_loss, so asking for metrics never adds the router-logit tensor or the aux term as a side effect. Not wired under either bias mode, where BalancingBiasCallback emits the same keys from its load counter instead, nor under pipeline parallelism, whose stage forwards return no router_logits (bias_update, on a family whose EP layers export the bias, is the route to load metrics there). It reports nothing under GRPO (backbone-only log-probs) or where the EP layer bypasses HF's router recorder (DeepSeek-V4), warning loudly rather than going silent.
moe_balancing"auto" | "none" | "aux_loss" | "bias_update" | "bias_update_transient""auto"Router balancing; auto resolves per model. Resolution rules, per-family support, the trainers aux_loss reaches and the weight-sync downgrade: MoE balancing modes.
router_balancing_ratefloat1.0e-3Sign-step magnitude (γ) for BalancingBiasCallback when a bias-update mode is active. Refused with an explicit moe_balancing: none or aux_loss; not read where auto resolves to another mode, or on the weight-sync RL scripts.
num_full_model_paramsfloat | nullnullTotal param count across all EP/TP ranks. When set, EfficiencyCallback computes distributed_efficiency = params_ratio * mfu. Refused with enable_efficiency_metrics off.
enable_torch_profilerboolfalseConstruct TorchProfilerCallback — captures a step window, writes per-rank Chrome trace + flame-graph stacks + memory timeline.
profiler_output_dirstr$HALO_DATA_ROOT/profiling/torchOutput dir for torch.profiler artifacts (derives from HALO_DATA_ROOT; set explicitly to override). Like every profiler_* knob, refused off its default with enable_torch_profiler off.
profiler_wait / profiler_warmup / profiler_activeint5 / 1 / 3torch.profiler schedule: skip wait, warm up warmup, record active steps (one-shot).
profiler_ranksstr"0"Which global ranks profile: "0", "all", or a comma list like "0,8".
profiler_record_memory_snapshotboolfalseAlso dump a CUDA memory snapshot (.pickle) over the active window for memory_viz.

Optimizer selection

optim is HuggingFace's field, extended at import time with two toolkit values (halo/optimizers/registry.py registers everything in NAMED_OPTIMIZER_BUILDERS) so they pass TrainingArguments validation.

optimOptimizerNotes
adamw_torch_fused (HF default) / adamw_torchAdamW, or AdamWBF16 when auto-enabledWith bf16: true this resolves to AdamWBF16 (6 B/param, stochastic rounding) everywhere except an accelerate-managed DDP launch, which is outside the validated SR matrix — set bf16_optimizer: true to opt in there. Mixed dtypes are handled internally: bf16 params take the SR kernel, fp32 params (fp32_router, fp32_experts, fp32_non_ep_params) take standard in-place AdamW.
muonMuonNewton-Schulz orthogonalization on 2-D params; embeddings and the LM head are excluded.
flash_adamwFlashAdamWQuantized optimizer states, ~5 B/param.

bf16_optimizer: true together with optim: muon or optim: flash_adamw raises — both select an optimizer and the bf16 path would silently win (reject_conflicting_optimizer_choice). Any other explicit optim (e.g. adamw_bnb_8bit) suppresses the AdamWBF16 auto-enable. Pipeline parallelism refuses muon and flash_adamw (reject_pp_named_optimizer): its stage-local optimizer build and per-stage optimizer shards are validated with the AdamW family only (PP rejected combinations).


Model, checkpoint and from-scratch flags

Every training script parses these. reset_sinks, train_sinks and text_only_model (ModelLoadArguments) and the four checkpoint flags (CheckpointArguments) are on every script's CommonScriptArguments (halo/configs/script_arguments.py); init_from_scratch and merge_expert_lora_on_save are ParallelismArguments flags (halo/configs/parallelism_arguments.py, whose parallelism sizes are under ParallelismConfig).

ParameterTypeDefaultDescription
init_from_scratchboolfalseBuild the model with random weights from model_name_or_path's config instead of loading pretrained weights. SFT script only (other scripts reject it); dense DP / FSDP2 only — raises for EP/TP/CP/ETP/PP, and for a model holding an external table (Qwen3.8-Flash-Next), which only a checkpoint supplies. See Pre-training.
reset_sinksbooltrueGptOss only: neutralize the attention sinks for SFT — dtype.min and frozen (removed under FA2). false keeps the pretrained sinks live and frozen, the on-policy RL setting (the trainer scores with the sinks the rollout engine serves). Live sinks need a sink-carrying attention implementation — FA4 on Blackwell, flex_attention or eager on Hopper; FA2 and SDPA are rejected — and are refused under Context Parallelism. A reference model whose sinks differ from the policy's is rejected, in every parallelism mode, since the KL would be biased on every token: the frozen copy the DPO/KTO and GRPO scripts load takes the policy's reset_sinks and passes, while a reference TRL builds itself keeps its pretrained sinks live (GptOss's default eager attention applies them) and is refused beside neutralized ones. Auto-skipped under init_from_scratch. See GPT-OSS.
train_sinksboolfalseGptOss only: keep the live sinks trainable (requires reset_sinks: false; the contradiction is rejected). Full fine-tuning under FA4 (the loader installs an exact sink-less + sigmoid(lse - sink) rescale on both kernel entry points instead of the kernel's own sink backward, which is not validated here) or eager; flex_attention and FA3 are rejected (no usable sink gradient), as are adapter runs, weight-sync RL and a model without sinks. Counts as live for every sink gate.
text_only_modelboolfalseLoad a multimodal checkpoint through its text-only CausalLM class instead of the vision-bearing wrapper (Qwen3.5/3.6: Qwen3_5MoeForCausalLM). Drops the vision tower and, unless mtp_loss_weight trains it, any MTP tail from the build and the export (a text-only artifact: no processor_config.json, no vision token ids). Image-bearing datasets are refused loudly. Honored by SFT, DPO/SMPO/KTO, teacher/self-distillation and offline GRPO. Rejected at startup by online/async GRPO (the weight sync sends model.layers.* names where the served multimodal checkpoint expects model.language_model.layers.*) and by reward, classification and embedding (their model class is pinned). For Qwen3.5/3.6 both pinned engines (vLLM 0.32.0, SGLang 0.5.22) register the text-only class and load the export unchanged; SGLang registers only Qwen3.8-Flash-Next's multimodal class, so run scripts/after_training/reattach_vision_tower.py on that export first (Qwen3.8-Flash-Next). The tool rebuilds the multimodal artifact (Qwen3.5).
save_sharded_epboolfalseSave EP checkpoints as per-rank shards (every rank writes its own slice in parallel) instead of gathering to one rank — write bandwidth, not host memory, since the gathered path streams. The result loads nowhere until merge_ep_shards.py runs; the merged directory resumes exactly, optimizer included, on the save's topology. The default gathered path writes an HF-standard sharded checkpoint (model-XXXXX-of-YYYYY.safetensors + index, save_max_shard_size 5 GB), which resume reads directly. Refused at trainer construction under PP and for every shape the merge cannot reassemble — more than one EP group, ETP, CP, native expert LoRA, a multi-node run on a non-shared output filesystem (full list).
save_max_shard_sizestr"5GB"Max size of one safetensors shard written by the distributed gathered (EP/TP/FSDP2/CP) save paths; anything but a positive size like 5GB is refused at parse. HF's own save_pretrained keeps its own default. A sharded EP save writes one file per rank by design and reports that it does not apply; merge_ep_shards.py sizes its output by its own --max_shard_size.
overwrite_output_dirboolfalseAllow starting a run in a non-empty output_dir. Off by default so a fresh run cannot interleave its checkpoints with an existing one's. See Checkpoints.
allow_optimizer_warm_restartboolfalseResume only. When restoring optimizer shards that match this run's topology fails on any rank (an unreadable shard, a CUDA OOM while applying it), or the checkpoint holds optimizer state this run cannot restore (an interrupted save's shards with a missing or unreadable optimizer_meta.pt, a replicated optimizer.pt with no shards beside it), resume with a freshly initialized optimizer and a warning instead of raising on every rank; weights, step and LR schedule still resume. Which resume outcomes it covers: Checkpoints.
merge_expert_lora_on_saveboolfalseFold the adapter delta into the base on save, so the checkpoint is a fully-merged, servable HF model. Covers both halves of a mixed run: the expert deltas inside each family's gather, and any attention adapters folded out of place into each tensor as it is written (the frozen base is never written, so training continues unchanged). Each training checkpoint also keeps the unmerged adapter in resume_adapter/, which resume restores onto the base; the final export carries none (Merge-on-save checkpoints). Requires native grouped expert adapters — rejected when the run built none, under accelerate-managed FSDP, and with save_sharded_ep: true (merging is only wired through the gathered EP save). false writes a standalone adapter file and leaves the base unchanged.

Environment variables

Read from the environment (not the YAML); set in the launch command / .env. Toolkit knobs plus the third-party variables a run sets or depends on. A toolkit boolean takes 1/true/yes/on or 0/false/no/off in any case; another value warns once and keeps the default, as a malformed number does. The image-baked CuTe DSL variables are in Docker, the Liger RoPE pin in Liger Kernels, and third-party credentials (WANDB_API_KEY, HF_TOKEN, OPENAI_API_KEY, the web-search backend keys) in Dev Environment.

VariableDefaultDescription
DIST_SHARED_FILESYSTEM1Umbrella for both sides: 1 = shared FS (only global rank 0 writes downloads/checkpoints); 0 = per-node local disk (each node's local rank 0 writes). Must be identical on every rank — the flags select the coordination scope, so init_distributed() broadcasts rank 0's resolved values for all three and warns any rank whose env disagreed. See Filesystem Handling.
DIST_INPUT_SHARED_FILESYSTEMumbrellaRead side only (model/dataset downloads, dataset map/pack, HF caches): picks the fs_aware_main_first coordination scope and is_fs_load_rank(). Falls back to the umbrella while unset, and wins over it once set. Set 0 with a shared umbrella when a multi-node NFS/EFS mount returns stale file handles on cross-node cache reads. Multi-node runs probe HF_HUB_CACHE and HF_DATASETS_CACHE against it at startup and raise on a contradiction in either direction.
DIST_OUTPUT_SHARED_FILESYSTEMumbrellaWrite side only (checkpoints, run.log, dumped artifacts): picks is_fs_save_rank(). Falls back to the umbrella while unset, and wins over it once set. Non-shared forces save_on_each_node on and rejects a multi-node sharded-EP save (per-rank shards scatter across nodes with no gather path); shared forces it off across nodes, where every node's local rank 0 would write and rotate the same files. Multi-node runs probe the declaration against the filesystem at startup and raise on a contradiction in either direction.
DIST_NCCL_TIMEOUT_MINUTES30NCCL collective-watchdog timeout, read as nccl_timeout() (halo/env.py), applied by init_distributed (PyTorch's own default is 10) and pinned as the default for every later dist.new_group(), so EP/CP/TP subgroup collectives inherit it. The one knob for that watchdog: the group starts before the config is parsed, so a non-default ddp_timeout in the config is refused. Does not bound DeepEP V2's dispatch/combine (its own ElasticBuffer barrier over the NCCL Gin backend). Raise for slow cross-node all-to-all, or for a gathered save whose single part (save_max_shard_size) writes slower than this. Must be >= 1 — a non-positive value warns and falls back to the default. Rank-uniform (agree_rank_uniform_env): a node left at a smaller budget aborts its process group first, and the survivors then die on the next collective and blame it.
DIST_STORE_TIMEOUT_HOURS4Wall-clock bound for the c10d-store coordination waits, read as store_timeout() (halo/env.py) — fs_aware_main_first (main-first downloads), run_load_rank_first (dataset load, map/filter, packing, TRL dataset preparation), store_reject_across_ranks (the joins behind it, the sharded-load join, the model-source agreement, and the checkpoint joins after the per-rank optimizer and sharded-EP shard writes, a gathered save's last part and index, and each pipeline stage's parts), and throttled_load_within_node (the max_concurrent_loading throttle, joined world-wide by joined_node_load). These wait out ONE rank's work rather than a collective, so they are hours-scale and independent of DIST_NCCL_TIMEOUT_MINUTES. Must be >= 1 (same fallback rule) and rank-uniform — the rank with the smaller budget gives up on a join its peers are still serving. See Filesystem Handling.
DIST_FS_PROBE_TIMEOUT_SECONDS60How long a rank declared on shared storage polls for the multi-node filesystem probe's sentinel, read as fs_probe_timeout() (halo/env.py); a per-node declaration never waits. The default is NFS's acdirmax: raise it above a mount's actimeo/acdirmax, or where lookupcache keeps negative lookups, so a shared mount is not refused as unshared. Must be >= 1 (same fallback rule) and rank-uniform (agree_rank_uniform_env). See Filesystem Handling.
NVLINK_DOMAIN_SIZEgpus_per_nodeThe locality unit for node-local EP/CP/TP/ETP grouping — validation checks NVLink-locality against this, not gpus_per_node. Set to 72 on GB200/GB300 NVL72 to run NVLink-wide across OS nodes; leave unset on ≤8-GPU nodes. Also settable as --nvlink_domain_size. Rank-uniform, refused at startup when ranks differ. See NVL72.
VLLM_GROUP_HOSTloopback (local server) / the address on the route to the serverTrainer interface advertised to vLLM workers for the NCCL weight-sync group, and the one address its rendezvous store listens on unless HALO_WEIGHT_SYNC_BIND_ALL is set. Set where a remote vLLM cannot dial back to the trainer's address on its route to the server (NAT, asymmetric routing). Resolution order and the multi-homed cases: Online GRPO → Multi-homed nodes (async GRPO also takes a per-server group_host in rollout_server_configs; Online GRPO has only the env var).
SGLANG_GROUP_HOSTloopback (local server) / the address on the route to the serverTrainer interface advertised to SGLang workers for the NCCL weight-sync group, and the one address its rendezvous store listens on unless HALO_WEIGHT_SYNC_BIND_ALL is set: the VLLM_GROUP_HOST equivalent for rollout_backend: sglang. Separate per engine because the two servers can sit on different hosts.
HALO_WEIGHT_SYNC_BIND_ALLoff1 puts the weight-sync rendezvous store (both engines) on every interface instead of the advertised group address alone, for a trainer the server reaches through NAT or a port mapping; each group formation then logs a WARNING. Off, an advertised address this host cannot bind alone (not local, 0.0.0.0, or a name resolving to loopback for a remote server) raises at group formation. An IPv6 group address raises in either mode: the store listens on IPv4 only. The store is unauthenticated, so set it only where no untrusted host reaches the group port (Group rendezvous).
HALO_SCRATCHlocal mount with the most free space (make)Host-side scratch root the Makefile, the devcontainer and the vLLM compose file read (never Python): bind-mounted into containers and deriving the in-container HALO_DATA_ROOT. Unset, make takes the local mount other than / with the most free space over 20 GB and stops a target that needs it when none qualifies; the devcontainer refuses to start, and so does every service of the vLLM compose file, vllm-server included (compose interpolates the whole file; set it once in the repo-root .env).
HALO_DATA_ROOT~/.cache/haloToolkit scratch root — the single knob for toolkit-owned caches and outputs. The S3 dataset cache (<root>/s3_datasets), profiler artifacts (<root>/profiling: torch traces, memory snapshots, py-spy captures) and the batched-generation resume store (<root>/inference/resume) derive from it. Point it at a verified large volume. HuggingFace caches are governed separately by HF_HOME / HF_DATASETS_CACHE.
HALO_S3_DATASET_CACHE_DIR$HALO_DATA_ROOT/s3_datasetsExplicit override for the S3 dataset cache. A rank waits DIST_STORE_TIMEOUT_HOURS for the peer downloading the same entry.
HALO_S3_DEFAULT_BUCKETunsetBucket for the key-only S3 helpers (build_s3_uri, a bucket-less S3Client()): a bare-key --dataset / --output_path of openai_batched_generation.py (a full s3:// URI needs none) and s3.py's --bucket default. A key-only path raises while it is unset. A training dataset: entry always needs a full s3:// URI.
HALO_S3_MAX_FOLDER_CONCURRENCY16Files transferred in parallel per S3 folder operation, clamped to >= 1. Each file above boto3's multipart threshold uses up to 5 connections from one 50-connection client pool, so large-file folders exceed the pool at the default (urllib3 warns Connection pool is full); 10 stays inside it.
HALO_DATASET_NUM_PROCmax(1, min(3/4 × usable CPUs, 64))Worker processes for the rank-coordinated dataset.map/filter and sequence packing (the packed rows do not depend on the count). Unset, each host derives it from the CPUs the process may use: its affinity set, capped by the tightest cgroup CPU quota on its cgroup or an ancestor (docker --cpus, a Kubernetes CPU limit, a systemd CPUQuota). The load rank of each input filesystem maps with the whole count while its node's other ranks wait, and they then read its cache files back whatever their own count, so hosts that derive different values still share one cache. Those ranks compute only on a per-rank key (a pre-sharded dataset), all at once, so each maps with its 1/LOCAL_WORLD_SIZE share of the count, at least 1; a count set here or by a config dataset_num_proc (which overrides this) is shared the same way. prepare_dataset.py runs alone, so it defaults to every usable CPU but two instead. Must be >= 1 (below 1 fails loud); 1 disables map multiprocessing. Resolved once at import. The waiting ranks are bounded by DIST_STORE_TIMEOUT_HOURS: raise it for a training-time map of a large corpus, or bake the corpus offline.
HALO_DSA_MATMULon on Blackwell, else off0 leaves GLM-5 Next's DSA layers on stock SDPA (mem-efficient at their head width on Blackwell, cuDNN on Hopper) instead of the masked matmul kernel the sdpa slot of modeling_glm5_next runs on CUDA. Read at each load, and not compared across ranks, as the kernel runs no collective and its one Triton launch compiles at load. See GLM-5 Next.
HALO_FLEX_SLIDINGon0 builds an SDPA run of a sliding-window or wide-head model (Step-3.7, Gemma 4) with plain sdpa instead of sdpa_flex_sliding (FlexAttention on sliding-window layers and on packed or padded full layers, matmul or per-document mem-efficient attention on global layers SDPA can only run mem-efficient). Rank-uniform, refused at startup when ranks differ. See Flash Attention.
HALO_FUSED_GLUon0 runs every GLU combine (expert and dense MLP) through its eager form on CUDA instead of the fused Triton kernels, and each EP layer's construction line then reads glu_combine=eager. The switch for a GPU where a GLU kernel fails to compile or launch (Troubleshooting).
HALO_HC_KERNELSon0 leaves upstream's hyper-connection and decoder-layer forwards on GLM-5 Next and DeepSeek-V4 instead of the hyper-connection kernels: the A/B, and the switch for a GPU where one fails to build. Read when the family's patches apply, at load.
HALO_QSA_KERNELautoauto, on or off: whether Qwen3.8-Flash-Next's QSA layers attend on the block-sparse kernel or on the tiled SDPA path. on raises where the kernel cannot run (not CUDA, a dtype other than bf16 or fp16, a head width off a power of two, full_determinism), auto takes the tiled path there and logs the reason once on rank 0; a layer whose kernel fails to build at load (Triton OutOfResources, a compile error) takes the same path under auto and raises under on. Read per call, and at load by the kernel warm-up.
HALO_QSA_INDEXER_MIB256Memory budget of one chunk of the QSA indexer's batched block scores (fp32 products, their ReLU and the scores); a larger budget scores more queries per matmul. It does not change the selected blocks.
HALO_GATED_RESIDUAL_KERNELSon1 runs both gated residuals of every Qwen3.8-Flash-Next layer, and the injection of each block's output into the four streams, on the gated-residual kernels where a module can (CUDA bf16, fp16 or fp32 streams, plain hook-free mix projections in the streams' dtype), else on upstream's forward, logging the reason once on rank 0; trained and frozen weights both run on them. 0 runs upstream's forward. Read per call, and at load by the kernel warm-up, so the launch must set it on every rank.
HALO_FLA_L2NORM_FIXED_NBoff1 launches FLA's q/k l2norm kernels (gated-delta-rule modules holding head_k_dim) with a constant NB and drops NB from their autotune key, so one compiled kernel and one tuned config serve every row count (Qwen3.5). Raises at load where the locked FLA's kernels no longer look like that. Read at load, by apply_fla_launch_options.
HALO_EP_SHARED_OVERLAPoff1 runs the shared-expert FFN on a side stream concurrent with the routed dispatch all-to-all (shared-expert families only; GPT-OSS, Qwen3 MoE, LFM-2, Gemma 4 and Zaya have none). See DeepEP.
HALO_EP_CAPACITY_DEDUP10 restores the per-MoE-layer DeepEP buffer-capacity all-reduce instead of reusing the first layer's capacity for the whole forward. One capacity per forward is also what lets every MoE layer share one ElasticBuffer, so 0 additionally gives each layer its own arena — multiplying it by the MoE layer count (DeepEP). Rank-uniform, refused at startup when ranks differ. Read once at import, with the other DeepEP knobs, in halo/distributed/expert_parallel/deepep/settings.py.
HALO_GRAD_BUCKET_MB256Flat-buffer size of the bucketed gradient reduction — the deferred cross-replica EP sweep, the TP replicated / per-head-norm sweep, and the QLoRA sweep. (The per-parameter EP grad hooks reduce one tensor at a time and ignore it.) It sets the chunk boundaries every rank must agree on, so it belongs to the rank-uniform set verified in distributed setup (DeepEP). Two buckets run at once, so peak transient is two flat buffers, and under fp32_grad_reduce each also holds an fp32 copy — ~3× the bucket size from bf16. Must be >= 1; a non-positive value warns and keeps the default.
HALO_DEEPEP_GPU_TIMEOUT_SECONDS100Device-side spin budget for the DeepEP dispatch/combine NVLink barrier (DeepEP's own default). The clock starts when a rank enters the barrier, so it bounds rank skew, not idle time between steps — a rank traps if a peer is still this far behind when it arrives. Save paths fence with a host-side barrier first, which absorbs skew under DIST_NCCL_TIMEOUT_MINUTES instead. Raising it also delays how fast a real hang surfaces. Elastic backend only — the legacy V1 buffer takes no timeout and warns when the parsed value is not the default (an explicit 100, or an empty pass-through, is not a request). Rank-uniform, refused at startup when ranks differ. See DeepEP.
HALO_DEEPEP_NUM_SMS / HALO_DEEPEP_NUM_QPSautoPin the DeepEP dispatch/combine SM count (else get_theoretical_num_sms) and RDMA queue-pair count. NUM_SMS applies to both buffer backends (legacy needs an even count); NUM_QPS is elastic-only and also sizes the buffer's QP allocation. 0 means auto, and a negative warns and means auto too. On EFA proxy Gin more QPs can raise the latency-bound internode all-to-all parallelism — A/B them. Rank-uniform, refused at startup when ranks differ; see DeepEP.
HALO_DEEPEP_GIN_MAX_TOKENS_PER_RANK8192Cross-node (Gin) dispatch payload ceiling, tokens/rank. Above the measured ~8k boundary an EFA proxy-GIN dispatch wedges instead of erroring (receive counts never arrive; Xid 109→43), so ParallelismConfig refuses a declared budget above it at config time and the dispatcher re-checks at buffer sizing. 0 disables, and a negative warns and keeps the default; raise only after validating end-to-end on your fabric. Intra-node dispatch is unaffected. Rank-uniform, refused at startup when ranks differ. Read in halo/distributed/expert_parallel/deepep/capacity.py. See DeepEP → AWS EFA.
HALO_PP_FUSED_HEAD_LOSS1Fold lm_head + cross-entropy into the last pipeline stage's forward in token chunks, so the [microbatch, seq, vocab] logits plane never exists. Applies only to the causal-LM pipeline contract; 0 restores the logits path. Read only under pipeline parallelism.
EP_DISABLE_GIN / EP_SUPPRESS_NCCL_CHECKbackend-set / 1DeepEP-owned. The elastic backend sets EP_DISABLE_GIN once per process before its first buffer build (0 across NVLink domains, 1 within one) and honors an explicit value, a blank one reading as unset. EP_SUPPRESS_NCCL_CHECK is latched at import deep_ep, so it must be in the process environment (both images bake it), and suppresses DeepEP's duplicate-NCCL-runtime guard, which the NGC image's HPC-X libnccl-net transport plugin trips. Documented in DeepEP.
HALO_DEEPGEMM_NATIVEoffOpt into the native DeepGEMM forward for every grouped expert GEMM under lowp_precision: fp8|mxfp4; a shape the kernel rejects raises. The wheel is in the Blackwell image; on the Hopper image the flag warns once and stays inert. Whole steps measured at 0.37–0.54× bf16 (Low-Precision MoE).
HALO_LOWP_WEIGHT_CACHE10 disables the per-step cache of each weight's quantized round-trip, re-quantizing it on every forward.
HALO_ALLOW_MOCK_SEARCHoffMakes the fabricated-results mock search backend selectable. Off, naming it raises: mock snippets pay tool_success_reward exactly like a real search, so a training run that reached them would reward invented evidence. For tests and smoke runs.
HALO_SANDBOX_BACKEND / HALO_SANDBOX_URL / HALO_SANDBOX_MAX_CONCURRENCYlocal / unset / usable CPUs (the affinity set, capped by any cgroup CPU quota over the process)RL code-execution sandbox: backend (local / bubblewrap / remote; which ones warn), the SandboxFusion endpoint required by remote, and the process-wide concurrent-execution cap (local/bubblewrap only; no effect under remote). See Sandboxes.
HALO_NCCL_SYNC_TIMEOUT_SECONDS600The deadline for one chunk of a weight sync on both clients: the update RPC that declares it (vLLM answers it only once the chunk arrived) and the drain of its broadcasts (a vLLM packed buffer, or a SGLang chunk's sends, settled two chunks later). On expiry the group is aborted instead of parking the trainer, with a hint naming the transport causes (address, NCCL_NET / plugin mismatch, cuMem). The 120 s warm-up and the 30 s failure-path cleanup keep their own deadlines. Raise for large policies over sockets.
HALO_WEIGHT_SYNC_FORMATION_TIMEOUT_SECONDS120The deadline for forming one weight-sync group: the engine's join request and the trainer's rendezvous, per server. Raise it for engines that load slowly or span many workers; rank 0 forms the groups server by server while its peers wait over the c10d store, so the wait is not bounded by the NCCL watchdog (Rollout Servers).
HALO_PREFLIGHT_PARALLELISM16How many rollout servers rank 0 probes at once in the startup preflights (HTTP only; results are read in server order).
HALO_WEIGHT_SYNC_CHUNK_MB1024Budget of one streamed weight-sync chunk, staged on the forwarding rank's sync GPU; a tensor above it is a chunk of its own, and the staging every server's client sends from (SGLang's arena, vLLM's packed buffers) is one chunk, whatever the number of servers; the arena is at least this size (the vLLM packed buffers close at their own 1 GiB). Each chunk is one engine round trip, which bounds the vLLM rate (measured curve).
HALO_VLLM_GENERATION_TIMEOUT_SECONDS80% of the NCCL watchdog (1440 at the default DIST_NCCL_TIMEOUT_MINUTES)Per-request HTTP timeout (seconds) for a vLLM generation call from TrlVLLMClient, online GRPO only. The call blocks every peer at the next collective, so the default tracks DIST_NCCL_TIMEOUT_MINUTES and stays under it; an override at or above the watchdog is warned about when the client is built. Async GRPO generates from the Ray actors instead, bounded by request_timeout and episode_timeout.
HALO_WEIGHT_SYNC_GATHER_AHEAD1How many EP layers' expert gathers the RL weight sync issues ahead of the layer being forwarded, on a side stream, so the gather overlaps the send; 0 runs gather and send strictly in turn. Each layer ahead holds one more assembled layer on the forwarding GPU. Rank-uniform (agree_rank_uniform_env), as the value sets each rank's collective order (Rollout Servers).
HALO_WEIGHT_SYNC_MEM_LOGoff1 brackets each RL weight sync with a per-rank CUDA-memory line. The post−pre delta isolates the gathers from the training step, and the per-rank spread exposes the forwarding rank's extra full-tensor cost: reserved far above peak_alloc there means stranded allocator pools (Rollout Servers). Off by default: it is one INFO line per rank per sync.
HALO_WEIGHT_SYNC_PARITY_PROBEwarnThe parity probe after a trainer's first weight push compares the engine's log-probs on a fixed counting row with the trainer's: warn logs a mean gap above 1 nat per position, raise stops the run on every rank, off skips it (Rollout Servers → First-push parity probe). Skipped under context parallelism whatever its value. Rank-uniform, refused at startup when ranks differ.
HALO_FP32_MATMUL_PRECISIONhighestfp32 matmul precision applied at model load (highest/high/medium). The default overrides the NGC image's TF32 preference, whose 10-bit mantissa collapses adjacent RoPE token positions past 2048 wherever RoPE is a matmul. transformers' own RoPE multiplies (transformers#49265, in the pinned release), so the pin matters for remote code such as Ling and Ring, and for any other fp32 matmul. bf16 matmuls are unaffected. torch.backends.cudnn.allow_tf32 follows the same knob, so fp32 convolutions (Zaya's Conv1d) match the matmuls. Set high to opt back into TF32 for an fp32-heavy job. Rank-uniform, refused at startup when ranks differ.
HALO_ALLOW_MISSING_CHECKPOINT_KEYSoff1 downgrades the model-load coverage gate to a warning. The gate raises when a checkpoint does not carry a tensor the live model needs — see Checkpoints. A task head added on top of a base checkpoint, a tied lm_head and the class's own _keys_to_ignore_on_load_missing are excused without the flag.
HALO_CHECKPOINT_WRITERS_PER_NODE1Ranks per node that write a gathered EP save on a shared output filesystem. Each writer retains and writes the layers it owns (chunk 0, the non-expert tensors, stays with the save rank; each later layer goes round-robin) under its own part prefix, and the save rank renames the parts and writes the one index, so the saved tensors, keys, dtypes and index size are the same at any count. The default already stripes a multi-node job over its nodes' first ranks, and 0 keeps the save rank the only writer; a larger count adds writers within a node, each holding one gathered layer plus its filling part in host RAM. Capped at the node's ranks. Per-node storage keeps one writer per node, since each node's directory must hold the whole checkpoint. Rank-uniform (agree_rank_uniform_env); a negative count raises on every rank at startup. See Checkpoints.
HALO_CHECKPOINT_WRITE_THREADS0Opt-in background threads that write a streamed checkpoint's finished safetensors parts while the save rank gathers and fills the next: every gathered save (FSDP2, CP, TP, EP, save_model included) and the merge_models, merge_ep_shards, unfuse_moe_experts, reattach_vision_tower and FP8-to-bf16 conversion tools. Each thread holds up to one more part in host RAM, at the writer's shard cap (save_max_shard_size for a save, --max_shard_size for the tools, 5GB by default); the bytes on disk are the same at any count. 0, the default, writes each part in line, as the pipeline-stage writer always does. Rank-uniform (agree_rank_uniform_env): ranks that differ, or a negative count, raise on every rank at startup; each writer reads the count again when it is built, the pipeline-stage writer included. See Checkpoints.
HALO_TORCH_NUM_THREADS1CPU threads per rank, applied with torch.set_num_threads at package import (one process per GPU, so 1 avoids contention); it overrides OMP_NUM_THREADS, PyTorch's own knob. Must be >= 1; a non-positive value warns and keeps the default.
HF_HOME / HF_DATASETS_CACHE~/.cache/huggingface / $HF_HOME/datasetsHugging Face caches; point both at the scratch volume. HF_HOME also anchors the two JIT kernel caches below, and one elected reader per filesystem creates the datasets cache before the dataset load.
TRITON_CACHE_DIR / FLASH_ATTENTION_CUTE_DSL_CACHE_DIR$HF_HOME/triton_cache / $HF_HOME/flash_attn_cute_dsl_cache (the temp dir without HF_HOME, or with a warning where that location is not writable, such as a shared cache mounted read-only; unset, leaving the library's own default, when the temp dir is not writable either)JIT kernel caches, set at script start unless exported with a value, so a --rm container does not recompile every kernel on each launch. FLA's autotune picks live in the Triton cache, and different picks round differently (Docker).
FLASH_ATTENTION_DETERMINISTICunset; 1 under full_determinismRead by transformers, which passes it to the flash kernels as deterministic. parse_script_args sets it as it parses a full_determinism config (HF's Trainer sets it only at construction), so SFT's row plan, made before the run setup, and the loader's resolver both see it. FA4's head_dim-256 kernel is not validated with a deterministic backward, and such a run demotes FA4 to SDPA there, to FA2 on SFT rows that hold several documents (Flash Attention). Compared across ranks with the rank-uniform settings (DeepEP): set on some ranks only, a head_dim-256 model would load a different backend per rank.
CAUSAL_CONV1D_DETERMINISTICunsetRead by causal_conv1d's backward on every call. 1 takes its deterministic path and 0 its default one, which sums the conv weight gradient with atomics, so two identical steps can round apart and a gated-DeltaNet family (Qwen3.5/3.6 MoE, Qwen3-Next, Qwen3.8-Flash-Next) misses an exact replay. Unset or any other value follows torch.use_deterministic_algorithms, which HF's Trainer turns on under full_determinism, so only a run without it needs 1 for a deterministic backward.
CUDA_DEVICE_MAX_CONNECTIONS1 (baked into both images)Driver-owned, latched at deep_ep's cuInit, so a Python write is too late. 1 serializes work submission; it is the setting the EP suites are validated with, at no measurable throughput cost (DeepEP). The trainer reads it only to warn when it is not 1.
PYTORCH_CUDA_ALLOC_CONFunsetPyTorch's caching-allocator settings, set in the launch environment: halo launch passes the shell's value through, and make train takes it as EXTRA_DOCKER_ENV="-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True". The dense SFT, DPO and SMPO examples launch with expandable_segments:True; the EP examples leave it unset, and an EP run sets it only to fit, since there it costs throughput (measured). No image sets it, because one image runs dense and EP training alike; keep it out of a .env that EP runs also read.
VLLM_DISABLE_PYNCCL / VLLM_NCCL_SO_PATHoff / libnccl.so.2Vendored from vLLM. VLLM_NCCL_SO_PATH is honored by the trainer-side weight-sync client; VLLM_DISABLE_PYNCCL set in the trainer process is refused (RuntimeError): it belongs to the vLLM server only, and honoring it would silently disable the weight-sync broadcast. It is parsed the way vLLM parses it (1/true only, not the toolkit's yes/on), so the trainer's verdict matches the server's.
VLLM_USE_V2_MODEL_RUNNER1 (baked into the vLLM image)Read by the vLLM server, not the toolkit. The image pins Model Runner V2, the runner R3 capture starts on and the gpt-oss thinking-budget patches target; compose passes the variable through bare, so unset keeps the pin. 0 selects the V1 runner, which breaks R3 capture and the gpt-oss budget cut (Rollout Servers). 0 or 1 only — vLLM parses it with int(), so false or an empty value kills the server at startup.
VLLM_ENFORCE_STRICT_TOOL_CALLING0 (baked into the vLLM image)Read by the vLLM server. On, the GLM parsers sample tool-bearing requests under a grammar the trainer's log-probs never see, so the startup preflight refuses a server reporting it on a tool-bearing environment.
VLLM_HALO_PATCHESset by the vLLM imageNot a knob: the image's identity patch lists its patches under this name on /server_info, where the trainer refuses a server without it.
NCCL_CUMEM_ENABLE1 in docker-compose.sglang.ymlRead by NCCL in the SGLang server, never the toolkit: SGLang otherwise turns cuMem off process-wide while the trainer's stays on, and the mismatch fails the first cross-container buffer import (why). Keep both ends equal; the compose default does.
VLLM_API_KEYEMPTYOpenAI-compatible API key the scripts/inference/ and scripts/environments/ CLIs, and a judge term on a loopback base_url, send to a self-hosted server, matching the server's own --api-key convention. Falls back to OPENAI_API_KEY, then to the EMPTY placeholder a keyless local server accepts; each CLI's key flag overrides both.
WANDB_PROJECTproject_name argOverwritten unconditionally by the trainer setup before wandb initializes (an exported value does not survive). The run NAME does not travel through WANDB_NAME: transformers passes run_name straight to wandb.init, which wins over the environment, so the setup shortens run_name to its last path segment (else <method>-<mode>-<output_dir basename>, e.g. sft-ep-<run dir>) and writes it back onto the training config instead.
WANDB_RUN_IDmd5(output_dir:launch_time)[:8]Resume identity. An exported value is honored on global rank 0 and broadcast to every rank, so export it on the whole job (or nowhere) and reuse it to continue the same wandb run across restarts.
WANDB_RESUMEunsetConsumed by the wandb SDK, not by this toolkit: set allow alongside a fixed WANDB_RUN_ID to append to that run instead of starting a new one.
TOKENIZERS_PARALLELISMfalseSet with setdefault at package import (halo/__init__.py), before tokenizers loads: the Rust tokenizer's thread pool deadlocks against the dataset-map worker processes that fork around it. An exported value wins, so a single-process tool can opt back into parallel encoding.
CLEARML_PROJECT / CLEARML_TASKproject_name arg / run_name basenameSet by the same trainer setup, same unconditional overwrite, and read by the transformers ClearML integration when report_to includes clearml — which is why the task name IS an env var here and is not one for wandb.

Trainer-to-config quick reference

Each trainer combines a trainer config (hyperparameters) with script arguments (dataset, tokenizer, infra). All configs also accept standard TrainingArguments fields. Every script additionally parses ModelArguments (halo/configs/script_arguments.py: TRL's ModelConfig plus trust_remote_code, which TRL moved onto its own trainer configs, with a one-entry lora_target_modules list kept a list rather than collapsed to a regex string) — model_name_or_path, model_revision, dtype, attn_implementation, trust_remote_code and the LoRA fields (PEFT) — and ParallelismArguments (the ParallelismConfig knobs); all four dataclasses draw from the one YAML.

TrainerConfig ClassScript ArgumentsSource
DistributedSFTTrainerSFTConfig (TRL)SFTScriptArgumentssft.py
SMPOTrainerSMPOConfigSMPOScriptArgumentspreference/smpo.py
DistributedDPOTrainerDPOConfig (TRL)DPOScriptArgumentspreference/dpo.py
DistributedKTOTrainerKTOConfig (TRL)KTOScriptArgumentspreference/kto.py
OfflineGRPOTrainerOfflineGRPOConfigOfflineGRPOScriptArgumentsgrpo/offline.py
DistributedOnlineGRPOTrainer (RLVR)GRPOConfig (TRL)OnlineGRPOScriptArgumentsgrpo/online.py
DistributedSDPGTrainer (online SDPG)GRPOConfig (TRL)OnlineGRPOScriptArguments (--use_sdpg=true)grpo/online.py
DistributedAsyncEnvironmentalGRPOTrainerGRPOConfig (TRL) + EnvironmentConfig + AsyncRolloutConfigEnvironmentalGRPOScriptArgumentsgrpo/environmental.py
DistributedRewardTrainerRewardConfig (TRL)RewardScriptArgumentsreward/reward_model.py
ClassificationTrainerClassificationConfigClassificationScriptArgumentsclassification.py
DistributedTeacherDistillationTrainerTeacherDistillationConfigTeacherDistillScriptArgumentsdistillation/teacher_distill.py
DistributedSelfDistillationTrainer (offline SDPG)SFTConfig (TRL)SelfDistillScriptArgumentsdistillation/self_distill.py
EmbeddingTrainerEmbeddingConfig (extends SentenceTransformerTrainingArguments)EmbeddingScriptArgumentsembedding.py

All distributed trainers parse the ParallelismConfig parameters, in the YAML or on the CLI. A trainer refuses, before any model loads, what it does not support: a context_parallel_size or pipeline_parallel_size above 1 where its class does not declare that axis, the lowp_* knobs outside SFT, and init_from_scratch outside SFT.


TRL trainer configs

These configs come from TRL and are used directly. Only the commonly tuned fields are listed; all extend TrainingArguments.

SFTConfig — DistributedSFTTrainer (TRL docs)

ParameterTypeDefaultDescription
max_lengthint | None1024Total sequence budget. Over-length conversations are dropped by the chat processor; under packing it is the fixed pack size and must be explicit. null → model context window.
packingboolFalsePack multiple sequences into one example.
packing_strategystr"bfd"bfd (best-fit-decreasing, drops overflow), bfd_split (carries overflow into later examples) or wrapped (concatenate-and-chunk, cutting a document at the row edge).
eval_packingbool | NoneNonePacking for eval. Defaults to packing; it can only turn packing off for the eval split (true without packing raises), and a pre-processed dataset refuses it.
padding_freeboolFalseFlatten the batch into one varlen sequence instead of padding it. Needs a varlen Flash kernel (FA2/FA3/FA4) and raises on any other resolved attn_implementation — the cu_seqlens it exists to emit go unread there. See Padding-Free.
dataset_text_fieldstr"text"TRL's pre-formatted text column. Halo SFT renders conversation_field through the chat template and never reads it, so a non-default value raises.
dataset_kwargsdict | NoneNoneTRL's dataset-prep kwargs. Halo SFT replaces TRL's dataset prep with its own, so a user value raises.
chat_template_pathstr | NoneNoneTRL's template override, which its trainer applies after Halo SFT has rendered the rows, so a value raises. Set chat_template (plus force_chat_template: true over a shipped template) instead.
bf16bool | NoneTrueToolkit default True (TRL leaves it None); yields to an explicitly-enabled fp16. Applied before the config is built, so mixed_precision follows it.
use_liger_kernelboolTrueLiger Triton kernels. Toolkit default True (TRL default False).
logging_nan_inf_filterboolFalseToolkit default False (upstream True). Upstream's filter reads the loss scalar back to the host on every micro-batch — gradient_accumulation_steps stalls per optimizer step, and at large world size every rank then waits on the slowest — and replaces a NaN/Inf loss with the running average, hiding a diverged run from its own loss curve.
accelerator_configdict | str | None{non_blocking: true} with pinned batchesToolkit default for one key: non_blocking: true whenever dataloader_pin_memory is not false and use_cpu is off, merged into a dict the config gives and never over a key it sets (a JSON file path is left whole), so a pinned batch copies to the device behind the queued kernels instead of waiting for them (Qwen3-8B CP2 +11.4%, FSDP2 flat; 8× B300, 1.2.0, two rounds).

DPOConfig — DistributedDPOTrainer (TRL docs)

ParameterTypeDefaultDescription
betafloat0.1KL penalty coefficient. Higher = more conservative updates.
loss_typelist[str]["sigmoid"]sigmoid, hinge, ipo, exo_pair, nca_pair, robust, bco_pair, sppo_hard, aot, aot_unpaired, discopop, apo_zero, apo_down, sft, sigmoid_norm (a list — losses can be combined).
max_lengthint | None1024Total budget (prompt + completion). TRL truncates the concatenated sequence keep_start, so an over-long prompt eats its own completion — DPOConfig has no max_prompt_length / max_completion_length. Filter over-long prompts upstream. null → model context window.
label_smoothingfloat0.0Label smoothing for DPO loss.
truncation_modestr"keep_start"keep_start, or the upstream-deprecated keep_end.
disable_dropoutboolTrueDisable dropout.

KTOConfig — DistributedKTOTrainer

Unpaired {prompt, completion, label} preference optimization.

ParameterTypeDefaultDescription
betafloat0.1Deviation allowed from the reference.
desirable_weight / undesirable_weightfloat1.0 / 1.0Re-weight the two classes when the label counts are skewed.
loss_typestr"kto"kto, or apo_zero_unpaired, which carries no KL term.
max_lengthint | None1024Truncates the assembled prompt + completion, keeping the start. null → model context window.

GRPOConfig — DistributedOnlineGRPOTrainer, DistributedAsyncEnvironmentalGRPOTrainer (TRL docs)

ParameterTypeDefaultDescription
num_generationsint | None8Completions per prompt.
num_generations_evalint | NoneNoneEval group size; None = num_generations.
max_completion_lengthint | None512Generation budget (online / vanilla GRPO; TRL max_new_tokens). Set it explicitly. Async GRPO does not use it: it generates with rollout_max_tokens and is bounded by the model context (see Sequence length).
betafloat0.0KL penalty coefficient (0.0 = no reference model). Non-zero on a policy no PEFT adapter wraps (a full fine-tune, or expert-only LoRA), the online and async GRPO scripts load a frozen reference from model_name_or_path, never a resume checkpoint, as the policy loads (run dtype, model_revision, its attention and reset_sinks), and the trainer hands it to TRL in place of the fp32 copy TRL would build. A trainer constructed without one raises, as does one given a ref_model the run never reads.
use_bias_correction_klboolTrueMultiplies the KL term by the importance-sampling ratio, the unbiased reverse-KL gradient. It changes the KL gradient of every online and async GRPO run at beta != 0, on-policy included (the ratio is differentiable even where it equals 1); false keeps the plain k3 term.
temperaturefloat1.0Sampling temperature. Async GRPO overwrites it (with a log) with rollout_temperature, so log-probs are scored at the temperature the rollout sampled at.
top_kint0Top-k sampling (0 = disabled). Online GRPO only; refused with a sequence-level IS mode or vespo as top_p < 1 is.
top_pfloat1.0Top-p sampling. Online GRPO only. With a sequence_* vllm_importance_sampling_mode or loss_type: vespo, top_p < 1 (or a top_k, or a min_p) against a server whose logprobs are renormalized over the cut (vLLM processed_logprobs) is refused at startup (Online GRPO).
num_iterationsint1Policy update iterations per batch (mu-GRPO).
epsilonfloat0.2Clipping epsilon.
loss_typestr"dapo"grpo, dapo (default), bnpo, dr_grpo, plus TRL's cispo, sapo, vespo, luspo. Offline GRPO accepts only grpo, bnpo, dr_grpo. vespo weighs each sequence by the sum of its per-token log IS ratios in place of the per-token multiply: async GRPO refuses it at construction (a token its isr_* stages mask or a forced reasoning close sets the ratio to 0, which drops the whole turn row), and online GRPO's sampler-logprob preflight treats it as summing the ratio over the sequence, as a sequence_* mode does.
scale_rewardsstr"group""group"/True (per-group std), "batch", "none"/False.
mask_truncated_completionsboolFalseDrop completions the engine cut at its token cap from the loss. Async GRPO widens it to any truncated trajectory (turn overflow and a lost episode included) and applies it in its own drop set (its rollout path replaces TRL's generation), masking every row of a truncated trajectory and logging sampling/truncated_masked_frac.
off_policy_mask_thresholdfloat | NoneNoneTRL's off-policy token mask. Refused on async GRPO (raises at construction): TRL's mask reads sampling_per_token_logps, which that trainer's batch never carries, so it would threshold a KL of exactly 0; use isr_opsm_delta instead. Online GRPO refuses it beside balance_token_mass: it drops negative sequences after the balance has weighed them.
use_vllmboolFalseUse vLLM for generation. The toolkit's online GRPO trainer requires true: it raises on false (in-process HF generation is slow and desyncs ranks under FSDP2/EP).
vllm_modestr"colocate"server (external) or colocate (managed). The toolkit's online GRPO trainer supports only server: set it in the config (it raises on colocate).
vllm_server_host / vllm_server_portstr / int"0.0.0.0" / 8000vLLM server address/port.
vllm_group_portint51216NCCL group port of Online GRPO's weight sync. Async GRPO refuses a non-default value at construction: its port is AsyncRolloutConfig.rollout_group_port.
log_completionsboolFalsePrint the rich per-sample completions console table (TRL). Decoupled from the durable parquet record; see save_completions.
save_completionsboolTrueHalo CommonScriptArguments field, not TRL: the durable parquet record, see CommonScriptArguments.
remove_unused_columnsboolFalseAsync GRPO forces it False at construction and warns when a config sets it true: the rollout context is the row's non-prompt columns, and pruning would strip answer and every context_fields column, leaving the environment grading episodes it was handed no ground truth for.

Async GRPO samples on the rollout server, so top_p, top_k, min_p, repetition_penalty and generation_kwargs reach no sampler and are refused at trainer construction when set away from their TRL defaults — use the rollout_* fields on AsyncRolloutConfig. It refuses vllm_importance_sampling_clip_min the same way (its IS correction is token-level and clips from above only), and vllm_importance_sampling_mode at any value but TRL's default or token_truncate, the one that names that correction (Objective). It forces use_vllm: true while that correction is on.

Both on-policy trainers force use_liger_kernel off and refuse the TRL options they cannot honor at construction (Online GRPO → Forced and refused settings).

RewardConfig — DistributedRewardTrainer (TRL docs)

ParameterTypeDefaultDescription
max_lengthint | None1024Total sequence budget, applied as a filter: TRL drops pairs whose chosen or rejected exceeds it, never truncates. null → model context window.
center_rewards_coefficientfloat | NoneNoneReward-centering loss coefficient: penalizes (chosen + rejected)², pulling scores toward zero mean. Helps prevent reward hacking.
disable_dropoutboolTrueDisable dropout.

ParallelismConfig

Unified config for Expert (EP), Context (CP), Tensor (TP), Expert-Tensor (ETP), and Pipeline (PP) Parallelism. See Pipeline Parallelism for the pp_* fields. The class itself is built at construction; users set it through ParallelismArguments (halo/configs/parallelism_arguments.py), whose names go in the YAML or on the CLI and differ from the field names below.

ParallelismConfig fieldParallelismArguments name
ep_sizeexpert_parallel_size
cp_sizecontext_parallel_size
tp_sizetensor_parallel_size
expert_tp_sizeexpert_tensor_parallel_size
pp_sizepipeline_parallel_size
pp_schedulepipeline_schedule
pp_microbatchespipeline_microbatches
pp_splitpipeline_split
ep_fp32_routerfp32_router
ep_fp32_expertsfp32_experts

Every other field below keeps its name, except world_size / gpus_per_node (auto-detected, not settable), expert_lora (built from the PEFT LoraConfig, so it has no flag; row below) and merge_expert_lora_on_save (a ParallelismArguments flag, documented above).

Source: halo/distributed/parallelism_config.py

ParameterTypeDefaultDescription
world_size / gpus_per_nodeint00 = auto-detect from the launcher environment.
ep_sizeint1Expert-parallel dispatch size: the ranks one DeepEP all-to-all spans, each holding num_experts / ep_size experts. >1 enables EP for MoE. With expert_tp_size it forms the ep_group_size group.
cp_sizeint1Context parallel group size. >1 enables sequence splitting.
tp_sizeint1Tensor parallel size. >1 shards model weights.
expert_tp_sizeint1Expert Tensor Parallel size. Shards expert FFN weights within EP groups (MoE-only, experimental).
pp_sizeint1Pipeline stages; each must be a whole number of NVLink domains. The three PP-only knobs below are rejected at pp_size == 1. Outermost axis by design — it carves the world first (stage_world_size = world_size / pp_size). See Pipeline Parallelism.
pp_schedule"1f1b" | "gpipe""1f1b"1f1b (lowest activation memory, needs pp_microbatches >= pp_size) or gpipe (all forwards then all backwards, no microbatch-count constraint).
pp_microbatchesint0Microbatches per optimizer step — the pipeline's own accumulation. 0 = auto: gradient_accumulation_steps, raised to pp_size when 1f1b requires it. Setting it alongside gradient_accumulation_steps > 1 raises (the microbatches are the accumulation); per_device_train_batch_size must divide by it.
pp_splitlist[int] | NoneNonePer-stage decoder-layer counts (one entry per stage, each ≥ 1, summing to the model's layer count, with boundaries on the model's layer_types period). None = the head-weighted default, which shrinks the last stage's budget by the lm_head's layer-equivalent cost.
ep_scope"auto" | "node" | "global""auto"auto resolves to node-local if ep_group_size <= nvlink_domain_size, else global. node keeps groups within each NVLink domain. global spans domains (requires RDMA) and refuses full_determinism there (why).
nvlink_domain_sizeint0Locality unit for node-local EP/CP/TP/ETP grouping. 0 = auto: reads NVLINK_DOMAIN_SIZE, else gpus_per_node. Set to the rack size (e.g. 72) on NVL72/MNNVL. CLI: --nvlink_domain_size.
ep_fp32_routerboolFalseKeep MoE router weights as FP32 masters. The routing matmul runs in FP32 too, except where FSDP2 wraps an ep_group_size==1 router under fsdp_shard_ep1_experts: the router takes its own FSDP2 shard group and computes in BF16 there, while a run FSDP2 does not wrap (one rank, or TP at data_parallel_size == 1) and a frozen router the wrap leaves out of its shard groups keep FP32 (Precision control). Implied by fp32_non_ep_params, whose upcast skips the router: a BF16 router beside FP32 dense params would trip FSDP2's uniform-dtype check. Gemma 4 ignores it with a warning, its router sitting outside the EP layer.
ep_fp32_expertsboolFalseStore EP expert weights in FP32 for stable optimizer updates; compute stays BF16 under autocast. Skipped with a warning when FSDP2 manages the experts (ep_group_size==1 at the default fsdp_shard_ep1_experts), where it would do nothing.
fp32_non_ep_paramsboolFalseStore non-expert params in FP32 (compute stays BF16 via autocast). Every training loader then restores every parameter outside the EP-wrapped MoE blocks (shared experts stay in the run dtype) from the checkpoint at FP32 before any parallel wrapper (an FP32 checkpoint keeps its full precision), where without the flag it casts them to the run dtype (Load precision). Implies fp32 reduction for the FSDP2 reduce_dtype only — the EP, TP and QLoRA bucketed reductions read fp32_grad_reduce directly, so set that explicitly to cover them. Refused on an ep_group_size==1 MoE at the default fsdp_shard_ep1_experts (set fsdp_shard_ep1_experts: false, which TP/CP refuse), and under EP for a family whose spec declares supports_fp32_non_ep_params=False (Gemma 4). Outside AdamWBF16 the optimizer steps one group per weight decay, dtype and tensor type (FSDP2 DTensor or plain EP tensor), each fused under optim: adamw_torch_fused and single-tensor otherwise.
fp32_grad_reduceboolFalseReduce gradients across ranks in FP32 while storing BF16 (FSDP2 reduce_dtype=fp32 for dense params, the EP router grad-sync hooks and deferred sweep, the TP replicated-grad sync, the QLoRA adapter all-reduce). Keeps BF16 master + AdamWBF16 (6 B/param); ~2x bandwidth on the reduce. See BF16 Optimizer.
use_grouped_gemmboolTruetorch.nn.functional.grouped_mm for MoE expert compute on SM90+ (no-op on older HW). MoE layers are wrapped even standalone (ep_size=1, no EP), so an MoE under any accelerate launch is rejected at load — launch with torchrun or set false. Inert for dense models (they run under accelerate normally).
fsdp_reshard_after_forwardboolFalseFSDP2 resharding: False = ZeRO-2 analog (SHARD_GRAD_OP, params stay gathered, faster, higher peak memory); True = ZeRO-3 analog (FULL_SHARD, lower peak memory, and each decoder layer its own shard group where ZeRO-2 groups two — Shard groups). Allowed only where ep_group_size==1 (pure DP, pure TP with dp=1, CP, ep_size==1 MoE without expert TP); rejected with ep_size>1, Expert-TP, TP+DP (dp>1), PP or QLoRA — the backward re-gather races the DeepEP combine, or all-gathers TP-sharded DTensors with no sharding strategy, and QLoRA skips FSDP2. See ZeRO-2 vs ZeRO-3.
fsdp_reshard_after_backwardboolTrueFalse keeps params unsharded across a grad-accum window's microsteps (torch set_reshard_after_backward, re-armed for the window's last backward): drops the full-model re-all-gather FSDP2 otherwise issues after every microstep's backward to one per optimizer step — about 4–10% throughput over NVLink, ~15 s × gradient_accumulation_steps per step when the trainer's NCCL runs over TCP sockets. Saves nothing at gradient_accumulation_steps=1. The unsharded params (at the FSDP param dtype) are the ones the ZeRO-2 forward/backward already holds, kept between microsteps and freed at the window's last backward, so peak memory is unchanged. Plain-DP/CP/EP torchrun path only; rejected with fsdp_reshard_after_forward=True, TP, PP or QLoRA. See Data Parallelism.
fsdp_defer_grad_syncboolFalseTrue skips FSDP2's gradient reduce-scatter (and the HSDP all-reduce) on a grad-accum window's microsteps 1..n-1 (torch set_requires_gradient_sync, re-armed for the window's last backward), so each optimizer step reduces once instead of gradient_accumulation_steps times. Costs one unsharded gradient copy per GPU held across the window (2 B/param; 4 B/param when the reduce runs in fp32 — fp32_grad_reduce, fp32_non_ep_params or bf16: false), which raises the peak by that copy less the sharded gradient the default holds anyway. Nothing at gradient_accumulation_steps=1. Every torchrun FSDP2 wrap (DP, HSDP, CP, TP+DP, EP); rejected under PP, TP at data_parallel_size=1, QLoRA, and with fsdp_reshard_after_forward=True (the held unsharded gradient is the state ZeRO-3 shards; defer under ZeRO-2 instead). See Deferred gradient reduce.
fsdp_shard_ep1_expertsboolTrueApplies only when experts are truly replicated — ep_group_size==1 (ep_size==1 and expert_tp_size==1): FSDP-shards the replicated MoE experts so their reduce-scatter becomes the sole gradient sync. Default True frees memory that grows with DP, is grad-equivalent, and is faster than replicated experts (measured); False keeps a full replicated copy per rank, which fp32_non_ep_params and ep_fp32_experts need at ep1. No effect when ep_group_size>1. Composes with fsdp_reshard_after_forward for full ZeRO-3 experts, and works for online / async GRPO (the weight-sync gather full_tensor-s the sharded experts first). False raises at config time under TP or CP (those paths shard ep1 experts unconditionally, so the flag would be ignored) and, under PP, wherever EP layers meet ep_group_size==1 (a MoE stage and a dense stage would deadlock at the gradient norm).
use_hsdpboolFalseHybrid Sharded DP on the standard DP path (pure DP or CP, no EP). Shards non-expert params within each NVLink domain and replicates across domains, so only one gradient all-reduce crosses RDMA per backward (per optimizer step with fsdp_defer_grad_sync). No-op on a single domain. Rejected with any EP (including EP+CP), TP, or Expert-TP — EP already shards over the EP group, so HSDP would be a no-op or race the DeepEP combine — under PP, whose stages cannot restrict its 2-D mesh, and under QLoRA, which skips FSDP2. See HSDP.
lowpLowpSpecLowpSpec()Low-precision compute, built from the lowp_* keys below (lowp_<field> sets lowp.<field>); validated by LowpSpec.validate.
lowp_precision"bf16" | "fp8" | "fp4" | "mxfp4""bf16"Low-precision matmul compute — bf16/fp32 master weights, low-precision GEMM operands; params and checkpoints unchanged. bf16 (off), fp8 (mxfp8), fp4 (nvfp4, most accurate), mxfp4 (fast fp4). All are the block-scale fake-quant QAT oracle, not a speedup: fp4 runs its activation quant eager and stays several times slower than bf16, while fp8 and mxfp4 round-trip activations in one fused kernel (simulated mxfp4 whole steps at 0.93–0.99× bf16 on 8× H200). The native DeepGEMM kernel is opt-in (HALO_DEEPGEMM_NATIVE=1, fp8 and mxfp4 only), measured slower than bf16, and never auto-selected. Applies to dense MLP + MoE expert GEMMs; attention/embeddings/lm_head/norms stay bf16. SFT script only — every other script rejects a non-bf16 lowp_precision and any non-default lowp_apply_* / lowp_keep_*; pipeline parallelism rejects a non-bf16 one too (why). See Low-Precision Kernels.
lowp_apply_dense_mlpboolTrueApply lowp_precision to dense MLP projections (gate/up/down).
lowp_apply_moe_expertsboolTrueApply lowp_precision to MoE expert grouped GEMMs.
lowp_keep_first_blocks / lowp_keep_last_blocksint0Leading/trailing transformer blocks kept in bf16 (NVFP4 recipes keep about the last 15% of blocks).
bf16_optimizerbool | NoneNoneUse AdamWBF16 (bf16 master weights + stochastic rounding, 6 B/param). None = auto (on under bf16 with the default AdamW, off under replicated DDP); True forces it on. False builds the stock fused/foreach AdamW over the params as loaded — under bf16 that is bf16 masters with round-to-nearest updates, not fp32 — refused at optimizer build wherever plain-tensor experts sit beside FSDP2 DTensor params (ep_group_size > 1, or EP-wrapped experts at ep_size==1 with fsdp_shard_ep1_experts: false), since aten._fused_adamw_ cannot mix the two. An ep_size==1 MoE at the default fsdp_shard_ep1_experts is uniformly DTensor and steps fine. The refusal does not apply when fp32_non_ep_params is set, which routes to a per-tensor-type grouped AdamW with fp32 masters on the non-expert params only. See BF16 Optimizer.
ep_lazy_loadingboolTrueLazy safetensors loading for EP, EP+CP, EP+TP, EP+ETP and pure ETP — each rank reads only its expert slice (parallel, low CPU RAM). Covers every MoE family except Cohere2 MoE, across fused and per-expert safetensors layouts — the gate is the family spec's supports_lazy_loading (per-family EP restrictions), not a key list; a checkpoint whose expert layout the loader cannot address, or a non-local one, falls back to sequential CPU-staged loading. TP-only MoE ignores it and always uses from_pretrained + patching; PP ignores false and always loads each stage lazily.
ep_buffer_backend"auto" | "elastic" | "legacy""auto"DeepEP all-to-all transport. auto/elastic: V2 ElasticBuffer (NCCL Gin, cross-node capable). Within an NVLink domain the dispatch has no token ceiling; for an EP group spanning domains the Gin ceiling HALO_DEEPEP_GIN_MAX_TOKENS_PER_RANK (8192 tokens/rank) is checked at load, before any weight is read. legacy: V1 CUDA-IPC Buffer, numerically identical but intranode-only and rejected for any cross-node EP group; pick it for long-context ep8, where elastic at ≥~64k tokens/rank deadlocks its combine against FSDP2. See Transport backend.
max_concurrent_loadingint | NonenullMax ranks loading simultaneously per node. Unset resolves node-width-aware to min(4, max(1, local_world_size // 2)) — 4 on an 8-GPU node, 2 on a 4-GPU tray. Every explicit value is honored verbatim, 4 included: 0 = all-parallel, a value at or above the node width disarms the throttle, 1 is fully sequential for CPU-RAM-constrained machines. Also throttles the per-rank optimizer-shard writes on save and reads on resume. Which loaders it governs, and how each joins its peers: Multi-Node → Model loading.
expert_loraExpertLoraSpec | NoneNoneNative grouped LoRA on EP experts. No CLI flag — it is built from the PEFT LoraConfig and passed at construction so it cannot bypass validation. Carries r / alpha / dropout / projections / use_rslora (r a multiple of 8, the grouped GEMM's stride contract at the bf16 the adapters run in — refused at construction otherwise); knobs it cannot honor (use_dora, lora_target_parameters) are rejected rather than applied to the attention half alone. See PEFT.

Besides these, num_nodes, num_nvlink_domains, ep_group_size, data_parallel_size, stage_world_size, pp_rank, and the stage-local rank coordinates are computed in __post_init__.

Two more carry no ParallelismArguments spelling and are derived from the training config by parallelism_config_from_args (halo/configs/parallelism_arguments.py): ep_rows_per_device (the rows one MoE forward carries per device — the trainer's rows-per-example × per_device_train_batch_size, divided under PP by the microbatch count) and ep_declared_max_length, which the load-time dispatch-ceiling gate multiplies out into a per-rank token budget before any weight is read.

A trainer whose config declares no max_length field stamps neither and leaves the ceiling to the dispatcher's runtime backstop. max_length: null still stamps the rows, as does offline GRPO's internal field, which no key sets; the gate then takes max_prompt_length + max_completion_length where the config bounds both (offline GRPO, or SMPO with both shares set), else the model's context window.

Five knobs are implemented inside the mixin's own FSDP2 wrap and are therefore ignored under accelerate launch (accelerate owns the wrap): use_hsdp, fsdp_reshard_after_forward, fsdp_reshard_after_backward, fsdp_defer_grad_sync, fp32_grad_reduce. Setting any of them on an accelerate launch logs one warning — use torchrun. Accelerate's own no_sync already skips the gradient reduce on a window's non-final microsteps, so fsdp_defer_grad_sync has nothing to add there.


OfflineGRPOConfig

OfflineGRPOTrainer: group-relative policy optimization on pre-computed rewards. Source: halo/configs/grpo_offline.py

ParameterTypeDefaultDescription
max_prompt_lengthint | None512Prompt budget; a longer prompt is left-truncated (the tokens nearest the completion are kept). null = no cap, which pipeline parallelism refuses.
max_completion_lengthint | NoneNoneCompletion budget; a longer stored completion is truncated from the end. null (default) = no cap. Also the loss_type='dr_grpo' loss normalizer, which rejects null rather than normalize by a default nobody chose; pipeline parallelism rejects null too.
kl_betafloat0.0Coefficient of the KL penalty to the reference policy; 0 = no KL term. Full fine-tuning scores the reference once at run start and persists it in reference_logps.pt; PEFT scores its base with adapters disabled; native expert LoRA needs an explicit frozen base (Reference model).
disable_dropoutboolTrueDisable dropouts.
model_init_kwargsdict | NoneNoneOn every entry-script path these are model-config overrides, written onto the loaded config's fields before the load: a key that config does not declare raises, and so does dtype/torch_dtype (the run's precision comes from bf16 / fp32_*). They are model-loading kwargs only where a trainer is constructed programmatically with the model as a path string; beside an already-built model, a non-null value raises.
dataset_num_procint | NoneNoneDataset preprocessing worker processes; unset means the toolkit default (HALO_DATASET_NUM_PROC), not one worker.
advantage_method"z_norm" | "minmax" | "quantile_norm" | "quantile_uniform" | "robust""quantile_norm"How rewards map to advantages: z_norm (z-score), minmax and quantile_uniform (to [-1, 1]), robust (median/IQR, outlier-resistant).
best_completion_emphasisfloat | str0.0Boost for the best completion(s) per group: 0.0 = off, a float > 1.0 = fixed boost (1.5 = +50%), "auto" = std-adaptive (3.0 + 2.0 × std / (1 + std): 3.0 at std 0, rising toward 5.0), under every advantage method. A value in (0.0, 1.0] raises: the consumer applies the factor only above 1.0, so it would be a silent no-op.
min_log_probfloat | None-3.0Min log prob for negative-advantage tokens (numerical stability).
initial_min_log_probfloat | NoneNoneIf set, linearly schedules min_log_prob from here to the final value over training; below 0, and refused with min_log_prob: null. A start near minus the mean CE loss on the SFT data works.
loss_type"grpo" | "bnpo" | "dr_grpo""bnpo"grpo (per-sequence avg then weighted mean across groups), bnpo (global weighted token avg), dr_grpo (normalized by effective batch size + max completion length).
policy_gradient_formulation"prob_weighted" | "reinforce""prob_weighted"prob_weighted: L = -(π(a|s) * advantage), probability-weighted REINFORCE, more conservative offline. reinforce: L = -(log π(a|s) * advantage), equal weight, more aggressive but less stable offline.
drop_degenerate_groupsboolFalseDrop groups whose completions all scored exactly the same reward (or with < 2 completions) at tokenization: their advantages are all 0, so they spend forward compute only to dilute the loss normalizer. Near-ties are kept (the rank methods train them at full scale). Applies to the train and eval splits alike; raises if the whole dataset would be dropped.
use_chunked_grpo_logprobsboolFalseVocab-chunked completion log-probs instead of full [B, T, vocab] logits: the shared ChunkedLogprobsArguments switch; covers the policy and the kl_beta > 0 reference forward. CP always takes the chunked path, as do DeepSeek-V4 and Mistral 4 (Row positions). See Offline GRPO. Off, construction refuses a full-logits plane the device's free memory cannot hold; the check is skipped with max_completion_length unset (the logits wall). Refused under PP, whose pipeline drives its own stages.

SMPOConfig

SMPOTrainer (SMPO) — reference-model-free, margin-based preference optimization. Source: halo/configs/preference.py

The class also overrides four TrainingArguments defaults: learning_rate=1e-6, logging_steps=10, gradient_checkpointing=True, and bf16 (declared bool | None, resolved in __post_init__ to not fp16).

ParameterTypeDefaultDescription
betafloat1.2Temperature controlling loss sensitivity. Higher = stronger preference signal.
target_marginfloat0.35Target margin for the chosen/rejected log-prob ratio; >= 0.
loss_type"sigmoid" | "hinge" | "ipo" | "smooth_lower_bound""smooth_lower_bound"sigmoid (margin violation), hinge (hard margin), ipo (squared), smooth_lower_bound (squared hinge, smooth near boundary; recommended).
chosen_sft_ratiofloat0.8Weight of chosen in the SFT loss component. 1.0 = chosen only, 0.0 = rejected only.
use_margin_scheduleboolTrueLinearly raise target_margin from initial_margin.
initial_marginfloat0.01Starting margin when scheduling; must be below target_margin, and refused off its default with use_margin_schedule off.
lower_clip_percentilefloat | None0.02Clip low token log probs in rejected, in the margin term only (recommended 0.01–0.05); in (0, 0.5]. Refused under PP, which sees a microbatch's tokens only, so a PP run must set this default to null.
upper_clip_percentilefloat | NoneNoneClip high token log probs in chosen, in the margin term only; in [0.5, 1). Refused under PP.
min_log_probfloat | None-2.3Absolute min log-prob threshold for rejected tokens, applied after the percentile clip; < 0.
padding_freeboolFalseFlatten the batch into one varlen sequence instead of padding it. Needs a varlen Flash kernel (FA2/FA3/FA4) and raises on any other — including the sdpa the script defaults to off Blackwell under reset_sinks: true, which is what makes it need an explicit attn_implementation there (SMPO). Incompatible with CP, PP and VLM mode (PP rejected combinations).
allow_cross_document_linear_attentionboolFalseTrain padding_free rows on a model whose linear-attention layers receive no document boundary (Ring's Lightning Attention-2), or whose short convolutions receive none (Inkling); their state then carries from each row into the next. Off, such a run is refused (Document isolation).
max_lengthint | None1024Total budget (prompt + completion), split into the two shares below by resolve_length_budget(). null → model context window.
max_prompt_lengthint | NoneNonePrompt share; longer prompts truncate per truncation_mode. null (default) = half of max_length, so the split scales with a context-resolved budget. VLM prompts are never truncated — one that expands past this raises at collation.
max_completion_lengthint | NoneNoneCompletion share; longer completions truncate from the end. The budget includes the EOS, so a completion that reaches it ends without one, as in offline GRPO. null (default) = the max_length − max_prompt_length remainder. An explicit value must still fit that remainder (else the config raises).
truncation_mode"keep_end" | "keep_start""keep_end"Which end of an over-long prompt to keep (keep_end = the tokens nearest the completion); completions always truncate from the end. Refused off its default on a VLM run, which truncates no prompt.
disable_dropoutboolTrueDisable dropout.
dataset_num_procint | NoneNoneDataset preprocessing worker processes; unset means the toolkit default (HALO_DATASET_NUM_PROC), not one worker.
model_init_kwargsdict | NoneNoneModel-config overrides applied before the load — see OfflineGRPOConfig.

TeacherDistillationConfig

DistributedTeacherDistillationTrainer. Source: halo/configs/distillation.py. The teacher and the conversation-rendering fields live in TeacherDistillScriptArguments below.

ParameterTypeDefaultDescription
distill_loss"kl_divergence" | "reverse_kl" | "mse" | "soft_cross_entropy" | "cosine_similarity" | "jensen_shannon" | "slim""kl_divergence"Teacher-distillation divergence.
distill_temperaturefloat1.0Softmax temperature — forwarded only to the losses that declare it (kl_divergence, reverse_kl, soft_cross_entropy, jensen_shannon, slim). Refused off its default under mse and cosine_similarity, whose definitions take none. Must be > 0.
distill_jsd_betafloat0.5β of the generalized jensen_shannon loss, in [0, 1]: 0 = forward KL, 1 = reverse KL (Generalized JSD). Refused off its default under any other distill_loss.
distill_topkint | NoneNoneScore the loss on the teacher's top-k tokens plus a tail bin, a lower bound on the full-vocab loss that saves no memory (Top-k). An integer >= 1, below the compared vocabulary width; kl_divergence and soft_cross_entropy only. None = the full vocabulary.
distill_alphafloat1.0Weight of distillation vs CLM loss. 1.0 = distillation only. Must be in [0, 1].
apply_hard_labelsboolFalseGate the distillation term per token on the gold token (Teacher Distillation). Refused at trainer construction with distill_loss: slim, which applies its own gold-token weight.
max_lengthint | None2048Maximum tokenized sequence length. Over-length conversations are dropped, not truncated; null → the student's context window.
dataset_num_procint | NoneNoneDataset preprocessing worker processes; unset means the toolkit default (HALO_DATASET_NUM_PROC), not one worker.

TeacherDistillScriptArguments

Source: halo/configs/distillation.py

ParameterTypeDefaultDescription
teacher_modelstr | NoneNoneTeacher model name or path. Required — an unset or blank value raises at parse time, not at model load.
teacher_model_revisionstr | NoneNoneHub revision for the teacher. It is usually a different repo from the student, so it cannot share model_revision (one repo's commit sent to another 404s). None = the teacher repo's main.
conversation_fieldstr"messages"Dataset field with conversations; a null or blank value is refused.
images_fieldstr | NoneNoneVLM only: image column injected into the first user turn (same semantics as the SFT field).
freeze_vision_tower, min_pixels, max_pixelsAs the SFT fields.
system_promptstr | NoneNoneSystem prompt if none in dialogue.
model_supports_system_roleboolTrueIf False, system prompt is prepended to the first user message.
train_on_completions_onlyboolTrueMask prompt tokens, train only on assistant completions (text and VLM paths alike).
assistant_message_templatestr | NoneNoneThe rendered assistant-turn prefix of the model's chat template; required when train_on_completions_only is on (no default fits every template — a missing or mismatched marker raises at startup) and refused when it is off.
interleaved_thinkingboolFalsePass clear_thinking=False to apply_chat_template (GLM-family templates). Text-only: the VLM path raises (no supported VLM template has the switch).

ClassificationConfig

ClassificationTrainer. Source: halo/configs/classification.py

ParameterTypeDefaultDescription
max_lengthint | None1024Truncation length at tokenization (scripts/training/classification.py). A raw text/label dataset passed straight to the trainer is instead tokenized untruncated and filtered to it. Batches pad to the batch longest, not to this. null → model context window.
disable_dropoutboolTrueDisable dropout.
dataset_num_procint | NoneNoneDataset preprocessing worker processes; unset means the toolkit default (HALO_DATASET_NUM_PROC), not one worker.
remove_unused_columnsboolFalsePrune the columns the model's forward does not take, at the dataloader; the trainer tokenizes a raw text/label dataset before that.
loss_type"cross_entropy" | "focal" | "label_smoothing_ce""cross_entropy"label_smoothing_ce is single-label only (softmax CE) and raises on a multi-label dataset — use focal or class_weights there.
focal_gammafloat2.0Focusing parameter (focal loss). Refused off its default unless loss_type="focal".
focal_alphafloat | NoneNonePositive/negative balancing factor for focal loss, applied as alpha_t = alpha*y + (1-alpha)*(1-y). Multi-label (sigmoid) heads only — on a single-label softmax head every element has exactly one target, so a scalar alpha would be a uniform loss rescale rather than balancing, and it is rejected there, as it is unless loss_type="focal". Use class_weights / derive_class_weights for per-class balancing.
label_smoothingfloat0.0Smoothing epsilon (typical 0.05–0.1), which loss_type="label_smoothing_ce" requires non-zero and every other loss_type refuses. Not to be confused with HF's inherited label_smoothing_factor, which this trainer rejects: it builds its own loss and never consults HF's label_smoother.
class_weightslist[float] | NoneNonePer-class weights by label ID, finite and >= 0, one per label (refused otherwise). Applied to every loss_type (CE, focal, label-smoothed CE) and as pos_weight on the multi-label BCE.
derive_class_weightsboolFalseAuto-compute balanced class weights from label counts all-reduced across the world group (presharded/TP-safe). Classes absent from training keep weight 1.0. Mutually exclusive with class_weights — setting both raises, as does using it on a multi-label dataset (pass class_weights explicitly there).
multi_label_thresholdfloat0.5Sigmoid threshold for multi-label predictions; refused off its default on a single-label dataset.
compute_per_class_metricsboolFalseLog per-class precision/recall/F1. Single-label only — refused on a multi-label dataset.
compute_auc_rocboolFalseCompute AUC-ROC (multiclass = one-vs-rest macro).
compute_mccboolTrueCompute Matthews Correlation Coefficient. Single-label only — ignored on a multi-label dataset.

EmbeddingConfig

EmbeddingTrainer; extends sentence-transformers' SentenceTransformerTrainingArguments. Source: halo/configs/embedding.py

ParameterTypeDefaultDescription
loss_type"mnrl" | "cached_mnrl" | "cosent" | "angle" | "cosine_similarity" | "triplet" | "contrastive" | "online_contrastive" | "batch_all_triplet" | "batch_hard_triplet""mnrl"Sentence-transformers loss (Loss functions).
loss_scalefloat20.0Scale / inverse temperature of mnrl, cached_mnrl, cosent and angle; refused off its default under any other loss_type.
cached_mnrl_mini_batch_sizeint32Rows per gradient-cached sub-forward; refused off its default under any loss_type but cached_mnrl.
matryoshka_dimensionslist[int] | NoneNoneMatryoshka truncation dims, integers >= 1.
matryoshka_weightslist[float] | NoneNonePer-dimension weights, finite and >= 0, same length as the dimensions; refused without them.
pooling_mode"mean" | "cls" | "max" | "lasttoken" | "weightedmean" | "mean_sqrt_len_tokens""mean"Pooling over the token embeddings.
normalize_embeddingsboolTrueL2-normalize output embeddings.
max_lengthint | None512Truncation length; null or non-positive → the backbone's context window.
disable_dropoutboolFalseDisable dropout while training.
batch_samplerstr"batch_sampler"batch_sampler, no_duplicates, no_duplicates_hashed or group_by_label (Embedding).
dataloader_drop_lastboolFalseForced True by sentence-transformers on any multi-process launch (Embedding).

EnvironmentConfig

Environment selection, reward terms and turn budget for Async GRPO with Environments. Source: halo/configs/grpo_environmental.py

ParameterTypeDefaultDescription
environment_typestr"react_math"Registry name, case-insensitive (lowercased at parse): react_math, react_search, native_math, native_coding, native_combined, swe, mcp, qa_search, code_contests, codeforces, exam_qa.
rewardslist[dict][{source: environment}]Reward terms, each {source, name?, weight?, exponent?, ...}, summed into the episode reward as weight × score ^ exponent over a score in [0, 1]: environment (the environment's own grade, logged as reward/objective), judge (requirements for a scoring judge, checks for a veto judge whose fired veto check zeroes every positive component), reward_model; the external terms take view (final / full / judge-only digest) and on_error (invalid / neutral). The environment's per-turn and shaping terms add on top. Parsed at config time; forwarded to the environment as reward_terms. YAML-only. See Reward Terms.
max_turnsint | nullnullMax environment turns per episode, an int >= 1. null keeps the environment class's own default (per-environment defaults).
environment_kwargsdict{}Env-specific kwargs passed to the registry factory; a bare header (null) or any other non-mapping is refused. Merged with core settings in to_env_config(). A key the target environment's constructor chain does not bind raises TypeError; max_turns and reward_terms are refused at parse (set the top-level max_turns / rewards, which validate them).

environment_kwargs keys are per environment. The ones every environment takes (turn shaping, tool caps, requires_answer, carry_reasoning, max_length_cutoff_recoveries) are in Common knobs, reasoning_effort and reasoning_effort_profiles in Effort levels, tool_budgets in Native Tool-Use, and each environment's own on its page (Code Contests, QA Benchmarks, MCP, SWE).

YAML

environment_type: qa_search
environment_kwargs:
  search_backend: duckduckgo
  include_python_tools: true

AsyncRolloutConfig

DistributedAsyncEnvironmentalGRPOTrainer: Ray workers, rollout-server connections, weight sync, rollout generation, async prefetch. Source: halo/configs/grpo_environmental.py, where the IS-correction and reasoning-cost fields come from the ISCorrectionArguments and ReasoningCostArguments bundles it mixes in, and the sampling and budget fields from RolloutSamplingArguments and RolloutBudgetArguments in halo/configs/rollout_arguments.py. A field it copies into the actors' RolloutConfig (halo/environments/rollout_config.py) defaults to RolloutConfig's own value. Every numeric field also refuses a bool (true would read as 1) and a non-finite value, beside its own range.

ParameterTypeDefaultDescription
rollout_actor_budgetint256Ray environment actors per node: a local Ray (ray_address unset) splits it over that node's rollout ranks (this // rollout ranks on the node), a shared cluster (ray_address set) treats it as the cluster-wide total and splits it over every rollout rank. A rollout rank is every rank, or each TP/ETP group's leader. It replaces num_rollout_workers (per rank, default 64), which the parser refuses with this remedy: set the budget to the old value times the node's rollout ranks, or keep the old value under ray_address, where it was already the cluster-wide total. A rank gets at least 1 actor, and a split under 4 (or under the round's rows) warns once. >= 1. Async, so not a concurrency limit; size it to the env's blocking per-episode cost: CPU for sandboxed envs (bounded by the sandbox gate), remote latency for search/MCP envs. An actor needs 1 free CPU at placement only (Ray Cluster).
max_concurrent_rolloutsint | NoneNonePer-rank in-flight cap, the throughput throttle; server-pool load is this × rollout ranks ÷ servers per round (under TP/ETP only each group's leader collects). Each round in flight holds its own cap: the prefetch_depth prefetched rounds, and a cold or eval round beside them. Refused below 1 when set; null is the larger of a train round's rows (per_device_train_batch_size × steps_per_generation) and eval_rollout_batch_size, so a round runs in one wave. A value below this rank's actor count is raised to it, and one below a train round warns with the wave count it implies.
eval_rollout_batch_sizeint | NoneNoneRows per rank in one evaluation rollout round; rows, not prompts, since the eval sampler has already repeated each prompt num_generations_eval times. Eval rounds run without prefetch, so a round's wall time is its slowest episode and per_device_eval_batch_size-sized rounds idle the servers between them; size it to what the servers sustain: this × rollout ranks requests are in flight at once, and a turn that decodes slower than request_timeout allows fails its episode, so a whole split in one round can oversubscribe the servers that carry the training rounds fine. Keep it a multiple of num_generations_eval, at most max_concurrent_rollouts (refused at start), with dataloader_drop_last off (refused at construction). A short last round is padded to whole batches; the padding is neither rolled out nor scored (evaluation). It bounds the loader's batch, not eval peak memory: the loss forward still chunks per_device_eval_batch_size rows, but the round's widest completion sets the padded width. Refused below 1 when set; null leaves the round at the eval batch (per_device_eval_batch_size), whose geometry is checked at construction the same way.
ray_addressstr | NoneNoneRay cluster address (head-node:6379). None = a local instance per rank; the actor budget then counts per node, with it set cluster-wide; setup and sizing on Ray Cluster.
rollout_backend"vllm" | "sglang""vllm"Rollout engine for generation and weight sync. sglang supports routing_replay: rollout (serve with --enable-return-routed-experts --moe-runner-backend triton), rejects rollout_max_thinking_tokens at config time and the env's carry_reasoning at trainer construction, refuses the families 0.5.22's loaders cannot take an online update for (SGLangWeightSyncClient.UNSERVABLE_MODEL_TYPES, quoted at trainer construction), and must be served from the NCCL-aligned Dockerfile.sglang image with NCCL_CUMEM_ENABLE=1 (the compose default); see Rollout Servers.
rollout_server_urlstr"http://localhost:8000"Primary rollout-server URL (whichever engine rollout_backend selects) for weight sync + generation.
rollout_connection_timeoutfloat120.0Seconds to wait for the rollout server. Refused: non-finite or <= 0.
rollout_server_configslist[dict] | NoneNoneMulti-server configs (each a dict with url and optional group_port / group_host, the weight-transfer master address the serving node dials back to, and node, the label single_group orders servers by). Any other key, a missing url, a group_port outside 0–65535, and a non-default rollout_server_url beside a non-empty list raise.
rollout_sync_topology"auto" | "per_server" | "single_group""per_server"How the weight-sync groups span the servers: one NCCL group per server, or one over all of them that broadcasts each chunk once; auto picks single_group for two or more servers that report one served model and one set of weight-loading flags and per_server otherwise, logging its choice. A group needs every server to see every server's GPU (One group over every server). single_group refuses a per-entry group_port, differing group_host values and, at formation, local servers beside remote ones with no group_host; node is refused under per_server and auto. Environmental GRPO only: online GRPO talks to one server.
rollout_weight_precisionbf16 | fp8_blockbf16Precision of the weights the rollout servers hold. fp8_block pushes the attention and MLP projections as 128x128 block FP8 (power-of-two scales), for engine memory and SGLang decode at a slower sync than bf16; the server must serve a checkpoint from export_block_fp8.py with the triton MoE backend. Validated families only; SGLang needs SGLANG_ENABLE_JIT_DEEPGEMM=0; refused beside rollout_adapter_sync and QLoRA (FP8 rollouts). Online and environmental GRPO.
rollout_adapter_syncboolfalseServe a LoRA policy as an adapter beside the frozen base: the first sync pushes the base once, each later one loads the adapter alone under a new name that new episodes select. SGLang only, launched with --enable-lora; attention and dense-MLP linears only. A model or server it cannot serve is refused at startup with the reason (Adapter sync). Environmental GRPO only.
rollout_group_portint51216Trainer-side port of the NCCL weight-sync group, on either rollout backend; 0 takes a kernel-assigned port. With several servers it is the base: a rollout_server_configs entry without its own group_port binds rollout_group_port + index, or its own kernel-assigned port when this is 0.
sync_weights_every_n_stepsint1Sync weights every N training steps. Refused: < 1.
rollout_temperaturefloat0.7Rollout generation temperature. Overwrites GRPOConfig.temperature (logged) so log-probs are scored at the sampling temperature; the engine's logprobs must carry it too — vLLM --logprobs-mode processed_logprobs, SGLang's default (leave SGLANG_RETURN_ORIGINAL_LOGPROB unset). The startup probe runs against either engine and refuses raw logprobs. Refused: non-finite or <= 0.
rollout_top_pfloat0.95Rollout top-p. With a sequence-level ratio (the trajectory geometric band isr_geo_band_*, or OPSM's isr_opsm_delta) and a logprob reference renormalized over the cut (vLLM processed_logprobs) it must be 1.0 — or drop the sequence-level ratio, since every uncertain position otherwise reads as drift. Refused outside (0, 1].
rollout_top_kint-1Rollout top-k; -1 = off. Sent on every request, like rollout_min_p and rollout_repetition_penalty: both engines fill an omitted one from the model's generation_config.json. A cut under a sequence-level ratio is refused against a renormalized reference, as for rollout_top_p. Refused: a bool (true would sample greedily), a non-int, 0 (SGLang rejects it) and below -1.
rollout_min_pfloat0.0Rollout min-p; 0.0 = off. A cut under a sequence-level ratio is refused as for rollout_top_k. vLLM rejects every request with min_p > 0 on a server running speculative decoding (--speculative-config, MTP). Refused outside [0, 1].
rollout_repetition_penaltyfloat1.0Rollout repetition penalty; 1.0 = off. The sampling log-probs carry it and the trainer's recomputed ones do not, so a value other than 1.0 is refused at startup under a sequence-level ratio (isr_geo_band_*, isr_opsm_delta). Refused outside (0, 2].
rollout_max_tokensint32768The per-turn generation budget (one /chat/completions call). The multi-turn trajectory accumulates across turns, bounded by rollout_max_episode_tokens where set and by the served model's context window, not truncated (a trajectory exceeding the window fails). Refused: non-finite or <= 0.
rollout_max_episode_tokensint | NoneNoneThe most tokens one episode may sample over all its assistant turns, reasoning and visible output together, recovery turns included; None = only the per-turn cap and max_turns bound it (max_turns × rollout_max_tokens). Enforced by the engine per turn and never stated to the model: a turn's max_tokens is the smaller of its own cap and what the episode has left, its reasoning cap shrinks with it so the turn keeps its answer room (rollout_max_tokens less its reasoning cap, or rollout_max_answer_tokens where smaller), and an episode left with less than that room starts no further turn and ends truncated, priced like a max_turns overflow. Holds on both engines; the eval scripts take it from --training_config. Logged as episode/output_budget_exhausted. Refused: a bool, a non-int, or a value below rollout_max_tokens (one whole turn must fit). See Trajectory length.
rollout_max_thinking_tokensint | NoneNonePer-turn reasoning-token cap for reasoning models (vLLM thinking_token_budget): caps the chain-of-thought, then forces the answer out of the rest of the turn (its answer room). Every effort level runs under the same cap, and it is never stated to the model. None = rollout_max_tokens less rollout_max_answer_tokens caps reasoning where that is set, else nothing does. Needs a reasoning parser on the server (--reasoning-parser qwen3 for Qwen3.x, the openai_gptoss plugin for gpt-oss): without one vLLM answers thinking_token_budget with a 400 on every request, so rollouts error instead of generating. Refused unless an int in [1, rollout_max_tokens): at or above rollout_max_tokens, which bounds the whole turn, the answer room is 0 and every turn is cut mid-reasoning. Rejected under rollout_backend: sglang for every model: the trainer wires neither of SGLang's budget mechanisms, and harmony models have none server-side. It needs the IS correction where rollout_reasoning_end_token resolves (see train_on_sampled_tokens). See Reasoning cap.
rollout_max_answer_tokensint | NoneNoneThe most a turn may generate past its reasoning cap; None = rollout_max_tokens alone bounds the turn. A turn's max_tokens becomes the smaller of rollout_max_tokens and its reasoning cap (rollout_max_thinking_tokens, else rollout_max_tokens less this) plus this, so a turn the engine force-closed keeps this much room and no more; under rollout_max_episode_tokens a turn starts only while the episode has its answer room left (the smaller of this and rollout_max_tokens less the turn's reasoning cap). Never stated to the model. On SGLang, which enforces no reasoning cap, the cap it leaves is dropped from the request and the turn stays bounded by rollout_max_tokens alone. Refused unless an int in [1, rollout_max_tokens): at or above it the bound shrinks no turn. See Answer room.
rollout_reasoning_end_tokenstr"</think>"The string the server's reasoning parser ends reasoning with, encoded as vLLM encodes it (Qwen3.x, GLM-4, Step-3.7 and LFM-2 </think>, Gemma 4 <channel|>, gpt-oss <|end|><|start|>assistant<|channel|>final<|message|>). Wherever a vLLM reasoning cap can bind, the trainer gives a run of its ids ratio 0 where the engine forced it (every one at sampling log-prob 0) (Objective); a marker the tokenizer does not write (no added token in its encoding) is refused at construction where a vLLM cap can bind, since those closes would stay in the loss, and only warns on a soft-only run. Where it is one token, a turn's reasoning is counted as its sampled ids up to and including it (all of them for a turn cut before closing its reasoning), the count episode/thinking_cap_turns reads; under a marker of several tokens the count and the metric are absent.
rollout_stop_tokenslist[str][]Special-token strings that end a turn (resolved to the engine's stop_token_ids; a name the tokenizer does not know raises at trainer construction and in the eval scripts). Set to the model's tool-call terminator so a turn stops when the model emits its call; needed for a terminator that is not an eos (e.g. gpt-oss <|call|> under harmony-disabled serving). Empty = only eos stops a turn.
rollout_chat_template_kwargsdict{}Chat-template variables sent with every rollout request (chat_template_kwargs) and applied to the trainer's own renders, so both sides see one template state. The stock Qwen3.6 template reads preserve_thinking: with the env's carry_reasoning, the carried previous-turn reasoning stays rendered past a user message (a recovery nudge). reasoning_effort is refused here: it is per episode and travels as the request's top-level field (SGLang also gets an exact nested copy, the spelling it lets override that field). A profile key (reasoning_budget, reasoning_price) is refused here too, at trainer construction.
train_on_sampled_tokensboolTrueTrain on the server's actual sampled token ids (each assistant turn a training row sharing the trajectory advantage) instead of re-tokenizing a chat-template re-render, which eliminates re-tokenization mismatch. On vLLM this needs the server's --return-tokens-as-token-ids (passed by docker-compose.vllm.yml); on SGLang the capture is per-request and needs no server flag. Falls back to a single re-tokenized row for any trajectory with an uncaptured turn. Off, the IS correction is disabled. A run whose vLLM reasoning cap can bind (rollout_max_thinking_tokens, or rollout_max_tokens less rollout_max_answer_tokens) and whose rollout_reasoning_end_token resolves is refused at construction without the correction (this and vllm_importance_sampling_correction), since the reasoning closes the engine forces leave the loss only through its ratio.
max_train_row_tokensint | NoneNoneLongest training row (prompt + completion tokens) a training rank takes; None = no cap beyond the context check. A per-turn row over it is left out of the batch while the episode's other turns still train; a whole-trajectory row over it trains as a zero-weight row. Logged as sampling/rows_over_cap_frac: rows left out over rows left out plus rows that train. A memory bound for the training ranks, below the context check (which rejects rows the served model could not have produced). Refused: a bool, a non-int, a value below 1, or one at or below rollout_max_tokens, since a row is prompt + completion and such a cap would leave out every turn that used its budget.
drop_degenerate_groupsboolTrueDrop GRPO groups whose completions all settled the same environment reward (grade, shaping and external scores; the trainer's reasoning price and floor excluded: judged with them, a group tied on everything else would train on its reasoning lengths alone). Restores effective batch size on sparse verifiable rewards; logged as sampling/degenerate_group_frac.
balance_token_massboolFalseScale down the heavier sign of each generation round's advantages so its token-weighted advantage mass (each token weighted by its truncated IS ratio, across all ranks) nets to zero, the negative mass of untrainable turns, which train on a negative advantage only, included. Requires loss_type cispo, dapo or dr_grpo, top_entropy_quantile: 1.0 and no off_policy_mask_threshold (raises otherwise). Logged as advantage/net_token_mass (the trainable rows' share before balancing, also with the knob off), advantage/token_mass_scale and advantage/negative_only_mass (the untrainable rows' share of the trained mass). See Token-mass balance.
zero_gradient_round_limitint | None50Raise after this many train rounds in a row in which no token anywhere carries a policy gradient (every group tied or dropped, or every token masked) while no earlier round of this launch has: raise it for a reward that sparse. Past the first round that trained, a streak this long only warns, as a policy that nearly always or nearly never solves ties for long stretches and still trains. Warns on the first round and logs sampling/zero_gradient_round. Positive; null only logs.
episode_error_rate_limitfloat | None0.5Raise once a train round's world-wide episode/error_rate (episodes that errored or timed out, which train as masked rows) is over this for episode_error_round_limit rounds in a row: the batch is then mostly the episodes that finished fast, a bias toward short answers, or the rollout path is down. Counted in the checkpoint's sidecar, so a resume keeps the streak. Refused outside [0, 1); null = off.
episode_error_round_limitint3Consecutive train rounds over episode_error_rate_limit that raise. Positive.
tool_parser_watch_roundsint | None2On a tool environment, raise after this many rounds hold the template's rendered tool-call opening in an assistant turn's text while the run has parsed no call anywhere (tool parsers). A turn the engine cut is no evidence. Positive; null turns the watch off.
truncation_alarm_ratefloat | None0.25Warn when a rollout round's episode/truncation_rate (the share of its episodes ended by the max_turns cap, the episode output budget, or a cut turn past its recoveries) rises over this, and log episode/truncation_alarm (1 over, 0 under) every round; the warning repeats only after the rate drops back under. Refused outside [0, 1); null = off.
isr_geo_band_min / isr_geo_band_maxfloat | NoneNoneTrajectory geometric-mean band: mask a whole trajectory when exp(mean log-ratio) over its corrected tokens leaves the band. Set both, with 0 < min < 1 < max; one without the other raises at parse, as every isr_* bound does. Requires the IS correction (raises otherwise), like every isr_* knob below. Size it above the vLLM↔trainer numerical floor (the trust-region masks); a tighter band masks every step.
isr_veto_minfloat | NoneNoneCatastrophic-token veto: mask a whole trajectory when any corrected token's raw ratio falls below this. Refused outside (0, 1).
isr_opsm_deltafloat | NoneNoneOff-policy sequence masking: mask negative-advantage trajectories whose |mean log-ratio| exceeds this many nats; positives never masked. Refused: <= 0.
isr_engine_referenceboolFalseRe-score every training row on the rollout engine under the just-synced weights and feed the mask stages the engine's current-vs-sampling log-ratio: the trainer↔engine disagreement drops out, the engine's own decode-vs-prefill disagreement stays in the reference; the IS weight stays trainer-vs-sampling. Logs sampling/engine_logratio_mean, sampling/numerics_logratio_mean, sampling/engine_rescore_coverage, sampling/engine_rescore_miss_frac. Requires the IS correction, rollout_temperature / rollout_top_p of 1.0 and rollout_top_k / rollout_min_p / rollout_repetition_penalty off; vLLM re-scores through the completions prompt_logprobs echo, SGLang through /generate with logprob_start_len. The vLLM server needs headroom for the fp32 vocabulary log-softmax that pass materializes; sizing on Rollout Servers (isr_engine_reference headroom). See the trust-region masks.
isr_engine_rescore_per_serverint128Re-score requests in flight on each rollout server under isr_engine_reference, summed over every rank: each rank keeps at most its share (this ÷ world size, at least one) in flight on each server, so a server's load does not grow with the world size; below the rank count a startup warning names the load each server still takes. Refused with isr_engine_reference off. A request that misses its deadline reads the trainer's log-ratio and counts in sampling/engine_rescore_miss_frac.
skip_update_masked_fracfloat | NoneNoneTrust-region circuit breaker: when more than this fraction of IS-corrected trajectories are fully masked by the stages above, or more than this fraction of corrected tokens (the masked trajectories are the long ones), the round's advantages are zeroed and every optimizer step the generation round feeds is skipped: gradients dropped to None, so Adam's momentum cannot step the weights either; the KL term goes with the round. Logged as sampling/update_skipped with the two fractions (sampling/is_masked_traj_frac, sampling/is_masked_token_frac). Refused outside (0, 1]. Requires the IS correction and at least one mask stage (isr_geo_band/isr_veto/isr_opsm); raises otherwise, since nothing would ever be masked.
score_centering_top_kint | NoneNoneCandidates the SGLang rollout server returns per generated token for score centering (arXiv:2609.20807): each token's gradient subtracts their importance-weighted score-function mean, so on a constant reward the drift a sampler-vs-trainer mismatch causes cancels where the candidates hold nearly all of the sampler's mass on the mismatched tokens (all of it at the full vocabulary). None = off; raise it until score_centering/sampler_head_mass_mean is near 1. Refused at construction, and by the script before its preflights, on rollout_backend: vllm, train_on_sampled_tokens: false, a filtered sampler, use_chunked_grpo_logprobs: false, loss_forward_importance_weights, balance_token_mass or isr_engine_reference; at construction also on num_iterations > 1, misaligned accumulation, beta != 0 or importance_sampling_level other than token. See Score centering.
score_centering_weights"none" | "tis" | "mis""tis"The importance weight of the centered gradient: none, truncated at vllm_importance_sampling_clip_max, or kept inside [score_centering_mis_low, score_centering_mis_high] and else zero. It replaces the token ratio. Refused while score_centering_top_k is unset.
score_centering_mis_low / score_centering_mis_highfloat0.5 / 5.0Bounds of the mis weight on the trainer-to-sampler ratio, 0 < low <= high. Refused at a non-default value unless score_centering_weights is mis.
early_stop_entropy_bandlist[float] | NoneNoneEarly stop: end training once TRL's entropy is outside [low, high] on early_stop_patience readings in a row. That metric counts a micro-batch whose loss tokens were all dropped as 0, so under drops it reads below the policy's entropy. Refused unless 0 <= low < high. See Early stop.
early_stop_logratio_gapfloat | NoneNoneEarly stop: end training once the magnitude of sampling/logratio_mean is above this many nats per token on early_stop_patience readings in a row (one per generation round; a step reusing its round carries none and leaves the count). Refused unless finite and positive; raises without the IS correction, which logs the gap.
early_stop_on_skipped_updatesboolFalseEarly stop: end training once the breaker skipped every update on early_stop_patience readings in a row (one generation round each at logging_steps: 1). Raises without skip_update_masked_frac.
early_stop_patienceint3Breaching readings in a row an early-stop condition needs; a reading is one training log line, the window's mean under logging_steps > 1. Refused below 1, and at a non-default value when no early-stop condition is set.
scale_rewards_std_floorfloat0.0Floor (reward units) on the advantage-scaling std divisor: divide by max(std, floor), bounding degenerate-batch noise amplification. 0 = off. Refused: non-finite or negative, and above 0 with scale_rewards: none, which divides by no std.
routing_replay"none" | "recompute" | "rollout""none"MoE routing replay. recompute (R2) captures each EP MoE layer's top-k selection in the no-grad logprob-recompute pass and replays it in the update/GC forwards, re-deriving gate weights from live router scores, which leaves the trainer-versus-engine gap alone; it raises in a config that never runs that pass (num_iterations: 1, aligned accumulation, IS correction off, beta: 0). rollout (R3, experimental) replays the rollout engine's per-turn routing through both passes, which narrows that gap (measured); it needs train_on_sampled_tokens and a capture-capable server: vLLM 0.32.0 with --enable-return-routed-experts and --moe-backend triton (the backend weight sync needs), or SGLang 0.5.22 with --enable-return-routed-experts and --moe-runner-backend triton (the triton_kernel/flashinfer runners bypass the capture hook). EP-wrapped MoE only; Gemma4 and Zaya are rejected.
packed_microbatch_tokensint | NoneNonePack each training round's rows into micro-batches of up to this many attended tokens per rank, in place of one micro-batch per row; null keeps today's path. The PackedMicrobatchArguments bundle, shared by the environmental and the online script (SDPG excepted). The world runs the same number of micro-batches (at most steps_per_generation, in the window's last slots), and row-mean losses are rescaled so the objective is the per-row one. Refused at construction without use_chunked_grpo_logprobs: true, at per_device_train_batch_size != 1, under a loss_type outside grpo, sapo, dr_grpo, luspo, dapo, cispo, top_entropy_quantile < 1, entropy_coef != 0, length_ordered_microbatches: true (environmental), misaligned accumulation, an evaluation batch other than 1, a family whose rows cross documents, a backend that is not varlen flash, or a budget the EP dispatch cannot carry, and use_sdpg: true (online). See Packed micro-batches.
length_ordered_microbatchesbool | NoneNoneDeal each training round's rows into its micro-batches longest first, every rank alike, in place of TRL's per-rank shuffle, and run the round's no-grad forwards (recompute, KL reference) in the same order. Only the row order changes. Unset = on where that holds: accumulation aligned to the generation round, a loss_type other than bnpo, top_entropy_quantile: 1.0, no entropy bonus, and rows forwarded one per backbone call (use_chunked_grpo_logprobs: true, at per_device_train_batch_size: 1 or under FA4); true raises at construction where it does not; false keeps TRL's shuffle. See Memory and Throughput.
loss_forward_importance_weightsboolFalseTake the importance weight (and the KL tail clamp) from the loss forward's own detached log-probs in place of a separate no-grad pass over the round's rows. true raises at construction on balance_token_mass, an isr_* stage, isr_engine_reference, skip_update_masked_frac, routing_replay: recompute, num_iterations > 1, misaligned accumulation, or a run with no IS correction and beta: 0 (the default false keeps the pass). With an early_stop_logratio_gap the logged log-ratio reads loss tokens only. Composes with packed_microbatch_tokens. See Weight from the loss forward.
episode_timeoutfloat1200.0Deadline (seconds) for one whole rollout episode: generation + tool execution + grading, unlike request_timeout, which bounds a single HTTP call. Pause-aware: the seconds the engines spend paused for a weight sync are credited back, so a sync never times an episode out. A timed-out episode is canceled and counted in episode/error_rate, and a straggler holds its peers at the per-step collective for up to this long. It must stay under the NCCL watchdog (DIST_NCCL_TIMEOUT_MINUTES × 60): a larger value raises when training starts, and ≥80% of it warns. The default sits at two thirds of the 30-min watchdog, leaving ~10 min of margin; raise DIST_NCCL_TIMEOUT_MINUTES before raising it. Refused: non-finite or <= 0.
enable_prefetchbool | NoneNoneRoll the next rounds out while the current one trains (prefetch). Unset = on with two or more rollout servers, off with one; true prefetches on a single server too; false never. The resulting off-policy staleness, prefetch_depth rounds, is corrected by the truncated vLLM importance-sampling ratio built from the batch's sampling logprobs (clamped by TRL's vllm_importance_sampling_clip_max, default 3.0).
prefetch_depthint1Rounds collected ahead of the step that trains them, together on the engines and delivered in submission order: a round trains prefetch_depth rounds after it was submitted, on samples up to that many rounds old (one round per optimizer step at the default steps_per_generation). Raise it when every step misses (async/prefetch_misses) while the servers drain at each round's tail; the next round's episodes fill them. Costs staleness, more rounds in flight across each sync, engine KV for every round in flight, and prefetch_depth cold rounds at the start of a run, each trained twice (Prefetch). Refused: below 1, not an int, or above 1 on a run that does not prefetch (the refusal names enable_prefetch: false, or the opt-in a one-server run left unset).
use_chunked_grpo_logprobsboolFalseCompute completion log-probs from the hidden state + a vocab-chunked softmax instead of full [B, T, vocab] logits, bounding the loss-forward peak by chunk size, not B·T·vocab. For large-vocab models on long trajectories. See Chunked log-probs. Off, construction refuses a full-logits plane the device's free memory cannot hold (the logits wall).
balance_rank_tokensboolFalseDeal each training round's rows across ranks by attended tokens, every rank keeping its row count, in place of the contiguous slice the rollout hands each. The RankBalanceArguments bundle, online only. Refused under TP/ETP, a loss_type outside grpo, sapo, dr_grpo, luspo, dapo, cispo, top_entropy_quantile < 1, entropy_coef != 0, misaligned accumulation, a single rank; a round with image tensors or any per-row list raises. See Rank balance.
log_phase_timesboolFalseLog each optimizer step's wall time by phase as phase_time/* (Phase times). Costs a few CUDA events per micro-step and one all-reduce per log.
reasoning_price_capfloat0.1Cap of the reasoning price per episode, in reward units, so a long trace cannot outweigh the task reward. The price itself is per effort level: a level's profile (environment_kwargs.reasoning_effort_profiles) charges reasoning_price per 1,000 reasoning tokens an episode spends past its reasoning_budget, summed over its turns, -min(reasoning_price_cap, price × (tokens − budget) / 1000) (Price). Finite and > 0; refused at trainer construction at a non-default value while no level the environment draws sets a positive reasoning_price. Logged as reward/reasoning_price.
reasoning_floorfloat0.0Weight of the reasoning under-use floor (0 = off): a baseline episode that earned no objective and reasoned, summed over its turns, under its level's reasoning_budget pays -weight × shortfall / budget; a solve pays nothing. The one length term that pays for more reasoning (Floor). An episode pays it or the price, never both, so keep each below what the environment charges for the decisions it prices (a resubmission, in code contests). Refused at trainer construction while a drawn level sets no reasoning_budget above 0, or under a reward that prices no success above zero (no environment term, or one of weight ≤ 0). Logged as reward/reasoning_floor. Finite and >= 0.
reasoning_terms_in_failed_groupsboolFalseCharge the reasoning price and reasoning_floor in a training group none of whose baseline episodes earned the objective (a positive reward/objective, which a veto zeroes). Off, such a group is charged neither, so the length terms cannot become most of its advantage (Failed groups); eval episodes are always charged. Off with the price on, a reward that prices no success above zero is refused at trainer construction; set while no drawn level prices reasoning and the floor is off, refused as inert.
model_namestr | NoneNoneModel name for /v1/chat/completions. null is filled with model_name_or_path at script start, so a request always names a model.
request_timeoutfloat120.0HTTP timeout per rollout-server request (seconds). Pause-aware like episode_timeout: a request in flight across a weight sync is credited the pause, so a sync never expires it (async/requests_expired_in_sync counts the ones that overran even so). One request is one turn, so size it to at least twice the per-turn cap (the turn's total) divided by the per-sequence decode speed at peak concurrency (Sizing a run); a slower turn fails its episode. Refused: non-finite or <= 0.
max_retriesint3Max retries of one turn's rollout-server request after a transient failure (exponential backoff); the turn is re-issued, never the episode. It also caps the re-issues of a turn the engine aborts (Engine aborts). Refused: negative.
retry_base_waitfloat1.0Base wait (seconds) for backoff. Refused: non-finite or negative. The whole retry budget — (max_retries + 1) × request_timeout plus backoff — is only warned about at 80% of the NCCL watchdog, never refused: past it a stuck rollout keeps retrying until the watchdog fires and hangs the per-step barrier instead of giving up.

Script arguments

CommonScriptArguments

Base shared by all training scripts, mixing in TorchProfilerArguments (enable_torch_profiler, profiler_*), CheckpointArguments and ModelLoadArguments (their flags). Source: halo/configs/script_arguments.py

ParameterTypeDefaultDescription
datasetstr | list[str]"/path/to/dataset"An org/name[:config][@split] hub id, an s3:// URI, a save_to_disk directory, or a single .jsonl/.json/.parquet/.arrow/.csv file. A list of any of these for multi-dataset training. A bare org/name is a hub id, so a local directory needs ./ or an absolute path, and an org/name that also names a local directory is refused (Dataset path formats). Placeholder default with no validation; an unset value fails at dataset load.
dataset_ratiofloat | NoneNoneFraction of each dataset to take, in (0, 1] (0 is refused). A YAML list gives one ratio per dataset entry; that form is YAML-only (the field is annotated float so HfArgumentParser can build, so a list cannot be spelled on the CLI). A ratio below 1 is refused on a pre-processed dataset, whose rows were fixed at preparation.
test_sizefloat | NoneNoneTest split proportion (e.g. 0.05). Leave empty to keep the dataset's own split: set on a dataset that already carries a test split (or, with none, a validation split, read as test), the two are concatenated and re-split at this proportion, so a curated split is not preserved. Ignored with a warning on a sharded or pre-processed dataset, whose split was fixed at preparation time.
project_namestrper-scriptwandb/clearml project. __post_init__ refuses null and blank, then replaces the "default-project" sentinel with the script's own (e.g. SFT → sft-tuning).
pad_token / bos_token / eos_tokenstr | NoneNoneOverride special tokens. An eos_token whose id differs from one a module copied from the config when it was built (Qwen3.8-Flash-Next's n-gram embedding) is refused: the export would record the new id while the model computes with the copy.
chat_templatestr | NoneNonePath to a .jinja/.jinja2/.j2 file, or template string. Applied where the tokenizer ships none; one that differs from a shipped template raises without force_chat_template.
force_chat_templateboolFalseReplace the tokenizer's own template with chat_template, which it needs (set alone, it raises). A VLM run's processor takes it too (Chat templates). SFT on a dataset prepared with --chat_template needs it wherever the checkpoint ships a template, or the load raises (Pre-Processing).
added_special_tokenslist[str] | NoneNoneAdditional special tokens, added to the tokenizer's existing extra-special set rather than replacing it (transformers' own default replaces, which would drop every control token the checkpoint shipped from all_special_ids and from the exported tokenizer_config.json). Re-listing a token already present is a no-op. Applied before the pad_token/bos_token/eos_token roles, which read an id back. When the tokenizer then holds more ids than the embedding has rows, the embedding (and an untied head) grows to its length; an adapter run saves those rows with its adapters (PEFT saving). DeepSeek-V4, whose hash routers index gate.tid2eid by token id, refuses that grow.
tokenizer_backend"hf" | "gigatoken""hf"Text→ids backend; gigatoken is an optional extra whose token IDs are verified identical to hf's at startup. Resolved in configure_run_tokenizer, so every training method honors it; embedding rejects it (SentenceTransformer owns tokenization) — see Dataset Pre-Processing.
tools_fieldstr | NoneNoneDataset field with tool definitions passed to apply_chat_template. Reward modeling aliases it onto tools, the one column TRL's RewardTrainer templates. Everywhere the render cannot carry it, it is rejected rather than parsed and dropped: outright by KTO and embedding; by async GRPO, whose rollout schema comes from the environment's own tool registry; and on the vision arms of SMPO, DPO and reward modeling, whose pair render templates without tools=.
unfreeze_layers_patterns / freeze_layers_patternslist[str] | NoneNoneLayer-name patterns to unfreeze / freeze (freeze applied after unfreeze). A pattern that matches nothing raises on either knob — unfreeze would leave the model fully frozen, freeze would leave what it named training. They match different things: unfreeze_layers_patterns is fnmatch against full module names, freeze_layers_patterns against full parameter names. Both are refused on an adapter run (LoRA or native expert LoRA), which trains only the adapters. Under pipeline parallelism each stage holds its own layers re-based to index 0, so a layer-indexed pattern (model.layers.3.*) raises; use index-free ones.
enable_efficiency_metrics / gradient_checkpointing_plan / enable_moe_metrics / moe_balancing / router_balancing_rate / num_full_model_params / enable_torch_profiler / profiler_*——See Performance & balancing flags.
max_consecutive_nonfinite_stepsint3An optimizer step whose global gradient norm is NaN or inf is never applied: its gradients are discarded, weights and optimizer moments stay as they were, and the step counter and LR schedule advance. This many such steps in a row are skipped with a warning (the run logs nonfinite_steps_skipped); the next one raises, as does a non-finite step whose weights are already non-finite. Under AdamWBF16 or Muon the skip happens on the device and the warning or raise follows at the host's late read of the verdict, before the next step's update. 0 raises at the first; negative values are refused at parse time. Under fp16 the guard stands aside only where accelerate's optimizer wrapper steps through the loss scaler (an accelerate-launched DDP or FSDP run), which skips overflow steps itself; on the toolkit's no-wrap path (torchrun FSDP2, EP, TP, ETP, PP) it stays armed. See Trainer Architecture.
report_mfu_diagnosticsboolFalseAdd MFU / S-MFU / achieved-TFLOPS to the headline log. Needs enable_efficiency_metrics (refused without it), whose callback computes them every step regardless; this only controls headline visibility. tokens/s/GPU is the reported throughput metric: MFU's denominator (peak device FLOPS) shifts with GPU, dtype and kernel, and its MoE numerator is ambiguous, so it makes a misleading headline.
save_completionsboolTrueGRPO family (online / async): write per-step completions and trajectories to <output_dir>/completions/ (+ a wandb completions table): completions_<step>.parquet for online GRPO, a completions_<step>/ directory of one rank<NN>.parquet part per rank for async GRPO (Saved trajectories). Text is rendered from detokenized message content. Independent of TRL's console-only log_completions; ignored by non-generating trainers (SFT, offline GRPO).
log_decoded_samplesboolFalseWrite the first few decoded train/eval samples (skip_special_tokens=False) to <output_dir>/log/{train,eval}_sample.txt. Written on the FS-aware save rank (same gate as run.log: global rank 0 on a shared output FS, each node's local rank 0 when DIST_OUTPUT_SHARED_FILESYSTEM — or the umbrella it falls back to — is 0); datasets without input_ids are skipped. KTO, reward modeling and embedding refuse it.

SFTScriptArguments

Source: halo/configs/sft.py

ParameterTypeDefaultDescription
conversation_fieldstr"prompt"Dataset field with conversations; a null or blank value is refused.
images_fieldstr | NoneNoneVLM only: image column (HF Image, single or list) injected into the first user turn — hub datasets that keep images outside the conversation (FineVision/the_cauldron/Docmatix) train without preprocessing. Setting it declares the run VLM on a multimodal checkpoint, before the dataset is probed (declaration rules).
freeze_vision_towerboolFalseFreeze a multimodal checkpoint's vision and audio towers and their projectors, training the text tower on image data. Refused with adapters and on a checkpoint with no tower (Vision-language models). Mixed image and text rows need it only under accelerate-managed FSDP2, or beside a frozen input embedding with gradient checkpointing off (Shard groups).
min_pixels / max_pixelsint | NoneNoneImage run: bounds on the pixels per image, for a processor that resizes by pixel count; others refuse them, as does a pre-processed dataset. Exported with the processor.
system_promptstr | NoneNoneSystem prompt if none in dialogue.
train_on_completions_onlyboolTrueTrain only on assistant completions, masking prompt tokens.
train_on_last_assistant_onlyboolFalseTrain only on the last assistant message. Requires train_on_completions_only=True.
tree_packingboolFalsePack rows that share a prefix as trees: the prefix is stored once and each row's remainder attends it and itself only. Needs packing and tree_share_markers; GPT-OSS, Qwen3 and Qwen3.5/3.6 on FA4 or FA2, GLM-5 Next on SDPA. See Tree Packing.
tree_share_markerslist[str][]Tree packing: rows whose message before the final turn is identical up to and including one of these strings are examined together for a shared prefix.
tree_share_requiresstr | NoneNoneTree packing: a row is shared only if this text occurs before its marker.
tree_full_attention"split" | "gather" | NoneNoneTree packing, full-attention layers: split (FA4, except its head_dim-256 kernel) or gather (FA4 or FA2). Unset takes split where the model's backend runs it, else gather. Refused on GLM-5 Next, whose sparse-attention layers attend in their own forward.
mtp_loss_weightfloat0.0Weight of the MTP head's cross-entropy beside the next-token loss. Above 0 builds the family's released head and trains it; 0 builds nothing, and the other mtp_* fields are refused off their defaults.
mtp_loss_type"hard" | "soft""hard"hard fits the gold token two ahead. soft is Soft-MTP: it keeps the loss value and routes the head's gradient to the trunk's next-token distribution.
mtp_train_scope"joint" | "detached" | "head_only""joint"What the head's term reaches: head, trunk, embedding and output embedding; the head alone; or the head alone over a frozen trunk.
mtp_head_sourcestr | NoneNoneHub id or local directory to read the head from when the model's own is stale or absent. Ignored by a resume from a checkpoint, which keeps its trained head; refused under init_from_scratch.
mtp_min_acceptance_gainfloat | None0.0head_only: the least gain over its first held-out reading a finished fit must show on both acceptance metrics, or the run exits non-zero naming the checkpoint that holds the rejected head. null turns the gate off; refused off head_only.
allow_cross_document_linear_attentionboolFalseTrain packing / padding_free rows on a model whose linear-attention layers receive no document boundary (Ring's Lightning Attention-2), or whose short convolutions receive none (Inkling); their state then carries from each document of a row into the next. Off, such a run is refused; under pipeline parallelism it is refused either way (PP packing). A run making no such row (no packing, padding_free or packed pre-processed corpus) refuses the opt-in. GLM-5 Next's flat rows stay refused: its DSA layers read only the padding mask on the dense backends it runs (Document isolation). Refused under tree_packing, whose plan segments those layers.
generate_eval_examplesboolFalseGenerate text examples during eval.
assistant_message_templatestr | NoneNoneThe rendered assistant-turn prefix of the model's chat template; required when train_on_completions_only is on, refused when it is off.
num_eval_examplesint50Examples to generate during eval; refused off its default with generate_eval_examples off.
model_supports_system_roleboolTrueIf False, system prompt is prepended to the first user message.
interleaved_thinkingboolFalsePass clear_thinking=False to apply_chat_template. Needed for templates with a clear_thinking switch (e.g. GLM) when the rollout backend kept historical assistant reasoning — the template default otherwise strips <think>…</think> from history. Text-only: the VLM path raises.

OnlineGRPOScriptArguments

RLVR Online GRPO. Source: halo/configs/grpo_online.py

ParameterTypeDefaultDescription
max_prompt_lengthint | NoneNonePrompt budget applied as a dataset filter: rows above it are dropped, never truncated. null = no filtering.
prompt_fieldstr"prompt"Field with the prompt (string or conversation list).
answer_fieldstr"answer"Field with the ground-truth answer.
system_promptstr | NoneNoneSystem prompt to prepend.
reasoning_effort"low" | "medium" | "high" | "random" | NoneNoneChat-template reasoning-effort steer: low/medium/high, random (sampled per prompt), or None. Passed via chat_template_kwargs; needs a template that reads reasoning_effort (e.g. gpt-oss harmony).
rewardslist[dict][{source: accuracy}]Reward terms, each {source, name?, weight?, exponent?, ...}; sources accuracy, format, judge (requirements only; one listing checks is refused), reward_model; the external terms take view (final / full / judge-only digest) and on_error (invalid / neutral). One TRL reward function per term. YAML-only. See Reward Terms.
use_chunked_grpo_logprobsboolFalseVocab-chunked completion log-probs instead of full [B, T, vocab] logits, for large-vocab models on long completions. See Online GRPO. Off, construction refuses a full-logits plane the device's free memory cannot hold; the check is skipped with max_completion_length unset (the logits wall).
packed_microbatch_tokensint | NoneNoneAs on the environmental trainer (Packed micro-batches): TRL's old-policy and reference passes and the loss forward run the round's token-budget bins as one flattened varlen call each. Needs use_chunked_grpo_logprobs: true; refused with use_sdpg: true.
loss_forward_importance_weightsboolFalseBuild the vLLM importance-sampling ratio (any vllm_importance_sampling_mode) and the KL tail clamp from the loss forward's detached log-probs in place of TRL's no-grad old-policy pass. true raises at construction on balance_token_mass, num_iterations > 1, misaligned accumulation, or a run with vllm_importance_sampling_correction off. Composes with packed_microbatch_tokens and use_sdpg. See Weight from the loss forward.
log_phase_timesboolFalseLog each optimizer step's wall time by phase as phase_time/* (Phase times). Costs a few CUDA events per micro-step and one all-reduce per log.
use_rlrrboolFalseRLRR relative-reward shaping: advantages come from intra-group rankings instead of the group mean/std. Refused together with scale_rewards_std_floor > 0, which RLRR never divides by, or drop_degenerate_groups, whose raw-reward equality test would mask the all-correct groups RLRR ranks by length.
use_sdpgboolFalseSwap in DistributedSDPGTrainer: the GRPO loss plus a privileged-teacher reverse-KL OPD term. Text-only. See SDPG.
opd_positive_advantage_onlyboolTrueCharge the OPD term on positive-advantage tokens only. false with use_sdpg: false is refused.

The tuning fields behind the two toggles — rlrr_* (RLRRArguments, halo/configs/grpo.py) and the SDPG set (SDPGArguments plus opd_positive_advantage_only) — are refused at a non-default value when their toggle is off rather than ignored; both sets are additionally range-validated whether or not their toggle is on (their bounds: RLRRArguments, SDPGArguments).

It also inherits AdvantageShapingArguments (halo/configs/grpo.py): scale_rewards_std_floor, drop_degenerate_groups and balance_token_mass, with the same semantics as the AsyncRolloutConfig rows, except that drop_degenerate_groups defaults false here (it is true in async GRPO, whose sparse verifiable reward makes dead groups dominate the batch) and judges a group on the total weighted reward, since this arm's terms carry no shaping.

rollout_weight_precision (RolloutWeightPrecisionArguments) is an online field too, as in its AsyncRolloutConfig row.

It inherits GRPOEarlyStopArguments as well: early_stop_entropy_band, early_stop_logratio_gap and early_stop_patience, as in the AsyncRolloutConfig rows, with the gap read from TRL's sampling/sampling_logp_difference/mean (refused without vllm_importance_sampling_correction). The skipped-update condition is async-only: this trainer has no trust-region breaker.

RLRRArguments

The rlrr_* tunables of RLRR relative-reward shaping (arXiv:2601.23058), read under use_rlrr: true. Source: halo/configs/grpo.py

ParameterTypeDefaultDescription
rlrr_mode"hrr" | "prr""hrr"Hybrid rank correction (Eq. 3) or pure relative (Eq. 4).
rlrr_taufloat0.1HRR rank-correction magnitude; finite, > 0. Too high dilutes the rule reward.
rlrr_lambdafloat2048.0Length-bin granularity for re-ranking: correct responses bucket by floor(len / λ); finite, > 0.
rlrr_xi_pos / rlrr_xi_negfloat1.0e-3 / -1.0e-3Advantage cap for incorrect responses and floor for correct ones (Eq. 5 clip); xi_neg > xi_pos is refused.
rlrr_std_normalizeboolFalseDivide the centered advantage by the group std; false keeps the Dr. GRPO normalizer.
rlrr_length_rerankboolTrueThe length-bin tie-break in hierarchical re-ranking (Eq. 6).
rlrr_correctness_clipboolTrueCorrectness-aware advantage clipping (Eq. 5); disable for pure PRR with no gold labels.
rlrr_correctness_thresholdfloat0.5A response is correct iff its raw reward is at least this, the only correctness signal.

SDPGArguments

The privileged-teacher OPD term, shared by online SDPG (on OnlineGRPOScriptArguments, read only under use_sdpg: true) and self-distillation. Source: halo/configs/distillation.py

ParameterTypeDefaultDescription
sdpg_hint_templatestr"\n[Hint] The correct answer is: {answer}. Do NOT state that you were given the answer.\n"str.format template the teacher forward sees after its prompt. {answer} fills on both arms, {solution} on self-distillation only; any other placeholder is refused at parse time.
sdpg_loss"reverse_kl" | "forward_kl" | "unnormalized_kl" | "jensen_shannon""reverse_kl"OPD loss.
sdpg_jsd_betafloat0.5β of jensen_shannon, in [0, 1] (Generalized JSD). Refused off its default under any other sdpg_loss.
sdpg_temperaturefloat1.0OPD softmax temperature; finite, > 0.
sdpg_beta_basefloat1.0Base OPD coefficient; finite, >= 0. 0 drops the term.
sdpg_beta_warmup_steps / sdpg_beta_decay_stepsint0 / 0Steps over which beta ramps from 0 to sdpg_beta_base, and the final steps over which it decays back to 0; >= 0.

SelfDistillScriptArguments

The conversation-rendering and vision fields of SFTScriptArguments (no train_on_last_assistant_only, tree_*, allow_cross_document_linear_attention or generation-eval key) plus SDPGArguments and the fields below. Source: halo/configs/distillation.py

ParameterTypeDefaultDescription
sdpg_answer_field / privileged_solution_fieldstr | None"answer" / "solution"Columns filling the hint's {answer} and {solution}; a slot the template names needs its column while sdpg_beta_base > 0. At sdpg_beta_base: 0 the script refuses them off their defaults, as it does confidence_weight_opd, opd_exclude_eos and the sdpg_* knobs (sdpg_temperature stays legal beside reference_kl_coef > 0, whose anchor reads it).
opd_exclude_eosboolTrueDrop EOS/stop tokens from OPD but not from SFT.
reference_kl_coeffloat0.0Alpha on a frozen-reference KL anchor; finite, >= 0. 0 loads no reference.
reference_kl_loss"unnormalized_kl" | "reverse_kl" | "forward_kl""unnormalized_kl"The anchor's divergence, evaluated at sdpg_temperature.
reference_model_name_or_pathstr | NoneNoneThe anchor model; None = the student's init weights.
confidence_field / confidence_powerstr | None / floatNone / 4.0Per-sample weight conf**p from this column, each confidence in [0, 1]; p finite, > 0.
confidence_weight_opdboolTrueApply that weight to OPD as well as SFT.

EnvironmentalGRPOScriptArguments

Source: halo/configs/grpo_environmental.py

ParameterTypeDefaultDescription
max_prompt_lengthint | NoneNonePrompt budget applied as a dataset filter: rows above it are dropped, never truncated. null = no filtering.
prompt_fieldstr"prompt"Field with the prompt or question.
answer_fieldstr | None"answer"Field with the expected answer, carried into the row as answer when the column exists. Whether an answer is needed is the environment's requires_answer; a value other than the default must name a real column or startup raises.
context_fieldslist[str] | NoneNoneAdditional fields passed as context to the environment.

There is no system_prompt: the environment builds the rollout conversation from its own system prompt plus the dataset's user turn, so a config setting the key raises; set it with environment_kwargs: {system_prompt: ...} (Async GRPO). The ReAct presets (react_math, react_search) hardcode theirs and refuse the kwarg.

Other script arguments

These extend CommonScriptArguments with minimal additions:

ClassSourceExtra Fields
SMPOScriptArgumentshalo/configs/preference.pygenerate_eval_examples (True), num_eval_examples (50), and the vision fields freeze_vision_tower, min_pixels, max_pixels, as on DPO, KTO and reward below
DPOScriptArgumentshalo/configs/preference.pygenerate_eval_examples (True), num_eval_examples (50), generation_max_prompt_length (512) — bounds the eval-time generation dataset only, refused off its default with generate_eval_examples off; DPO has no training-side prompt cap — and images_field (None) — VLM only: image column renamed to the images spelling TRL's vision probe reads, ahead of the dispatch
OfflineGRPOScriptArgumentshalo/configs/grpo_offline.pygenerate_eval_examples (True), num_eval_examples (50)
KTOScriptArgumentshalo/configs/preference.pycompletion_field ("completion"), label_field ("label") — KTO uses an unpaired {prompt, completion, label} dataset — and images_field (None) — VLM only: renamed to TRL's images spelling; the vision path refuses precompute_ref_log_probs and paired columns
RewardScriptArgumentshalo/configs/reward.pyimages_field (None) — VLM only: image column, renamed to images and merged into the shared prompt conversation; declares the run VLM on a score-headed multimodal checkpoint (reward VLMs)
ClassificationScriptArgumentshalo/configs/classification.pytext_field (None) — raw-text column wrapped as one user turn when the dataset has no prompt conversation (e.g. text/label sets like imdb)
EmbeddingScriptArgumentshalo/configs/embedding.pyNone. The script refuses inherited knobs it has no stage for: pad_token, bos_token, eos_token, chat_template, force_chat_template, added_special_tokens, unfreeze_layers_patterns, freeze_layers_patterns, tools_field, log_decoded_samples, text_only_model and a non-default tokenizer_backend

YAML shape

A config is the method's fields plus any TrainingArguments field. bf16: true, use_liger_kernel: true, logging_nan_inf_filter: false and, with pinned batches, accelerator_config.non_blocking: true are toolkit defaults and need not be written.

YAML

model_name_or_path: "meta-llama/Llama-3.1-8B-Instruct"
dataset: "s3://bucket/sft-dataset"
conversation_field: "conversation"
train_on_completions_only: true
assistant_message_template: "<|start_header_id|>assistant<|end_header_id|>\n\n"
max_length: 16000
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 0.00004
gradient_checkpointing: true
output_dir: "checkpoints/sft-run"

Working configs for every method and model family live under examples/<method>/<family>/; each method page links its own.

Parsing

HaloArgumentParser (halo/configs/parser.py) reads the YAML (argv[0] when it ends in .yaml/.yml) or a flags-only command line, casts the --key=value overrides, applies the toolkit defaults above, then builds each config dataclass once from the merged values. Every __post_init__ (transformers', TRL's, the toolkit's) therefore derives its state from, and validates, the final configuration: --gradient_accumulation_steps=1 over a YAML's 8 also re-derives GRPO's steps_per_generation and generation_batch_size.

  • A toolkit default applies only where neither the YAML nor the CLI wrote the field, and reaches every config declaring it before that config is built, so TrainingArguments derives mixed_precision from bf16. The bf16 default yields to an explicitly-enabled fp16.
  • Every key no config declares raises and names the field. Nothing is migrated or stripped, ecosystem spellings included.
  • A Literal[...] field rejects an out-of-set value (advantage_method), in YAML and on the CLI alike. Mixed unions (float | Literal["auto"]) are not validated.
  • A string on a bool field is rejected: YAML 1.2 reads the 1.1 spellings yes/no/on/off as truthy strings, which would silently invert packing: no.

Every training script parses through parse_script_args (halo/entrypoints/script.py), which adds four effects around the pure parser:

  • init_distributed() runs before any dataclass is built. Constructing TrainingArguments touches self.device, which would let accelerate create the default process group without device_id; that binding lost costs a new_group per mesh dimension instead of one ncclCommSplit.
  • output_dir expands strftime directives (checkpoints/sft-%Y-%m-%dT%H-%M-%S) from one timestamp, broadcast from rank 0 so every node agrees. %% collapses to %; any other % (sft-100%-data) survives byte-identical. No other field expands them.
  • The run.log tee into that output_dir starts (Launch Recipes).
  • A full_determinism config turns transformers' FLASH_ATTENTION_DETERMINISTIC on before anything picks an attention backend: SFT plans its rows before the run setup, and HF's Trainer sets the variable only at construction, after the load. plan_row_attention refuses a full_determinism config the switch has not reached, such as one built without parse_script_args.

CLI overrides

--param=value overrides any field, dashed spellings normalized to the underscore name. The = is required: --key value raises "CLI overrides must be in --key=value form". A repeated override (--learning_rate=1e-5 --learning_rate=2e-5) raises instead of keeping the last. An override matching no field of the script's config dataclasses raises, as an unknown YAML key does.

  • --field=None, --field=null and --field=none clear an Optional field, container unions included. Three carve-outs: a Literal whose choices include "none" gets the string (--moe_balancing=none); a field whose default spells "no value" as a string of its own gets that string (--report_to=none reaches transformers as "none", the way to silence logging on a smoke run, where None would become [None], which the Trainer refuses); an optional bool refuses the spelling (--bf16=none raises).
  • A field with no confident string cast stays in the YAML: dict-typed fields, lists of containers (rollout_server_configs: list[dict]), and unions with a container member (dataset: str | list[str], --report_to=wandb).
  • Without a YAML the whole config is argparse flags: --key value works, an unknown flag is a usage error, each flag reaches every config dataclass that declares it (pad_token sits on the script args and on TRL's config), and an unset field keeps each dataclass's own default.

Source on GitHubUpdated 18 November 2026