Configuration Reference
Every training script takes a YAML config file as its first positional argument. Config classes extend HuggingFace TrainingArguments, so all its parameters (learning rate, batch size, scheduler, …) are valid everywhere. Any field can be overridden on the command line; CLI wins, and every config is built from the merged values (CLI overrides). Every numeric knob the toolkit declares refuses a bool and a NaN or infinity at parse time, and an integer knob a fraction (ParallelismArguments checks only the type; ParallelismConfig owns the ranges):
Bash
python scripts/training/sft.py examples/sft/qwen3/qwen3-4b-ultrachat.yaml \
--learning_rate=0.00001 --num_train_epochs=2 --per_device_train_batch_size=2A guard's refusal names the config class and the field in one form, <Config>: <field> must be [an integer ]<bounds>[ (<why>)], got <value> (halo/config_validation.py).
Working configs per method and model family live under examples/.
Sequence length: caps vs generation budgets
Three knob names carry the length settings. They mean the same thing everywhere; only the mechanism that enforces them differs per trainer, because some data can be safely truncated and some cannot.
| Knob | Meaning |
|---|---|
max_length | Total tokenized sequence budget (prompt + completion). null or any non-positive value resolves to the model's context window at launch (resolve_length_to_context, which reads max_position_embeddings / max_seq_length / n_positions off the text sub-config so composite VLM configs resolve too, then falls back to tokenizer.model_max_length). Also becomes tokenizer.model_max_length for the run. |
max_prompt_length | Prompt share of the budget. |
max_completion_length | Completion share of the budget, or — under RL — the number of tokens the policy may generate. Async GRPO with Environments' entry script pins it to rollout_max_tokens, the generation cap there, and raises on any value other than TRL's default or rollout_max_tokens. |
Omitting a field falls back to the dataclass default in the tables below. Writing max_length: null opts into the model's context window; null on max_prompt_length / max_completion_length means whatever the per-trainer row below says (no cap, no prompt filtering, or a derived share of max_length) — never the context window.
Per-trainer mechanism
| Trainer (config) | max_length | max_prompt_length | max_completion_length |
|---|---|---|---|
SFT (SFTConfig) | drop: the chat processor tokenizes untruncated and discards any conversation over the budget (cutting one mid-turn corrupts it). Under packing it is also the fixed pack size, which must be explicit — the script raises on null | — | — |
DPO (DPOConfig) | TRL truncates the concatenated prompt + completion keep_start | — (no training-side cap; see below) | — |
KTO (KTOConfig) | TRL truncates the assembled prompt + completion to it, keeping the start | — | — |
SMPO (SMPOConfig) | total budget, split into the two shares below (resolve_length_budget) | truncate per truncation_mode; null → half of max_length | truncate (EOS only within the budget); null → the max_length − max_prompt_length remainder. The two shares must fit inside max_length |
Reward (RewardConfig) | TRL filters: pairs whose chosen or rejected exceeds it are dropped, not truncated | — | — |
Classification (ClassificationConfig) | truncate at tokenization on the script path; a dataset handed straight to the trainer as raw text/label is tokenized untruncated and filtered instead | — | — |
Distillation (TeacherDistillationConfig) | over-length conversations are dropped, not truncated | — | — |
Embedding (EmbeddingConfig) | installed as SentenceTransformers' max_seq_length (truncates) | — | — |
Offline GRPO (OfflineGRPOConfig) | not a key: the budget is the two shares, and pipeline parallelism pads every row to their sum | left-truncate (keep the tokens nearest the completion) | truncate the stored completion (EOS only within the budget); also the dr_grpo loss normalizer |
Online GRPO — RLVR (GRPOConfig + script args) | — | dataset filter: over-length prompts are dropped, never truncated | generation budget (TRL max_new_tokens / vLLM SamplingParams) |
Async GRPO (AsyncRolloutConfig + script args) | — | dataset filter (as above) | not a knob — the script pins it to rollout_max_tokens and raises on any other explicit value; TRL reads it only as the dr_grpo normalizer |
Two knobs are not a plain cap:
- DPO has no training-side prompt cap. TRL truncates the concatenated sequence
keep_start, so an over-long prompt eats its own completion — filter them in the dataset.DPOScriptArguments.generation_max_prompt_lengthbounds only the eval-time generation dataset. - Async GRPO never truncates a trajectory.
rollout_max_tokensis the per-turn generation budget androllout_max_episode_tokensthe episode's (unset by default; an episode that runs it down ends truncated after a whole turn, never mid-row); the multi-turn trajectory accumulates across turns and is bounded by the served model's context window — one that exceeds it fails (trajectory length).
The tokenizer's window
Every training script but embedding routes its tokenizer.model_max_length write through one of three seams in halo/entrypoints/model_setup.py, each pinning its window through apply_tokenizer_window:
apply_max_length— every single-budget script (SFT, DPO, KTO, SMPO, reward, classification, distillation). It resolvesmax_lengthagainst the context window, writes the resolved number back onto the config, and pins the same number, so collators, length filters and packers all read one already-resolved value.apply_prompt_completion_window— offline and online GRPO, which have no singlemax_length. The window ismax_prompt_length + max_completion_length, pinned only when both halves are bounded.HF resolves any
truncation=True, max_length=Nonecall againstmodel_max_length, so pinning a partial sum would turn the unbounded half into a silent cap at the other half's budget. An unbounded half therefore leaves the tokenizer at its own value. On the on-policy scripts the completion half is a generation budget with no unbounded setting; an unset one is rejected here.apply_tokenizer_windowwithresolve_model_context_window— Async GRPO, whose multi-turn trajectory no configured length bounds (rollout_max_episode_tokenscaps the generated share alone). It pins the raw context window, the same hard limit the rollout server enforces, which the trainer fails against rather than truncating.
Boundedness is one predicate everywhere (is_bounded_length): a positive int bounds, while null and any non-positive value mean unset.
The pin is run-scoped and never exported. save_pretrained writes the live model_max_length into tokenizer_config.json, so an unrestored pin would ship the training budget as the served context — a run at max_length: 40000 capping a 262k-context model. configure_run_tokenizer records the tokenizer's own bound and DistributedTrainerMixin.save_model runs every writer under pristine_model_max_length, which serves that bound for the duration of the save and puts the pin back afterwards.
Embedding training pins nothing: its length is installed as SentenceTransformers' max_seq_length. The inference scripts (scripts/inference/) take a raw CLI number and resolve nothing.
The YAML parser migrates no spelling: max_seq_length is a field of no config, so it hits the unknown-key raise (Parsing).
Performance & balancing flags
Every standard training script inherits these from CommonScriptArguments; configure_moe_balancing (halo/callbacks/balancing.py) applies the balancing mode and build_run_callbacks (halo/callbacks/factory.py) builds the observability and balancing callbacks.
| Parameter | Type | Default | Description |
|---|---|---|---|
enable_efficiency_metrics | bool | false | Construct EfficiencyCallback. Off by default: multi-sequence trainers (DPO/SMPO/Reward forward chosen + rejected; Distillation also forwards a teacher) report a misleading utilization. Sets include_num_input_tokens_seen="all" when on — the HF field is tri-state ("no" | "all" | "non_padding"), not a bool. |
gradient_checkpointing_plan | "off" | "auto" | "off" | auto measures step 2 fully checkpointed, then checkpoints only the text decoder layers the memory budget needs; packed SFT rows only: GradientCheckpointingPlanCallback. |
enable_moe_metrics | bool | true | Construct MoEMetricsCallback: moe/load_{max,min,cv}, moe/dead_frac, moe/load_max_{first,last}, emitted every logging_steps. No-op for dense models. It turns output_router_logits on itself only under moe_balancing: aux_loss, so asking for metrics never adds the router-logit tensor or the aux term as a side effect. Not wired under either bias mode, where BalancingBiasCallback emits the same keys from its load counter instead, nor under pipeline parallelism, whose stage forwards return no router_logits (bias_update, on a family whose EP layers export the bias, is the route to load metrics there). It reports nothing under GRPO (backbone-only log-probs) or where the EP layer bypasses HF's router recorder (DeepSeek-V4), warning loudly rather than going silent. |
moe_balancing | "auto" | "none" | "aux_loss" | "bias_update" | "bias_update_transient" | "auto" | Router balancing; auto resolves per model. Resolution rules, per-family support, the trainers aux_loss reaches and the weight-sync downgrade: MoE balancing modes. |
router_balancing_rate | float | 1.0e-3 | Sign-step magnitude (γ) for BalancingBiasCallback when a bias-update mode is active. Refused with an explicit moe_balancing: none or aux_loss; not read where auto resolves to another mode, or on the weight-sync RL scripts. |
num_full_model_params | float | null | null | Total param count across all EP/TP ranks. When set, EfficiencyCallback computes distributed_efficiency = params_ratio * mfu. Refused with enable_efficiency_metrics off. |
enable_torch_profiler | bool | false | Construct TorchProfilerCallback — captures a step window, writes per-rank Chrome trace + flame-graph stacks + memory timeline. |
profiler_output_dir | str | $HALO_DATA_ROOT/profiling/torch | Output dir for torch.profiler artifacts (derives from HALO_DATA_ROOT; set explicitly to override). Like every profiler_* knob, refused off its default with enable_torch_profiler off. |
profiler_wait / profiler_warmup / profiler_active | int | 5 / 1 / 3 | torch.profiler schedule: skip wait, warm up warmup, record active steps (one-shot). |
profiler_ranks | str | "0" | Which global ranks profile: "0", "all", or a comma list like "0,8". |
profiler_record_memory_snapshot | bool | false | Also dump a CUDA memory snapshot (.pickle) over the active window for memory_viz. |
Optimizer selection
optim is HuggingFace's field, extended at import time with two toolkit values (halo/optimizers/registry.py registers everything in NAMED_OPTIMIZER_BUILDERS) so they pass TrainingArguments validation.
optim | Optimizer | Notes |
|---|---|---|
adamw_torch_fused (HF default) / adamw_torch | AdamW, or AdamWBF16 when auto-enabled | With bf16: true this resolves to AdamWBF16 (6 B/param, stochastic rounding) everywhere except an accelerate-managed DDP launch, which is outside the validated SR matrix — set bf16_optimizer: true to opt in there. Mixed dtypes are handled internally: bf16 params take the SR kernel, fp32 params (fp32_router, fp32_experts, fp32_non_ep_params) take standard in-place AdamW. |
muon | Muon | Newton-Schulz orthogonalization on 2-D params; embeddings and the LM head are excluded. |
flash_adamw | FlashAdamW | Quantized optimizer states, ~5 B/param. |
bf16_optimizer: true together with optim: muon or optim: flash_adamw raises — both select an optimizer and the bf16 path would silently win (reject_conflicting_optimizer_choice). Any other explicit optim (e.g. adamw_bnb_8bit) suppresses the AdamWBF16 auto-enable. Pipeline parallelism refuses muon and flash_adamw (reject_pp_named_optimizer): its stage-local optimizer build and per-stage optimizer shards are validated with the AdamW family only (PP rejected combinations).
Model, checkpoint and from-scratch flags
Every training script parses these. reset_sinks, train_sinks and text_only_model (ModelLoadArguments) and the four checkpoint flags (CheckpointArguments) are on every script's CommonScriptArguments (halo/configs/script_arguments.py); init_from_scratch and merge_expert_lora_on_save are ParallelismArguments flags (halo/configs/parallelism_arguments.py, whose parallelism sizes are under ParallelismConfig).
| Parameter | Type | Default | Description |
|---|---|---|---|
init_from_scratch | bool | false | Build the model with random weights from model_name_or_path's config instead of loading pretrained weights. SFT script only (other scripts reject it); dense DP / FSDP2 only — raises for EP/TP/CP/ETP/PP, and for a model holding an external table (Qwen3.8-Flash-Next), which only a checkpoint supplies. See Pre-training. |
reset_sinks | bool | true | GptOss only: neutralize the attention sinks for SFT — dtype.min and frozen (removed under FA2). false keeps the pretrained sinks live and frozen, the on-policy RL setting (the trainer scores with the sinks the rollout engine serves). Live sinks need a sink-carrying attention implementation — FA4 on Blackwell, flex_attention or eager on Hopper; FA2 and SDPA are rejected — and are refused under Context Parallelism. A reference model whose sinks differ from the policy's is rejected, in every parallelism mode, since the KL would be biased on every token: the frozen copy the DPO/KTO and GRPO scripts load takes the policy's reset_sinks and passes, while a reference TRL builds itself keeps its pretrained sinks live (GptOss's default eager attention applies them) and is refused beside neutralized ones. Auto-skipped under init_from_scratch. See GPT-OSS. |
train_sinks | bool | false | GptOss only: keep the live sinks trainable (requires reset_sinks: false; the contradiction is rejected). Full fine-tuning under FA4 (the loader installs an exact sink-less + sigmoid(lse - sink) rescale on both kernel entry points instead of the kernel's own sink backward, which is not validated here) or eager; flex_attention and FA3 are rejected (no usable sink gradient), as are adapter runs, weight-sync RL and a model without sinks. Counts as live for every sink gate. |
text_only_model | bool | false | Load a multimodal checkpoint through its text-only CausalLM class instead of the vision-bearing wrapper (Qwen3.5/3.6: Qwen3_5MoeForCausalLM). Drops the vision tower and, unless mtp_loss_weight trains it, any MTP tail from the build and the export (a text-only artifact: no processor_config.json, no vision token ids). Image-bearing datasets are refused loudly. Honored by SFT, DPO/SMPO/KTO, teacher/self-distillation and offline GRPO. Rejected at startup by online/async GRPO (the weight sync sends model.layers.* names where the served multimodal checkpoint expects model.language_model.layers.*) and by reward, classification and embedding (their model class is pinned). For Qwen3.5/3.6 both pinned engines (vLLM 0.32.0, SGLang 0.5.22) register the text-only class and load the export unchanged; SGLang registers only Qwen3.8-Flash-Next's multimodal class, so run scripts/after_training/reattach_vision_tower.py on that export first (Qwen3.8-Flash-Next). The tool rebuilds the multimodal artifact (Qwen3.5). |
save_sharded_ep | bool | false | Save EP checkpoints as per-rank shards (every rank writes its own slice in parallel) instead of gathering to one rank — write bandwidth, not host memory, since the gathered path streams. The result loads nowhere until merge_ep_shards.py runs; the merged directory resumes exactly, optimizer included, on the save's topology. The default gathered path writes an HF-standard sharded checkpoint (model-XXXXX-of-YYYYY.safetensors + index, save_max_shard_size 5 GB), which resume reads directly. Refused at trainer construction under PP and for every shape the merge cannot reassemble — more than one EP group, ETP, CP, native expert LoRA, a multi-node run on a non-shared output filesystem (full list). |
save_max_shard_size | str | "5GB" | Max size of one safetensors shard written by the distributed gathered (EP/TP/FSDP2/CP) save paths; anything but a positive size like 5GB is refused at parse. HF's own save_pretrained keeps its own default. A sharded EP save writes one file per rank by design and reports that it does not apply; merge_ep_shards.py sizes its output by its own --max_shard_size. |
overwrite_output_dir | bool | false | Allow starting a run in a non-empty output_dir. Off by default so a fresh run cannot interleave its checkpoints with an existing one's. See Checkpoints. |
allow_optimizer_warm_restart | bool | false | Resume only. When restoring optimizer shards that match this run's topology fails on any rank (an unreadable shard, a CUDA OOM while applying it), or the checkpoint holds optimizer state this run cannot restore (an interrupted save's shards with a missing or unreadable optimizer_meta.pt, a replicated optimizer.pt with no shards beside it), resume with a freshly initialized optimizer and a warning instead of raising on every rank; weights, step and LR schedule still resume. Which resume outcomes it covers: Checkpoints. |
merge_expert_lora_on_save | bool | false | Fold the adapter delta into the base on save, so the checkpoint is a fully-merged, servable HF model. Covers both halves of a mixed run: the expert deltas inside each family's gather, and any attention adapters folded out of place into each tensor as it is written (the frozen base is never written, so training continues unchanged). Each training checkpoint also keeps the unmerged adapter in resume_adapter/, which resume restores onto the base; the final export carries none (Merge-on-save checkpoints). Requires native grouped expert adapters — rejected when the run built none, under accelerate-managed FSDP, and with save_sharded_ep: true (merging is only wired through the gathered EP save). false writes a standalone adapter file and leaves the base unchanged. |
Environment variables
Read from the environment (not the YAML); set in the launch command / .env. Toolkit knobs plus the third-party variables a run sets or depends on. A toolkit boolean takes 1/true/yes/on or 0/false/no/off in any case; another value warns once and keeps the default, as a malformed number does. The image-baked CuTe DSL variables are in Docker, the Liger RoPE pin in Liger Kernels, and third-party credentials (WANDB_API_KEY, HF_TOKEN, OPENAI_API_KEY, the web-search backend keys) in Dev Environment.
| Variable | Default | Description |
|---|---|---|
DIST_SHARED_FILESYSTEM | 1 | Umbrella for both sides: 1 = shared FS (only global rank 0 writes downloads/checkpoints); 0 = per-node local disk (each node's local rank 0 writes). Must be identical on every rank — the flags select the coordination scope, so init_distributed() broadcasts rank 0's resolved values for all three and warns any rank whose env disagreed. See Filesystem Handling. |
DIST_INPUT_SHARED_FILESYSTEM | umbrella | Read side only (model/dataset downloads, dataset map/pack, HF caches): picks the fs_aware_main_first coordination scope and is_fs_load_rank(). Falls back to the umbrella while unset, and wins over it once set. Set 0 with a shared umbrella when a multi-node NFS/EFS mount returns stale file handles on cross-node cache reads. Multi-node runs probe HF_HUB_CACHE and HF_DATASETS_CACHE against it at startup and raise on a contradiction in either direction. |
DIST_OUTPUT_SHARED_FILESYSTEM | umbrella | Write side only (checkpoints, run.log, dumped artifacts): picks is_fs_save_rank(). Falls back to the umbrella while unset, and wins over it once set. Non-shared forces save_on_each_node on and rejects a multi-node sharded-EP save (per-rank shards scatter across nodes with no gather path); shared forces it off across nodes, where every node's local rank 0 would write and rotate the same files. Multi-node runs probe the declaration against the filesystem at startup and raise on a contradiction in either direction. |
DIST_NCCL_TIMEOUT_MINUTES | 30 | NCCL collective-watchdog timeout, read as nccl_timeout() (halo/env.py), applied by init_distributed (PyTorch's own default is 10) and pinned as the default for every later dist.new_group(), so EP/CP/TP subgroup collectives inherit it. The one knob for that watchdog: the group starts before the config is parsed, so a non-default ddp_timeout in the config is refused. Does not bound DeepEP V2's dispatch/combine (its own ElasticBuffer barrier over the NCCL Gin backend). Raise for slow cross-node all-to-all, or for a gathered save whose single part (save_max_shard_size) writes slower than this. Must be >= 1 — a non-positive value warns and falls back to the default. Rank-uniform (agree_rank_uniform_env): a node left at a smaller budget aborts its process group first, and the survivors then die on the next collective and blame it. |
DIST_STORE_TIMEOUT_HOURS | 4 | Wall-clock bound for the c10d-store coordination waits, read as store_timeout() (halo/env.py) — fs_aware_main_first (main-first downloads), run_load_rank_first (dataset load, map/filter, packing, TRL dataset preparation), store_reject_across_ranks (the joins behind it, the sharded-load join, the model-source agreement, and the checkpoint joins after the per-rank optimizer and sharded-EP shard writes, a gathered save's last part and index, and each pipeline stage's parts), and throttled_load_within_node (the max_concurrent_loading throttle, joined world-wide by joined_node_load). These wait out ONE rank's work rather than a collective, so they are hours-scale and independent of DIST_NCCL_TIMEOUT_MINUTES. Must be >= 1 (same fallback rule) and rank-uniform — the rank with the smaller budget gives up on a join its peers are still serving. See Filesystem Handling. |
DIST_FS_PROBE_TIMEOUT_SECONDS | 60 | How long a rank declared on shared storage polls for the multi-node filesystem probe's sentinel, read as fs_probe_timeout() (halo/env.py); a per-node declaration never waits. The default is NFS's acdirmax: raise it above a mount's actimeo/acdirmax, or where lookupcache keeps negative lookups, so a shared mount is not refused as unshared. Must be >= 1 (same fallback rule) and rank-uniform (agree_rank_uniform_env). See Filesystem Handling. |
NVLINK_DOMAIN_SIZE | gpus_per_node | The locality unit for node-local EP/CP/TP/ETP grouping — validation checks NVLink-locality against this, not gpus_per_node. Set to 72 on GB200/GB300 NVL72 to run NVLink-wide across OS nodes; leave unset on ≤8-GPU nodes. Also settable as --nvlink_domain_size. Rank-uniform, refused at startup when ranks differ. See NVL72. |
VLLM_GROUP_HOST | loopback (local server) / the address on the route to the server | Trainer interface advertised to vLLM workers for the NCCL weight-sync group, and the one address its rendezvous store listens on unless HALO_WEIGHT_SYNC_BIND_ALL is set. Set where a remote vLLM cannot dial back to the trainer's address on its route to the server (NAT, asymmetric routing). Resolution order and the multi-homed cases: Online GRPO → Multi-homed nodes (async GRPO also takes a per-server group_host in rollout_server_configs; Online GRPO has only the env var). |
SGLANG_GROUP_HOST | loopback (local server) / the address on the route to the server | Trainer interface advertised to SGLang workers for the NCCL weight-sync group, and the one address its rendezvous store listens on unless HALO_WEIGHT_SYNC_BIND_ALL is set: the VLLM_GROUP_HOST equivalent for rollout_backend: sglang. Separate per engine because the two servers can sit on different hosts. |
HALO_WEIGHT_SYNC_BIND_ALL | off | 1 puts the weight-sync rendezvous store (both engines) on every interface instead of the advertised group address alone, for a trainer the server reaches through NAT or a port mapping; each group formation then logs a WARNING. Off, an advertised address this host cannot bind alone (not local, 0.0.0.0, or a name resolving to loopback for a remote server) raises at group formation. An IPv6 group address raises in either mode: the store listens on IPv4 only. The store is unauthenticated, so set it only where no untrusted host reaches the group port (Group rendezvous). |
HALO_SCRATCH | local mount with the most free space (make) | Host-side scratch root the Makefile, the devcontainer and the vLLM compose file read (never Python): bind-mounted into containers and deriving the in-container HALO_DATA_ROOT. Unset, make takes the local mount other than / with the most free space over 20 GB and stops a target that needs it when none qualifies; the devcontainer refuses to start, and so does every service of the vLLM compose file, vllm-server included (compose interpolates the whole file; set it once in the repo-root .env). |
HALO_DATA_ROOT | ~/.cache/halo | Toolkit scratch root — the single knob for toolkit-owned caches and outputs. The S3 dataset cache (<root>/s3_datasets), profiler artifacts (<root>/profiling: torch traces, memory snapshots, py-spy captures) and the batched-generation resume store (<root>/inference/resume) derive from it. Point it at a verified large volume. HuggingFace caches are governed separately by HF_HOME / HF_DATASETS_CACHE. |
HALO_S3_DATASET_CACHE_DIR | $HALO_DATA_ROOT/s3_datasets | Explicit override for the S3 dataset cache. A rank waits DIST_STORE_TIMEOUT_HOURS for the peer downloading the same entry. |
HALO_S3_DEFAULT_BUCKET | unset | Bucket for the key-only S3 helpers (build_s3_uri, a bucket-less S3Client()): a bare-key --dataset / --output_path of openai_batched_generation.py (a full s3:// URI needs none) and s3.py's --bucket default. A key-only path raises while it is unset. A training dataset: entry always needs a full s3:// URI. |
HALO_S3_MAX_FOLDER_CONCURRENCY | 16 | Files transferred in parallel per S3 folder operation, clamped to >= 1. Each file above boto3's multipart threshold uses up to 5 connections from one 50-connection client pool, so large-file folders exceed the pool at the default (urllib3 warns Connection pool is full); 10 stays inside it. |
HALO_DATASET_NUM_PROC | max(1, min(3/4 × usable CPUs, 64)) | Worker processes for the rank-coordinated dataset.map/filter and sequence packing (the packed rows do not depend on the count). Unset, each host derives it from the CPUs the process may use: its affinity set, capped by the tightest cgroup CPU quota on its cgroup or an ancestor (docker --cpus, a Kubernetes CPU limit, a systemd CPUQuota). The load rank of each input filesystem maps with the whole count while its node's other ranks wait, and they then read its cache files back whatever their own count, so hosts that derive different values still share one cache. Those ranks compute only on a per-rank key (a pre-sharded dataset), all at once, so each maps with its 1/LOCAL_WORLD_SIZE share of the count, at least 1; a count set here or by a config dataset_num_proc (which overrides this) is shared the same way. prepare_dataset.py runs alone, so it defaults to every usable CPU but two instead. Must be >= 1 (below 1 fails loud); 1 disables map multiprocessing. Resolved once at import. The waiting ranks are bounded by DIST_STORE_TIMEOUT_HOURS: raise it for a training-time map of a large corpus, or bake the corpus offline. |
HALO_DSA_MATMUL | on on Blackwell, else off | 0 leaves GLM-5 Next's DSA layers on stock SDPA (mem-efficient at their head width on Blackwell, cuDNN on Hopper) instead of the masked matmul kernel the sdpa slot of modeling_glm5_next runs on CUDA. Read at each load, and not compared across ranks, as the kernel runs no collective and its one Triton launch compiles at load. See GLM-5 Next. |
HALO_FLEX_SLIDING | on | 0 builds an SDPA run of a sliding-window or wide-head model (Step-3.7, Gemma 4) with plain sdpa instead of sdpa_flex_sliding (FlexAttention on sliding-window layers and on packed or padded full layers, matmul or per-document mem-efficient attention on global layers SDPA can only run mem-efficient). Rank-uniform, refused at startup when ranks differ. See Flash Attention. |
HALO_FUSED_GLU | on | 0 runs every GLU combine (expert and dense MLP) through its eager form on CUDA instead of the fused Triton kernels, and each EP layer's construction line then reads glu_combine=eager. The switch for a GPU where a GLU kernel fails to compile or launch (Troubleshooting). |
HALO_HC_KERNELS | on | 0 leaves upstream's hyper-connection and decoder-layer forwards on GLM-5 Next and DeepSeek-V4 instead of the hyper-connection kernels: the A/B, and the switch for a GPU where one fails to build. Read when the family's patches apply, at load. |
HALO_QSA_KERNEL | auto | auto, on or off: whether Qwen3.8-Flash-Next's QSA layers attend on the block-sparse kernel or on the tiled SDPA path. on raises where the kernel cannot run (not CUDA, a dtype other than bf16 or fp16, a head width off a power of two, full_determinism), auto takes the tiled path there and logs the reason once on rank 0; a layer whose kernel fails to build at load (Triton OutOfResources, a compile error) takes the same path under auto and raises under on. Read per call, and at load by the kernel warm-up. |
HALO_QSA_INDEXER_MIB | 256 | Memory budget of one chunk of the QSA indexer's batched block scores (fp32 products, their ReLU and the scores); a larger budget scores more queries per matmul. It does not change the selected blocks. |
HALO_GATED_RESIDUAL_KERNELS | on | 1 runs both gated residuals of every Qwen3.8-Flash-Next layer, and the injection of each block's output into the four streams, on the gated-residual kernels where a module can (CUDA bf16, fp16 or fp32 streams, plain hook-free mix projections in the streams' dtype), else on upstream's forward, logging the reason once on rank 0; trained and frozen weights both run on them. 0 runs upstream's forward. Read per call, and at load by the kernel warm-up, so the launch must set it on every rank. |
HALO_FLA_L2NORM_FIXED_NB | off | 1 launches FLA's q/k l2norm kernels (gated-delta-rule modules holding head_k_dim) with a constant NB and drops NB from their autotune key, so one compiled kernel and one tuned config serve every row count (Qwen3.5). Raises at load where the locked FLA's kernels no longer look like that. Read at load, by apply_fla_launch_options. |
HALO_EP_SHARED_OVERLAP | off | 1 runs the shared-expert FFN on a side stream concurrent with the routed dispatch all-to-all (shared-expert families only; GPT-OSS, Qwen3 MoE, LFM-2, Gemma 4 and Zaya have none). See DeepEP. |
HALO_EP_CAPACITY_DEDUP | 1 | 0 restores the per-MoE-layer DeepEP buffer-capacity all-reduce instead of reusing the first layer's capacity for the whole forward. One capacity per forward is also what lets every MoE layer share one ElasticBuffer, so 0 additionally gives each layer its own arena — multiplying it by the MoE layer count (DeepEP). Rank-uniform, refused at startup when ranks differ. Read once at import, with the other DeepEP knobs, in halo/distributed/expert_parallel/deepep/settings.py. |
HALO_GRAD_BUCKET_MB | 256 | Flat-buffer size of the bucketed gradient reduction — the deferred cross-replica EP sweep, the TP replicated / per-head-norm sweep, and the QLoRA sweep. (The per-parameter EP grad hooks reduce one tensor at a time and ignore it.) It sets the chunk boundaries every rank must agree on, so it belongs to the rank-uniform set verified in distributed setup (DeepEP). Two buckets run at once, so peak transient is two flat buffers, and under fp32_grad_reduce each also holds an fp32 copy — ~3× the bucket size from bf16. Must be >= 1; a non-positive value warns and keeps the default. |
HALO_DEEPEP_GPU_TIMEOUT_SECONDS | 100 | Device-side spin budget for the DeepEP dispatch/combine NVLink barrier (DeepEP's own default). The clock starts when a rank enters the barrier, so it bounds rank skew, not idle time between steps — a rank traps if a peer is still this far behind when it arrives. Save paths fence with a host-side barrier first, which absorbs skew under DIST_NCCL_TIMEOUT_MINUTES instead. Raising it also delays how fast a real hang surfaces. Elastic backend only — the legacy V1 buffer takes no timeout and warns when the parsed value is not the default (an explicit 100, or an empty pass-through, is not a request). Rank-uniform, refused at startup when ranks differ. See DeepEP. |
HALO_DEEPEP_NUM_SMS / HALO_DEEPEP_NUM_QPS | auto | Pin the DeepEP dispatch/combine SM count (else get_theoretical_num_sms) and RDMA queue-pair count. NUM_SMS applies to both buffer backends (legacy needs an even count); NUM_QPS is elastic-only and also sizes the buffer's QP allocation. 0 means auto, and a negative warns and means auto too. On EFA proxy Gin more QPs can raise the latency-bound internode all-to-all parallelism — A/B them. Rank-uniform, refused at startup when ranks differ; see DeepEP. |
HALO_DEEPEP_GIN_MAX_TOKENS_PER_RANK | 8192 | Cross-node (Gin) dispatch payload ceiling, tokens/rank. Above the measured ~8k boundary an EFA proxy-GIN dispatch wedges instead of erroring (receive counts never arrive; Xid 109→43), so ParallelismConfig refuses a declared budget above it at config time and the dispatcher re-checks at buffer sizing. 0 disables, and a negative warns and keeps the default; raise only after validating end-to-end on your fabric. Intra-node dispatch is unaffected. Rank-uniform, refused at startup when ranks differ. Read in halo/distributed/expert_parallel/deepep/capacity.py. See DeepEP → AWS EFA. |
HALO_PP_FUSED_HEAD_LOSS | 1 | Fold lm_head + cross-entropy into the last pipeline stage's forward in token chunks, so the [microbatch, seq, vocab] logits plane never exists. Applies only to the causal-LM pipeline contract; 0 restores the logits path. Read only under pipeline parallelism. |
EP_DISABLE_GIN / EP_SUPPRESS_NCCL_CHECK | backend-set / 1 | DeepEP-owned. The elastic backend sets EP_DISABLE_GIN once per process before its first buffer build (0 across NVLink domains, 1 within one) and honors an explicit value, a blank one reading as unset. EP_SUPPRESS_NCCL_CHECK is latched at import deep_ep, so it must be in the process environment (both images bake it), and suppresses DeepEP's duplicate-NCCL-runtime guard, which the NGC image's HPC-X libnccl-net transport plugin trips. Documented in DeepEP. |
HALO_DEEPGEMM_NATIVE | off | Opt into the native DeepGEMM forward for every grouped expert GEMM under lowp_precision: fp8|mxfp4; a shape the kernel rejects raises. The wheel is in the Blackwell image; on the Hopper image the flag warns once and stays inert. Whole steps measured at 0.37–0.54× bf16 (Low-Precision MoE). |
HALO_LOWP_WEIGHT_CACHE | 1 | 0 disables the per-step cache of each weight's quantized round-trip, re-quantizing it on every forward. |
HALO_ALLOW_MOCK_SEARCH | off | Makes the fabricated-results mock search backend selectable. Off, naming it raises: mock snippets pay tool_success_reward exactly like a real search, so a training run that reached them would reward invented evidence. For tests and smoke runs. |
HALO_SANDBOX_BACKEND / HALO_SANDBOX_URL / HALO_SANDBOX_MAX_CONCURRENCY | local / unset / usable CPUs (the affinity set, capped by any cgroup CPU quota over the process) | RL code-execution sandbox: backend (local / bubblewrap / remote; which ones warn), the SandboxFusion endpoint required by remote, and the process-wide concurrent-execution cap (local/bubblewrap only; no effect under remote). See Sandboxes. |
HALO_NCCL_SYNC_TIMEOUT_SECONDS | 600 | The deadline for one chunk of a weight sync on both clients: the update RPC that declares it (vLLM answers it only once the chunk arrived) and the drain of its broadcasts (a vLLM packed buffer, or a SGLang chunk's sends, settled two chunks later). On expiry the group is aborted instead of parking the trainer, with a hint naming the transport causes (address, NCCL_NET / plugin mismatch, cuMem). The 120 s warm-up and the 30 s failure-path cleanup keep their own deadlines. Raise for large policies over sockets. |
HALO_WEIGHT_SYNC_FORMATION_TIMEOUT_SECONDS | 120 | The deadline for forming one weight-sync group: the engine's join request and the trainer's rendezvous, per server. Raise it for engines that load slowly or span many workers; rank 0 forms the groups server by server while its peers wait over the c10d store, so the wait is not bounded by the NCCL watchdog (Rollout Servers). |
HALO_PREFLIGHT_PARALLELISM | 16 | How many rollout servers rank 0 probes at once in the startup preflights (HTTP only; results are read in server order). |
HALO_WEIGHT_SYNC_CHUNK_MB | 1024 | Budget of one streamed weight-sync chunk, staged on the forwarding rank's sync GPU; a tensor above it is a chunk of its own, and the staging every server's client sends from (SGLang's arena, vLLM's packed buffers) is one chunk, whatever the number of servers; the arena is at least this size (the vLLM packed buffers close at their own 1 GiB). Each chunk is one engine round trip, which bounds the vLLM rate (measured curve). |
HALO_VLLM_GENERATION_TIMEOUT_SECONDS | 80% of the NCCL watchdog (1440 at the default DIST_NCCL_TIMEOUT_MINUTES) | Per-request HTTP timeout (seconds) for a vLLM generation call from TrlVLLMClient, online GRPO only. The call blocks every peer at the next collective, so the default tracks DIST_NCCL_TIMEOUT_MINUTES and stays under it; an override at or above the watchdog is warned about when the client is built. Async GRPO generates from the Ray actors instead, bounded by request_timeout and episode_timeout. |
HALO_WEIGHT_SYNC_GATHER_AHEAD | 1 | How many EP layers' expert gathers the RL weight sync issues ahead of the layer being forwarded, on a side stream, so the gather overlaps the send; 0 runs gather and send strictly in turn. Each layer ahead holds one more assembled layer on the forwarding GPU. Rank-uniform (agree_rank_uniform_env), as the value sets each rank's collective order (Rollout Servers). |
HALO_WEIGHT_SYNC_MEM_LOG | off | 1 brackets each RL weight sync with a per-rank CUDA-memory line. The post−pre delta isolates the gathers from the training step, and the per-rank spread exposes the forwarding rank's extra full-tensor cost: reserved far above peak_alloc there means stranded allocator pools (Rollout Servers). Off by default: it is one INFO line per rank per sync. |
HALO_WEIGHT_SYNC_PARITY_PROBE | warn | The parity probe after a trainer's first weight push compares the engine's log-probs on a fixed counting row with the trainer's: warn logs a mean gap above 1 nat per position, raise stops the run on every rank, off skips it (Rollout Servers → First-push parity probe). Skipped under context parallelism whatever its value. Rank-uniform, refused at startup when ranks differ. |
HALO_FP32_MATMUL_PRECISION | highest | fp32 matmul precision applied at model load (highest/high/medium). The default overrides the NGC image's TF32 preference, whose 10-bit mantissa collapses adjacent RoPE token positions past 2048 wherever RoPE is a matmul. transformers' own RoPE multiplies (transformers#49265, in the pinned release), so the pin matters for remote code such as Ling and Ring, and for any other fp32 matmul. bf16 matmuls are unaffected. torch.backends.cudnn.allow_tf32 follows the same knob, so fp32 convolutions (Zaya's Conv1d) match the matmuls. Set high to opt back into TF32 for an fp32-heavy job. Rank-uniform, refused at startup when ranks differ. |
HALO_ALLOW_MISSING_CHECKPOINT_KEYS | off | 1 downgrades the model-load coverage gate to a warning. The gate raises when a checkpoint does not carry a tensor the live model needs — see Checkpoints. A task head added on top of a base checkpoint, a tied lm_head and the class's own _keys_to_ignore_on_load_missing are excused without the flag. |
HALO_CHECKPOINT_WRITERS_PER_NODE | 1 | Ranks per node that write a gathered EP save on a shared output filesystem. Each writer retains and writes the layers it owns (chunk 0, the non-expert tensors, stays with the save rank; each later layer goes round-robin) under its own part prefix, and the save rank renames the parts and writes the one index, so the saved tensors, keys, dtypes and index size are the same at any count. The default already stripes a multi-node job over its nodes' first ranks, and 0 keeps the save rank the only writer; a larger count adds writers within a node, each holding one gathered layer plus its filling part in host RAM. Capped at the node's ranks. Per-node storage keeps one writer per node, since each node's directory must hold the whole checkpoint. Rank-uniform (agree_rank_uniform_env); a negative count raises on every rank at startup. See Checkpoints. |
HALO_CHECKPOINT_WRITE_THREADS | 0 | Opt-in background threads that write a streamed checkpoint's finished safetensors parts while the save rank gathers and fills the next: every gathered save (FSDP2, CP, TP, EP, save_model included) and the merge_models, merge_ep_shards, unfuse_moe_experts, reattach_vision_tower and FP8-to-bf16 conversion tools. Each thread holds up to one more part in host RAM, at the writer's shard cap (save_max_shard_size for a save, --max_shard_size for the tools, 5GB by default); the bytes on disk are the same at any count. 0, the default, writes each part in line, as the pipeline-stage writer always does. Rank-uniform (agree_rank_uniform_env): ranks that differ, or a negative count, raise on every rank at startup; each writer reads the count again when it is built, the pipeline-stage writer included. See Checkpoints. |
HALO_TORCH_NUM_THREADS | 1 | CPU threads per rank, applied with torch.set_num_threads at package import (one process per GPU, so 1 avoids contention); it overrides OMP_NUM_THREADS, PyTorch's own knob. Must be >= 1; a non-positive value warns and keeps the default. |
HF_HOME / HF_DATASETS_CACHE | ~/.cache/huggingface / $HF_HOME/datasets | Hugging Face caches; point both at the scratch volume. HF_HOME also anchors the two JIT kernel caches below, and one elected reader per filesystem creates the datasets cache before the dataset load. |
TRITON_CACHE_DIR / FLASH_ATTENTION_CUTE_DSL_CACHE_DIR | $HF_HOME/triton_cache / $HF_HOME/flash_attn_cute_dsl_cache (the temp dir without HF_HOME, or with a warning where that location is not writable, such as a shared cache mounted read-only; unset, leaving the library's own default, when the temp dir is not writable either) | JIT kernel caches, set at script start unless exported with a value, so a --rm container does not recompile every kernel on each launch. FLA's autotune picks live in the Triton cache, and different picks round differently (Docker). |
FLASH_ATTENTION_DETERMINISTIC | unset; 1 under full_determinism | Read by transformers, which passes it to the flash kernels as deterministic. parse_script_args sets it as it parses a full_determinism config (HF's Trainer sets it only at construction), so SFT's row plan, made before the run setup, and the loader's resolver both see it. FA4's head_dim-256 kernel is not validated with a deterministic backward, and such a run demotes FA4 to SDPA there, to FA2 on SFT rows that hold several documents (Flash Attention). Compared across ranks with the rank-uniform settings (DeepEP): set on some ranks only, a head_dim-256 model would load a different backend per rank. |
CAUSAL_CONV1D_DETERMINISTIC | unset | Read by causal_conv1d's backward on every call. 1 takes its deterministic path and 0 its default one, which sums the conv weight gradient with atomics, so two identical steps can round apart and a gated-DeltaNet family (Qwen3.5/3.6 MoE, Qwen3-Next, Qwen3.8-Flash-Next) misses an exact replay. Unset or any other value follows torch.use_deterministic_algorithms, which HF's Trainer turns on under full_determinism, so only a run without it needs 1 for a deterministic backward. |
CUDA_DEVICE_MAX_CONNECTIONS | 1 (baked into both images) | Driver-owned, latched at deep_ep's cuInit, so a Python write is too late. 1 serializes work submission; it is the setting the EP suites are validated with, at no measurable throughput cost (DeepEP). The trainer reads it only to warn when it is not 1. |
PYTORCH_CUDA_ALLOC_CONF | unset | PyTorch's caching-allocator settings, set in the launch environment: halo launch passes the shell's value through, and make train takes it as EXTRA_DOCKER_ENV="-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True". The dense SFT, DPO and SMPO examples launch with expandable_segments:True; the EP examples leave it unset, and an EP run sets it only to fit, since there it costs throughput (measured). No image sets it, because one image runs dense and EP training alike; keep it out of a .env that EP runs also read. |
VLLM_DISABLE_PYNCCL / VLLM_NCCL_SO_PATH | off / libnccl.so.2 | Vendored from vLLM. VLLM_NCCL_SO_PATH is honored by the trainer-side weight-sync client; VLLM_DISABLE_PYNCCL set in the trainer process is refused (RuntimeError): it belongs to the vLLM server only, and honoring it would silently disable the weight-sync broadcast. It is parsed the way vLLM parses it (1/true only, not the toolkit's yes/on), so the trainer's verdict matches the server's. |
VLLM_USE_V2_MODEL_RUNNER | 1 (baked into the vLLM image) | Read by the vLLM server, not the toolkit. The image pins Model Runner V2, the runner R3 capture starts on and the gpt-oss thinking-budget patches target; compose passes the variable through bare, so unset keeps the pin. 0 selects the V1 runner, which breaks R3 capture and the gpt-oss budget cut (Rollout Servers). 0 or 1 only — vLLM parses it with int(), so false or an empty value kills the server at startup. |
VLLM_ENFORCE_STRICT_TOOL_CALLING | 0 (baked into the vLLM image) | Read by the vLLM server. On, the GLM parsers sample tool-bearing requests under a grammar the trainer's log-probs never see, so the startup preflight refuses a server reporting it on a tool-bearing environment. |
VLLM_HALO_PATCHES | set by the vLLM image | Not a knob: the image's identity patch lists its patches under this name on /server_info, where the trainer refuses a server without it. |
NCCL_CUMEM_ENABLE | 1 in docker-compose.sglang.yml | Read by NCCL in the SGLang server, never the toolkit: SGLang otherwise turns cuMem off process-wide while the trainer's stays on, and the mismatch fails the first cross-container buffer import (why). Keep both ends equal; the compose default does. |
VLLM_API_KEY | EMPTY | OpenAI-compatible API key the scripts/inference/ and scripts/environments/ CLIs, and a judge term on a loopback base_url, send to a self-hosted server, matching the server's own --api-key convention. Falls back to OPENAI_API_KEY, then to the EMPTY placeholder a keyless local server accepts; each CLI's key flag overrides both. |
WANDB_PROJECT | project_name arg | Overwritten unconditionally by the trainer setup before wandb initializes (an exported value does not survive). The run NAME does not travel through WANDB_NAME: transformers passes run_name straight to wandb.init, which wins over the environment, so the setup shortens run_name to its last path segment (else <method>-<mode>-<output_dir basename>, e.g. sft-ep-<run dir>) and writes it back onto the training config instead. |
WANDB_RUN_ID | md5(output_dir:launch_time)[:8] | Resume identity. An exported value is honored on global rank 0 and broadcast to every rank, so export it on the whole job (or nowhere) and reuse it to continue the same wandb run across restarts. |
WANDB_RESUME | unset | Consumed by the wandb SDK, not by this toolkit: set allow alongside a fixed WANDB_RUN_ID to append to that run instead of starting a new one. |
TOKENIZERS_PARALLELISM | false | Set with setdefault at package import (halo/__init__.py), before tokenizers loads: the Rust tokenizer's thread pool deadlocks against the dataset-map worker processes that fork around it. An exported value wins, so a single-process tool can opt back into parallel encoding. |
CLEARML_PROJECT / CLEARML_TASK | project_name arg / run_name basename | Set by the same trainer setup, same unconditional overwrite, and read by the transformers ClearML integration when report_to includes clearml — which is why the task name IS an env var here and is not one for wandb. |
Trainer-to-config quick reference
Each trainer combines a trainer config (hyperparameters) with script arguments (dataset, tokenizer, infra). All configs also accept standard TrainingArguments fields. Every script additionally parses ModelArguments (halo/configs/script_arguments.py: TRL's ModelConfig plus trust_remote_code, which TRL moved onto its own trainer configs, with a one-entry lora_target_modules list kept a list rather than collapsed to a regex string) — model_name_or_path, model_revision, dtype, attn_implementation, trust_remote_code and the LoRA fields (PEFT) — and ParallelismArguments (the ParallelismConfig knobs); all four dataclasses draw from the one YAML.
| Trainer | Config Class | Script Arguments | Source |
|---|---|---|---|
DistributedSFTTrainer | SFTConfig (TRL) | SFTScriptArguments | sft.py |
SMPOTrainer | SMPOConfig | SMPOScriptArguments | preference/smpo.py |
DistributedDPOTrainer | DPOConfig (TRL) | DPOScriptArguments | preference/dpo.py |
DistributedKTOTrainer | KTOConfig (TRL) | KTOScriptArguments | preference/kto.py |
OfflineGRPOTrainer | OfflineGRPOConfig | OfflineGRPOScriptArguments | grpo/offline.py |
DistributedOnlineGRPOTrainer (RLVR) | GRPOConfig (TRL) | OnlineGRPOScriptArguments | grpo/online.py |
DistributedSDPGTrainer (online SDPG) | GRPOConfig (TRL) | OnlineGRPOScriptArguments (--use_sdpg=true) | grpo/online.py |
DistributedAsyncEnvironmentalGRPOTrainer | GRPOConfig (TRL) + EnvironmentConfig + AsyncRolloutConfig | EnvironmentalGRPOScriptArguments | grpo/environmental.py |
DistributedRewardTrainer | RewardConfig (TRL) | RewardScriptArguments | reward/reward_model.py |
ClassificationTrainer | ClassificationConfig | ClassificationScriptArguments | classification.py |
DistributedTeacherDistillationTrainer | TeacherDistillationConfig | TeacherDistillScriptArguments | distillation/teacher_distill.py |
DistributedSelfDistillationTrainer (offline SDPG) | SFTConfig (TRL) | SelfDistillScriptArguments | distillation/self_distill.py |
EmbeddingTrainer | EmbeddingConfig (extends SentenceTransformerTrainingArguments) | EmbeddingScriptArguments | embedding.py |
All distributed trainers parse the ParallelismConfig parameters, in the YAML or on the CLI. A trainer refuses, before any model loads, what it does not support: a context_parallel_size or pipeline_parallel_size above 1 where its class does not declare that axis, the lowp_* knobs outside SFT, and init_from_scratch outside SFT.
TRL trainer configs
These configs come from TRL and are used directly. Only the commonly tuned fields are listed; all extend TrainingArguments.
SFTConfig — DistributedSFTTrainer (TRL docs)
| Parameter | Type | Default | Description |
|---|---|---|---|
max_length | int | None | 1024 | Total sequence budget. Over-length conversations are dropped by the chat processor; under packing it is the fixed pack size and must be explicit. null → model context window. |
packing | bool | False | Pack multiple sequences into one example. |
packing_strategy | str | "bfd" | bfd (best-fit-decreasing, drops overflow), bfd_split (carries overflow into later examples) or wrapped (concatenate-and-chunk, cutting a document at the row edge). |
eval_packing | bool | None | None | Packing for eval. Defaults to packing; it can only turn packing off for the eval split (true without packing raises), and a pre-processed dataset refuses it. |
padding_free | bool | False | Flatten the batch into one varlen sequence instead of padding it. Needs a varlen Flash kernel (FA2/FA3/FA4) and raises on any other resolved attn_implementation — the cu_seqlens it exists to emit go unread there. See Padding-Free. |
dataset_text_field | str | "text" | TRL's pre-formatted text column. Halo SFT renders conversation_field through the chat template and never reads it, so a non-default value raises. |
dataset_kwargs | dict | None | None | TRL's dataset-prep kwargs. Halo SFT replaces TRL's dataset prep with its own, so a user value raises. |
chat_template_path | str | None | None | TRL's template override, which its trainer applies after Halo SFT has rendered the rows, so a value raises. Set chat_template (plus force_chat_template: true over a shipped template) instead. |
bf16 | bool | None | True | Toolkit default True (TRL leaves it None); yields to an explicitly-enabled fp16. Applied before the config is built, so mixed_precision follows it. |
use_liger_kernel | bool | True | Liger Triton kernels. Toolkit default True (TRL default False). |
logging_nan_inf_filter | bool | False | Toolkit default False (upstream True). Upstream's filter reads the loss scalar back to the host on every micro-batch — gradient_accumulation_steps stalls per optimizer step, and at large world size every rank then waits on the slowest — and replaces a NaN/Inf loss with the running average, hiding a diverged run from its own loss curve. |
accelerator_config | dict | str | None | {non_blocking: true} with pinned batches | Toolkit default for one key: non_blocking: true whenever dataloader_pin_memory is not false and use_cpu is off, merged into a dict the config gives and never over a key it sets (a JSON file path is left whole), so a pinned batch copies to the device behind the queued kernels instead of waiting for them (Qwen3-8B CP2 +11.4%, FSDP2 flat; 8× B300, 1.2.0, two rounds). |
DPOConfig — DistributedDPOTrainer (TRL docs)
| Parameter | Type | Default | Description |
|---|---|---|---|
beta | float | 0.1 | KL penalty coefficient. Higher = more conservative updates. |
loss_type | list[str] | ["sigmoid"] | sigmoid, hinge, ipo, exo_pair, nca_pair, robust, bco_pair, sppo_hard, aot, aot_unpaired, discopop, apo_zero, apo_down, sft, sigmoid_norm (a list — losses can be combined). |
max_length | int | None | 1024 | Total budget (prompt + completion). TRL truncates the concatenated sequence keep_start, so an over-long prompt eats its own completion — DPOConfig has no max_prompt_length / max_completion_length. Filter over-long prompts upstream. null → model context window. |
label_smoothing | float | 0.0 | Label smoothing for DPO loss. |
truncation_mode | str | "keep_start" | keep_start, or the upstream-deprecated keep_end. |
disable_dropout | bool | True | Disable dropout. |
KTOConfig — DistributedKTOTrainer
Unpaired {prompt, completion, label} preference optimization.
| Parameter | Type | Default | Description |
|---|---|---|---|
beta | float | 0.1 | Deviation allowed from the reference. |
desirable_weight / undesirable_weight | float | 1.0 / 1.0 | Re-weight the two classes when the label counts are skewed. |
loss_type | str | "kto" | kto, or apo_zero_unpaired, which carries no KL term. |
max_length | int | None | 1024 | Truncates the assembled prompt + completion, keeping the start. null → model context window. |
GRPOConfig — DistributedOnlineGRPOTrainer, DistributedAsyncEnvironmentalGRPOTrainer (TRL docs)
| Parameter | Type | Default | Description |
|---|---|---|---|
num_generations | int | None | 8 | Completions per prompt. |
num_generations_eval | int | None | None | Eval group size; None = num_generations. |
max_completion_length | int | None | 512 | Generation budget (online / vanilla GRPO; TRL max_new_tokens). Set it explicitly. Async GRPO does not use it: it generates with rollout_max_tokens and is bounded by the model context (see Sequence length). |
beta | float | 0.0 | KL penalty coefficient (0.0 = no reference model). Non-zero on a policy no PEFT adapter wraps (a full fine-tune, or expert-only LoRA), the online and async GRPO scripts load a frozen reference from model_name_or_path, never a resume checkpoint, as the policy loads (run dtype, model_revision, its attention and reset_sinks), and the trainer hands it to TRL in place of the fp32 copy TRL would build. A trainer constructed without one raises, as does one given a ref_model the run never reads. |
use_bias_correction_kl | bool | True | Multiplies the KL term by the importance-sampling ratio, the unbiased reverse-KL gradient. It changes the KL gradient of every online and async GRPO run at beta != 0, on-policy included (the ratio is differentiable even where it equals 1); false keeps the plain k3 term. |
temperature | float | 1.0 | Sampling temperature. Async GRPO overwrites it (with a log) with rollout_temperature, so log-probs are scored at the temperature the rollout sampled at. |
top_k | int | 0 | Top-k sampling (0 = disabled). Online GRPO only; refused with a sequence-level IS mode or vespo as top_p < 1 is. |
top_p | float | 1.0 | Top-p sampling. Online GRPO only. With a sequence_* vllm_importance_sampling_mode or loss_type: vespo, top_p < 1 (or a top_k, or a min_p) against a server whose logprobs are renormalized over the cut (vLLM processed_logprobs) is refused at startup (Online GRPO). |
num_iterations | int | 1 | Policy update iterations per batch (mu-GRPO). |
epsilon | float | 0.2 | Clipping epsilon. |
loss_type | str | "dapo" | grpo, dapo (default), bnpo, dr_grpo, plus TRL's cispo, sapo, vespo, luspo. Offline GRPO accepts only grpo, bnpo, dr_grpo. vespo weighs each sequence by the sum of its per-token log IS ratios in place of the per-token multiply: async GRPO refuses it at construction (a token its isr_* stages mask or a forced reasoning close sets the ratio to 0, which drops the whole turn row), and online GRPO's sampler-logprob preflight treats it as summing the ratio over the sequence, as a sequence_* mode does. |
scale_rewards | str | "group" | "group"/True (per-group std), "batch", "none"/False. |
mask_truncated_completions | bool | False | Drop completions the engine cut at its token cap from the loss. Async GRPO widens it to any truncated trajectory (turn overflow and a lost episode included) and applies it in its own drop set (its rollout path replaces TRL's generation), masking every row of a truncated trajectory and logging sampling/truncated_masked_frac. |
off_policy_mask_threshold | float | None | None | TRL's off-policy token mask. Refused on async GRPO (raises at construction): TRL's mask reads sampling_per_token_logps, which that trainer's batch never carries, so it would threshold a KL of exactly 0; use isr_opsm_delta instead. Online GRPO refuses it beside balance_token_mass: it drops negative sequences after the balance has weighed them. |
use_vllm | bool | False | Use vLLM for generation. The toolkit's online GRPO trainer requires true: it raises on false (in-process HF generation is slow and desyncs ranks under FSDP2/EP). |
vllm_mode | str | "colocate" | server (external) or colocate (managed). The toolkit's online GRPO trainer supports only server: set it in the config (it raises on colocate). |
vllm_server_host / vllm_server_port | str / int | "0.0.0.0" / 8000 | vLLM server address/port. |
vllm_group_port | int | 51216 | NCCL group port of Online GRPO's weight sync. Async GRPO refuses a non-default value at construction: its port is AsyncRolloutConfig.rollout_group_port. |
log_completions | bool | False | Print the rich per-sample completions console table (TRL). Decoupled from the durable parquet record; see save_completions. |
save_completions | bool | True | Halo CommonScriptArguments field, not TRL: the durable parquet record, see CommonScriptArguments. |
remove_unused_columns | bool | False | Async GRPO forces it False at construction and warns when a config sets it true: the rollout context is the row's non-prompt columns, and pruning would strip answer and every context_fields column, leaving the environment grading episodes it was handed no ground truth for. |
Async GRPO samples on the rollout server, so top_p, top_k, min_p, repetition_penalty and generation_kwargs reach no sampler and are refused at trainer construction when set away from their TRL defaults — use the rollout_* fields on AsyncRolloutConfig. It refuses vllm_importance_sampling_clip_min the same way (its IS correction is token-level and clips from above only), and vllm_importance_sampling_mode at any value but TRL's default or token_truncate, the one that names that correction (Objective). It forces use_vllm: true while that correction is on.
Both on-policy trainers force use_liger_kernel off and refuse the TRL options they cannot honor at construction (Online GRPO → Forced and refused settings).
RewardConfig — DistributedRewardTrainer (TRL docs)
| Parameter | Type | Default | Description |
|---|---|---|---|
max_length | int | None | 1024 | Total sequence budget, applied as a filter: TRL drops pairs whose chosen or rejected exceeds it, never truncates. null → model context window. |
center_rewards_coefficient | float | None | None | Reward-centering loss coefficient: penalizes (chosen + rejected)², pulling scores toward zero mean. Helps prevent reward hacking. |
disable_dropout | bool | True | Disable dropout. |
ParallelismConfig
Unified config for Expert (EP), Context (CP), Tensor (TP), Expert-Tensor (ETP), and Pipeline (PP) Parallelism. See Pipeline Parallelism for the pp_* fields. The class itself is built at construction; users set it through ParallelismArguments (halo/configs/parallelism_arguments.py), whose names go in the YAML or on the CLI and differ from the field names below.
ParallelismConfig field | ParallelismArguments name |
|---|---|
ep_size | expert_parallel_size |
cp_size | context_parallel_size |
tp_size | tensor_parallel_size |
expert_tp_size | expert_tensor_parallel_size |
pp_size | pipeline_parallel_size |
pp_schedule | pipeline_schedule |
pp_microbatches | pipeline_microbatches |
pp_split | pipeline_split |
ep_fp32_router | fp32_router |
ep_fp32_experts | fp32_experts |
Every other field below keeps its name, except world_size / gpus_per_node (auto-detected, not settable), expert_lora (built from the PEFT LoraConfig, so it has no flag; row below) and merge_expert_lora_on_save (a ParallelismArguments flag, documented above).
Source: halo/distributed/parallelism_config.py
| Parameter | Type | Default | Description |
|---|---|---|---|
world_size / gpus_per_node | int | 0 | 0 = auto-detect from the launcher environment. |
ep_size | int | 1 | Expert-parallel dispatch size: the ranks one DeepEP all-to-all spans, each holding num_experts / ep_size experts. >1 enables EP for MoE. With expert_tp_size it forms the ep_group_size group. |
cp_size | int | 1 | Context parallel group size. >1 enables sequence splitting. |
tp_size | int | 1 | Tensor parallel size. >1 shards model weights. |
expert_tp_size | int | 1 | Expert Tensor Parallel size. Shards expert FFN weights within EP groups (MoE-only, experimental). |
pp_size | int | 1 | Pipeline stages; each must be a whole number of NVLink domains. The three PP-only knobs below are rejected at pp_size == 1. Outermost axis by design — it carves the world first (stage_world_size = world_size / pp_size). See Pipeline Parallelism. |
pp_schedule | "1f1b" | "gpipe" | "1f1b" | 1f1b (lowest activation memory, needs pp_microbatches >= pp_size) or gpipe (all forwards then all backwards, no microbatch-count constraint). |
pp_microbatches | int | 0 | Microbatches per optimizer step — the pipeline's own accumulation. 0 = auto: gradient_accumulation_steps, raised to pp_size when 1f1b requires it. Setting it alongside gradient_accumulation_steps > 1 raises (the microbatches are the accumulation); per_device_train_batch_size must divide by it. |
pp_split | list[int] | None | None | Per-stage decoder-layer counts (one entry per stage, each ≥ 1, summing to the model's layer count, with boundaries on the model's layer_types period). None = the head-weighted default, which shrinks the last stage's budget by the lm_head's layer-equivalent cost. |
ep_scope | "auto" | "node" | "global" | "auto" | auto resolves to node-local if ep_group_size <= nvlink_domain_size, else global. node keeps groups within each NVLink domain. global spans domains (requires RDMA) and refuses full_determinism there (why). |
nvlink_domain_size | int | 0 | Locality unit for node-local EP/CP/TP/ETP grouping. 0 = auto: reads NVLINK_DOMAIN_SIZE, else gpus_per_node. Set to the rack size (e.g. 72) on NVL72/MNNVL. CLI: --nvlink_domain_size. |
ep_fp32_router | bool | False | Keep MoE router weights as FP32 masters. The routing matmul runs in FP32 too, except where FSDP2 wraps an ep_group_size==1 router under fsdp_shard_ep1_experts: the router takes its own FSDP2 shard group and computes in BF16 there, while a run FSDP2 does not wrap (one rank, or TP at data_parallel_size == 1) and a frozen router the wrap leaves out of its shard groups keep FP32 (Precision control). Implied by fp32_non_ep_params, whose upcast skips the router: a BF16 router beside FP32 dense params would trip FSDP2's uniform-dtype check. Gemma 4 ignores it with a warning, its router sitting outside the EP layer. |
ep_fp32_experts | bool | False | Store EP expert weights in FP32 for stable optimizer updates; compute stays BF16 under autocast. Skipped with a warning when FSDP2 manages the experts (ep_group_size==1 at the default fsdp_shard_ep1_experts), where it would do nothing. |
fp32_non_ep_params | bool | False | Store non-expert params in FP32 (compute stays BF16 via autocast). Every training loader then restores every parameter outside the EP-wrapped MoE blocks (shared experts stay in the run dtype) from the checkpoint at FP32 before any parallel wrapper (an FP32 checkpoint keeps its full precision), where without the flag it casts them to the run dtype (Load precision). Implies fp32 reduction for the FSDP2 reduce_dtype only — the EP, TP and QLoRA bucketed reductions read fp32_grad_reduce directly, so set that explicitly to cover them. Refused on an ep_group_size==1 MoE at the default fsdp_shard_ep1_experts (set fsdp_shard_ep1_experts: false, which TP/CP refuse), and under EP for a family whose spec declares supports_fp32_non_ep_params=False (Gemma 4). Outside AdamWBF16 the optimizer steps one group per weight decay, dtype and tensor type (FSDP2 DTensor or plain EP tensor), each fused under optim: adamw_torch_fused and single-tensor otherwise. |
fp32_grad_reduce | bool | False | Reduce gradients across ranks in FP32 while storing BF16 (FSDP2 reduce_dtype=fp32 for dense params, the EP router grad-sync hooks and deferred sweep, the TP replicated-grad sync, the QLoRA adapter all-reduce). Keeps BF16 master + AdamWBF16 (6 B/param); ~2x bandwidth on the reduce. See BF16 Optimizer. |
use_grouped_gemm | bool | True | torch.nn.functional.grouped_mm for MoE expert compute on SM90+ (no-op on older HW). MoE layers are wrapped even standalone (ep_size=1, no EP), so an MoE under any accelerate launch is rejected at load — launch with torchrun or set false. Inert for dense models (they run under accelerate normally). |
fsdp_reshard_after_forward | bool | False | FSDP2 resharding: False = ZeRO-2 analog (SHARD_GRAD_OP, params stay gathered, faster, higher peak memory); True = ZeRO-3 analog (FULL_SHARD, lower peak memory, and each decoder layer its own shard group where ZeRO-2 groups two — Shard groups). Allowed only where ep_group_size==1 (pure DP, pure TP with dp=1, CP, ep_size==1 MoE without expert TP); rejected with ep_size>1, Expert-TP, TP+DP (dp>1), PP or QLoRA — the backward re-gather races the DeepEP combine, or all-gathers TP-sharded DTensors with no sharding strategy, and QLoRA skips FSDP2. See ZeRO-2 vs ZeRO-3. |
fsdp_reshard_after_backward | bool | True | False keeps params unsharded across a grad-accum window's microsteps (torch set_reshard_after_backward, re-armed for the window's last backward): drops the full-model re-all-gather FSDP2 otherwise issues after every microstep's backward to one per optimizer step — about 4–10% throughput over NVLink, ~15 s × gradient_accumulation_steps per step when the trainer's NCCL runs over TCP sockets. Saves nothing at gradient_accumulation_steps=1. The unsharded params (at the FSDP param dtype) are the ones the ZeRO-2 forward/backward already holds, kept between microsteps and freed at the window's last backward, so peak memory is unchanged. Plain-DP/CP/EP torchrun path only; rejected with fsdp_reshard_after_forward=True, TP, PP or QLoRA. See Data Parallelism. |
fsdp_defer_grad_sync | bool | False | True skips FSDP2's gradient reduce-scatter (and the HSDP all-reduce) on a grad-accum window's microsteps 1..n-1 (torch set_requires_gradient_sync, re-armed for the window's last backward), so each optimizer step reduces once instead of gradient_accumulation_steps times. Costs one unsharded gradient copy per GPU held across the window (2 B/param; 4 B/param when the reduce runs in fp32 — fp32_grad_reduce, fp32_non_ep_params or bf16: false), which raises the peak by that copy less the sharded gradient the default holds anyway. Nothing at gradient_accumulation_steps=1. Every torchrun FSDP2 wrap (DP, HSDP, CP, TP+DP, EP); rejected under PP, TP at data_parallel_size=1, QLoRA, and with fsdp_reshard_after_forward=True (the held unsharded gradient is the state ZeRO-3 shards; defer under ZeRO-2 instead). See Deferred gradient reduce. |
fsdp_shard_ep1_experts | bool | True | Applies only when experts are truly replicated — ep_group_size==1 (ep_size==1 and expert_tp_size==1): FSDP-shards the replicated MoE experts so their reduce-scatter becomes the sole gradient sync. Default True frees memory that grows with DP, is grad-equivalent, and is faster than replicated experts (measured); False keeps a full replicated copy per rank, which fp32_non_ep_params and ep_fp32_experts need at ep1. No effect when ep_group_size>1. Composes with fsdp_reshard_after_forward for full ZeRO-3 experts, and works for online / async GRPO (the weight-sync gather full_tensor-s the sharded experts first). False raises at config time under TP or CP (those paths shard ep1 experts unconditionally, so the flag would be ignored) and, under PP, wherever EP layers meet ep_group_size==1 (a MoE stage and a dense stage would deadlock at the gradient norm). |
use_hsdp | bool | False | Hybrid Sharded DP on the standard DP path (pure DP or CP, no EP). Shards non-expert params within each NVLink domain and replicates across domains, so only one gradient all-reduce crosses RDMA per backward (per optimizer step with fsdp_defer_grad_sync). No-op on a single domain. Rejected with any EP (including EP+CP), TP, or Expert-TP — EP already shards over the EP group, so HSDP would be a no-op or race the DeepEP combine — under PP, whose stages cannot restrict its 2-D mesh, and under QLoRA, which skips FSDP2. See HSDP. |
lowp | LowpSpec | LowpSpec() | Low-precision compute, built from the lowp_* keys below (lowp_<field> sets lowp.<field>); validated by LowpSpec.validate. |
lowp_precision | "bf16" | "fp8" | "fp4" | "mxfp4" | "bf16" | Low-precision matmul compute — bf16/fp32 master weights, low-precision GEMM operands; params and checkpoints unchanged. bf16 (off), fp8 (mxfp8), fp4 (nvfp4, most accurate), mxfp4 (fast fp4). All are the block-scale fake-quant QAT oracle, not a speedup: fp4 runs its activation quant eager and stays several times slower than bf16, while fp8 and mxfp4 round-trip activations in one fused kernel (simulated mxfp4 whole steps at 0.93–0.99× bf16 on 8× H200). The native DeepGEMM kernel is opt-in (HALO_DEEPGEMM_NATIVE=1, fp8 and mxfp4 only), measured slower than bf16, and never auto-selected. Applies to dense MLP + MoE expert GEMMs; attention/embeddings/lm_head/norms stay bf16. SFT script only — every other script rejects a non-bf16 lowp_precision and any non-default lowp_apply_* / lowp_keep_*; pipeline parallelism rejects a non-bf16 one too (why). See Low-Precision Kernels. |
lowp_apply_dense_mlp | bool | True | Apply lowp_precision to dense MLP projections (gate/up/down). |
lowp_apply_moe_experts | bool | True | Apply lowp_precision to MoE expert grouped GEMMs. |
lowp_keep_first_blocks / lowp_keep_last_blocks | int | 0 | Leading/trailing transformer blocks kept in bf16 (NVFP4 recipes keep about the last 15% of blocks). |
bf16_optimizer | bool | None | None | Use AdamWBF16 (bf16 master weights + stochastic rounding, 6 B/param). None = auto (on under bf16 with the default AdamW, off under replicated DDP); True forces it on. False builds the stock fused/foreach AdamW over the params as loaded — under bf16 that is bf16 masters with round-to-nearest updates, not fp32 — refused at optimizer build wherever plain-tensor experts sit beside FSDP2 DTensor params (ep_group_size > 1, or EP-wrapped experts at ep_size==1 with fsdp_shard_ep1_experts: false), since aten._fused_adamw_ cannot mix the two. An ep_size==1 MoE at the default fsdp_shard_ep1_experts is uniformly DTensor and steps fine. The refusal does not apply when fp32_non_ep_params is set, which routes to a per-tensor-type grouped AdamW with fp32 masters on the non-expert params only. See BF16 Optimizer. |
ep_lazy_loading | bool | True | Lazy safetensors loading for EP, EP+CP, EP+TP, EP+ETP and pure ETP — each rank reads only its expert slice (parallel, low CPU RAM). Covers every MoE family except Cohere2 MoE, across fused and per-expert safetensors layouts — the gate is the family spec's supports_lazy_loading (per-family EP restrictions), not a key list; a checkpoint whose expert layout the loader cannot address, or a non-local one, falls back to sequential CPU-staged loading. TP-only MoE ignores it and always uses from_pretrained + patching; PP ignores false and always loads each stage lazily. |
ep_buffer_backend | "auto" | "elastic" | "legacy" | "auto" | DeepEP all-to-all transport. auto/elastic: V2 ElasticBuffer (NCCL Gin, cross-node capable). Within an NVLink domain the dispatch has no token ceiling; for an EP group spanning domains the Gin ceiling HALO_DEEPEP_GIN_MAX_TOKENS_PER_RANK (8192 tokens/rank) is checked at load, before any weight is read. legacy: V1 CUDA-IPC Buffer, numerically identical but intranode-only and rejected for any cross-node EP group; pick it for long-context ep8, where elastic at ≥~64k tokens/rank deadlocks its combine against FSDP2. See Transport backend. |
max_concurrent_loading | int | None | null | Max ranks loading simultaneously per node. Unset resolves node-width-aware to min(4, max(1, local_world_size // 2)) — 4 on an 8-GPU node, 2 on a 4-GPU tray. Every explicit value is honored verbatim, 4 included: 0 = all-parallel, a value at or above the node width disarms the throttle, 1 is fully sequential for CPU-RAM-constrained machines. Also throttles the per-rank optimizer-shard writes on save and reads on resume. Which loaders it governs, and how each joins its peers: Multi-Node → Model loading. |
expert_lora | ExpertLoraSpec | None | None | Native grouped LoRA on EP experts. No CLI flag — it is built from the PEFT LoraConfig and passed at construction so it cannot bypass validation. Carries r / alpha / dropout / projections / use_rslora (r a multiple of 8, the grouped GEMM's stride contract at the bf16 the adapters run in — refused at construction otherwise); knobs it cannot honor (use_dora, lora_target_parameters) are rejected rather than applied to the attention half alone. See PEFT. |
Besides these, num_nodes, num_nvlink_domains, ep_group_size, data_parallel_size, stage_world_size, pp_rank, and the stage-local rank coordinates are computed in __post_init__.
Two more carry no ParallelismArguments spelling and are derived from the training config by parallelism_config_from_args (halo/configs/parallelism_arguments.py): ep_rows_per_device (the rows one MoE forward carries per device — the trainer's rows-per-example × per_device_train_batch_size, divided under PP by the microbatch count) and ep_declared_max_length, which the load-time dispatch-ceiling gate multiplies out into a per-rank token budget before any weight is read.
A trainer whose config declares no max_length field stamps neither and leaves the ceiling to the dispatcher's runtime backstop. max_length: null still stamps the rows, as does offline GRPO's internal field, which no key sets; the gate then takes max_prompt_length + max_completion_length where the config bounds both (offline GRPO, or SMPO with both shares set), else the model's context window.
Five knobs are implemented inside the mixin's own FSDP2 wrap and are therefore ignored under accelerate launch (accelerate owns the wrap): use_hsdp, fsdp_reshard_after_forward, fsdp_reshard_after_backward, fsdp_defer_grad_sync, fp32_grad_reduce. Setting any of them on an accelerate launch logs one warning — use torchrun. Accelerate's own no_sync already skips the gradient reduce on a window's non-final microsteps, so fsdp_defer_grad_sync has nothing to add there.
OfflineGRPOConfig
OfflineGRPOTrainer: group-relative policy optimization on pre-computed rewards. Source: halo/configs/grpo_offline.py
| Parameter | Type | Default | Description |
|---|---|---|---|
max_prompt_length | int | None | 512 | Prompt budget; a longer prompt is left-truncated (the tokens nearest the completion are kept). null = no cap, which pipeline parallelism refuses. |
max_completion_length | int | None | None | Completion budget; a longer stored completion is truncated from the end. null (default) = no cap. Also the loss_type='dr_grpo' loss normalizer, which rejects null rather than normalize by a default nobody chose; pipeline parallelism rejects null too. |
kl_beta | float | 0.0 | Coefficient of the KL penalty to the reference policy; 0 = no KL term. Full fine-tuning scores the reference once at run start and persists it in reference_logps.pt; PEFT scores its base with adapters disabled; native expert LoRA needs an explicit frozen base (Reference model). |
disable_dropout | bool | True | Disable dropouts. |
model_init_kwargs | dict | None | None | On every entry-script path these are model-config overrides, written onto the loaded config's fields before the load: a key that config does not declare raises, and so does dtype/torch_dtype (the run's precision comes from bf16 / fp32_*). They are model-loading kwargs only where a trainer is constructed programmatically with the model as a path string; beside an already-built model, a non-null value raises. |
dataset_num_proc | int | None | None | Dataset preprocessing worker processes; unset means the toolkit default (HALO_DATASET_NUM_PROC), not one worker. |
advantage_method | "z_norm" | "minmax" | "quantile_norm" | "quantile_uniform" | "robust" | "quantile_norm" | How rewards map to advantages: z_norm (z-score), minmax and quantile_uniform (to [-1, 1]), robust (median/IQR, outlier-resistant). |
best_completion_emphasis | float | str | 0.0 | Boost for the best completion(s) per group: 0.0 = off, a float > 1.0 = fixed boost (1.5 = +50%), "auto" = std-adaptive (3.0 + 2.0 × std / (1 + std): 3.0 at std 0, rising toward 5.0), under every advantage method. A value in (0.0, 1.0] raises: the consumer applies the factor only above 1.0, so it would be a silent no-op. |
min_log_prob | float | None | -3.0 | Min log prob for negative-advantage tokens (numerical stability). |
initial_min_log_prob | float | None | None | If set, linearly schedules min_log_prob from here to the final value over training; below 0, and refused with min_log_prob: null. A start near minus the mean CE loss on the SFT data works. |
loss_type | "grpo" | "bnpo" | "dr_grpo" | "bnpo" | grpo (per-sequence avg then weighted mean across groups), bnpo (global weighted token avg), dr_grpo (normalized by effective batch size + max completion length). |
policy_gradient_formulation | "prob_weighted" | "reinforce" | "prob_weighted" | prob_weighted: L = -(π(a|s) * advantage), probability-weighted REINFORCE, more conservative offline. reinforce: L = -(log π(a|s) * advantage), equal weight, more aggressive but less stable offline. |
drop_degenerate_groups | bool | False | Drop groups whose completions all scored exactly the same reward (or with < 2 completions) at tokenization: their advantages are all 0, so they spend forward compute only to dilute the loss normalizer. Near-ties are kept (the rank methods train them at full scale). Applies to the train and eval splits alike; raises if the whole dataset would be dropped. |
use_chunked_grpo_logprobs | bool | False | Vocab-chunked completion log-probs instead of full [B, T, vocab] logits: the shared ChunkedLogprobsArguments switch; covers the policy and the kl_beta > 0 reference forward. CP always takes the chunked path, as do DeepSeek-V4 and Mistral 4 (Row positions). See Offline GRPO. Off, construction refuses a full-logits plane the device's free memory cannot hold; the check is skipped with max_completion_length unset (the logits wall). Refused under PP, whose pipeline drives its own stages. |
SMPOConfig
SMPOTrainer (SMPO) — reference-model-free, margin-based preference optimization. Source: halo/configs/preference.py
The class also overrides four TrainingArguments defaults: learning_rate=1e-6, logging_steps=10, gradient_checkpointing=True, and bf16 (declared bool | None, resolved in __post_init__ to not fp16).
| Parameter | Type | Default | Description |
|---|---|---|---|
beta | float | 1.2 | Temperature controlling loss sensitivity. Higher = stronger preference signal. |
target_margin | float | 0.35 | Target margin for the chosen/rejected log-prob ratio; >= 0. |
loss_type | "sigmoid" | "hinge" | "ipo" | "smooth_lower_bound" | "smooth_lower_bound" | sigmoid (margin violation), hinge (hard margin), ipo (squared), smooth_lower_bound (squared hinge, smooth near boundary; recommended). |
chosen_sft_ratio | float | 0.8 | Weight of chosen in the SFT loss component. 1.0 = chosen only, 0.0 = rejected only. |
use_margin_schedule | bool | True | Linearly raise target_margin from initial_margin. |
initial_margin | float | 0.01 | Starting margin when scheduling; must be below target_margin, and refused off its default with use_margin_schedule off. |
lower_clip_percentile | float | None | 0.02 | Clip low token log probs in rejected, in the margin term only (recommended 0.01–0.05); in (0, 0.5]. Refused under PP, which sees a microbatch's tokens only, so a PP run must set this default to null. |
upper_clip_percentile | float | None | None | Clip high token log probs in chosen, in the margin term only; in [0.5, 1). Refused under PP. |
min_log_prob | float | None | -2.3 | Absolute min log-prob threshold for rejected tokens, applied after the percentile clip; < 0. |
padding_free | bool | False | Flatten the batch into one varlen sequence instead of padding it. Needs a varlen Flash kernel (FA2/FA3/FA4) and raises on any other — including the sdpa the script defaults to off Blackwell under reset_sinks: true, which is what makes it need an explicit attn_implementation there (SMPO). Incompatible with CP, PP and VLM mode (PP rejected combinations). |
allow_cross_document_linear_attention | bool | False | Train padding_free rows on a model whose linear-attention layers receive no document boundary (Ring's Lightning Attention-2), or whose short convolutions receive none (Inkling); their state then carries from each row into the next. Off, such a run is refused (Document isolation). |
max_length | int | None | 1024 | Total budget (prompt + completion), split into the two shares below by resolve_length_budget(). null → model context window. |
max_prompt_length | int | None | None | Prompt share; longer prompts truncate per truncation_mode. null (default) = half of max_length, so the split scales with a context-resolved budget. VLM prompts are never truncated — one that expands past this raises at collation. |
max_completion_length | int | None | None | Completion share; longer completions truncate from the end. The budget includes the EOS, so a completion that reaches it ends without one, as in offline GRPO. null (default) = the max_length − max_prompt_length remainder. An explicit value must still fit that remainder (else the config raises). |
truncation_mode | "keep_end" | "keep_start" | "keep_end" | Which end of an over-long prompt to keep (keep_end = the tokens nearest the completion); completions always truncate from the end. Refused off its default on a VLM run, which truncates no prompt. |
disable_dropout | bool | True | Disable dropout. |
dataset_num_proc | int | None | None | Dataset preprocessing worker processes; unset means the toolkit default (HALO_DATASET_NUM_PROC), not one worker. |
model_init_kwargs | dict | None | None | Model-config overrides applied before the load — see OfflineGRPOConfig. |
TeacherDistillationConfig
DistributedTeacherDistillationTrainer. Source: halo/configs/distillation.py. The teacher and the conversation-rendering fields live in TeacherDistillScriptArguments below.
| Parameter | Type | Default | Description |
|---|---|---|---|
distill_loss | "kl_divergence" | "reverse_kl" | "mse" | "soft_cross_entropy" | "cosine_similarity" | "jensen_shannon" | "slim" | "kl_divergence" | Teacher-distillation divergence. |
distill_temperature | float | 1.0 | Softmax temperature — forwarded only to the losses that declare it (kl_divergence, reverse_kl, soft_cross_entropy, jensen_shannon, slim). Refused off its default under mse and cosine_similarity, whose definitions take none. Must be > 0. |
distill_jsd_beta | float | 0.5 | β of the generalized jensen_shannon loss, in [0, 1]: 0 = forward KL, 1 = reverse KL (Generalized JSD). Refused off its default under any other distill_loss. |
distill_topk | int | None | None | Score the loss on the teacher's top-k tokens plus a tail bin, a lower bound on the full-vocab loss that saves no memory (Top-k). An integer >= 1, below the compared vocabulary width; kl_divergence and soft_cross_entropy only. None = the full vocabulary. |
distill_alpha | float | 1.0 | Weight of distillation vs CLM loss. 1.0 = distillation only. Must be in [0, 1]. |
apply_hard_labels | bool | False | Gate the distillation term per token on the gold token (Teacher Distillation). Refused at trainer construction with distill_loss: slim, which applies its own gold-token weight. |
max_length | int | None | 2048 | Maximum tokenized sequence length. Over-length conversations are dropped, not truncated; null → the student's context window. |
dataset_num_proc | int | None | None | Dataset preprocessing worker processes; unset means the toolkit default (HALO_DATASET_NUM_PROC), not one worker. |
TeacherDistillScriptArguments
Source: halo/configs/distillation.py
| Parameter | Type | Default | Description |
|---|---|---|---|
teacher_model | str | None | None | Teacher model name or path. Required — an unset or blank value raises at parse time, not at model load. |
teacher_model_revision | str | None | None | Hub revision for the teacher. It is usually a different repo from the student, so it cannot share model_revision (one repo's commit sent to another 404s). None = the teacher repo's main. |
conversation_field | str | "messages" | Dataset field with conversations; a null or blank value is refused. |
images_field | str | None | None | VLM only: image column injected into the first user turn (same semantics as the SFT field). |
freeze_vision_tower, min_pixels, max_pixels | As the SFT fields. | ||
system_prompt | str | None | None | System prompt if none in dialogue. |
model_supports_system_role | bool | True | If False, system prompt is prepended to the first user message. |
train_on_completions_only | bool | True | Mask prompt tokens, train only on assistant completions (text and VLM paths alike). |
assistant_message_template | str | None | None | The rendered assistant-turn prefix of the model's chat template; required when train_on_completions_only is on (no default fits every template — a missing or mismatched marker raises at startup) and refused when it is off. |
interleaved_thinking | bool | False | Pass clear_thinking=False to apply_chat_template (GLM-family templates). Text-only: the VLM path raises (no supported VLM template has the switch). |
ClassificationConfig
ClassificationTrainer. Source: halo/configs/classification.py
| Parameter | Type | Default | Description |
|---|---|---|---|
max_length | int | None | 1024 | Truncation length at tokenization (scripts/training/classification.py). A raw text/label dataset passed straight to the trainer is instead tokenized untruncated and filtered to it. Batches pad to the batch longest, not to this. null → model context window. |
disable_dropout | bool | True | Disable dropout. |
dataset_num_proc | int | None | None | Dataset preprocessing worker processes; unset means the toolkit default (HALO_DATASET_NUM_PROC), not one worker. |
remove_unused_columns | bool | False | Prune the columns the model's forward does not take, at the dataloader; the trainer tokenizes a raw text/label dataset before that. |
loss_type | "cross_entropy" | "focal" | "label_smoothing_ce" | "cross_entropy" | label_smoothing_ce is single-label only (softmax CE) and raises on a multi-label dataset — use focal or class_weights there. |
focal_gamma | float | 2.0 | Focusing parameter (focal loss). Refused off its default unless loss_type="focal". |
focal_alpha | float | None | None | Positive/negative balancing factor for focal loss, applied as alpha_t = alpha*y + (1-alpha)*(1-y). Multi-label (sigmoid) heads only — on a single-label softmax head every element has exactly one target, so a scalar alpha would be a uniform loss rescale rather than balancing, and it is rejected there, as it is unless loss_type="focal". Use class_weights / derive_class_weights for per-class balancing. |
label_smoothing | float | 0.0 | Smoothing epsilon (typical 0.05–0.1), which loss_type="label_smoothing_ce" requires non-zero and every other loss_type refuses. Not to be confused with HF's inherited label_smoothing_factor, which this trainer rejects: it builds its own loss and never consults HF's label_smoother. |
class_weights | list[float] | None | None | Per-class weights by label ID, finite and >= 0, one per label (refused otherwise). Applied to every loss_type (CE, focal, label-smoothed CE) and as pos_weight on the multi-label BCE. |
derive_class_weights | bool | False | Auto-compute balanced class weights from label counts all-reduced across the world group (presharded/TP-safe). Classes absent from training keep weight 1.0. Mutually exclusive with class_weights — setting both raises, as does using it on a multi-label dataset (pass class_weights explicitly there). |
multi_label_threshold | float | 0.5 | Sigmoid threshold for multi-label predictions; refused off its default on a single-label dataset. |
compute_per_class_metrics | bool | False | Log per-class precision/recall/F1. Single-label only — refused on a multi-label dataset. |
compute_auc_roc | bool | False | Compute AUC-ROC (multiclass = one-vs-rest macro). |
compute_mcc | bool | True | Compute Matthews Correlation Coefficient. Single-label only — ignored on a multi-label dataset. |
EmbeddingConfig
EmbeddingTrainer; extends sentence-transformers' SentenceTransformerTrainingArguments. Source: halo/configs/embedding.py
| Parameter | Type | Default | Description |
|---|---|---|---|
loss_type | "mnrl" | "cached_mnrl" | "cosent" | "angle" | "cosine_similarity" | "triplet" | "contrastive" | "online_contrastive" | "batch_all_triplet" | "batch_hard_triplet" | "mnrl" | Sentence-transformers loss (Loss functions). |
loss_scale | float | 20.0 | Scale / inverse temperature of mnrl, cached_mnrl, cosent and angle; refused off its default under any other loss_type. |
cached_mnrl_mini_batch_size | int | 32 | Rows per gradient-cached sub-forward; refused off its default under any loss_type but cached_mnrl. |
matryoshka_dimensions | list[int] | None | None | Matryoshka truncation dims, integers >= 1. |
matryoshka_weights | list[float] | None | None | Per-dimension weights, finite and >= 0, same length as the dimensions; refused without them. |
pooling_mode | "mean" | "cls" | "max" | "lasttoken" | "weightedmean" | "mean_sqrt_len_tokens" | "mean" | Pooling over the token embeddings. |
normalize_embeddings | bool | True | L2-normalize output embeddings. |
max_length | int | None | 512 | Truncation length; null or non-positive → the backbone's context window. |
disable_dropout | bool | False | Disable dropout while training. |
batch_sampler | str | "batch_sampler" | batch_sampler, no_duplicates, no_duplicates_hashed or group_by_label (Embedding). |
dataloader_drop_last | bool | False | Forced True by sentence-transformers on any multi-process launch (Embedding). |
EnvironmentConfig
Environment selection, reward terms and turn budget for Async GRPO with Environments. Source: halo/configs/grpo_environmental.py
| Parameter | Type | Default | Description |
|---|---|---|---|
environment_type | str | "react_math" | Registry name, case-insensitive (lowercased at parse): react_math, react_search, native_math, native_coding, native_combined, swe, mcp, qa_search, code_contests, codeforces, exam_qa. |
rewards | list[dict] | [{source: environment}] | Reward terms, each {source, name?, weight?, exponent?, ...}, summed into the episode reward as weight × score ^ exponent over a score in [0, 1]: environment (the environment's own grade, logged as reward/objective), judge (requirements for a scoring judge, checks for a veto judge whose fired veto check zeroes every positive component), reward_model; the external terms take view (final / full / judge-only digest) and on_error (invalid / neutral). The environment's per-turn and shaping terms add on top. Parsed at config time; forwarded to the environment as reward_terms. YAML-only. See Reward Terms. |
max_turns | int | null | null | Max environment turns per episode, an int >= 1. null keeps the environment class's own default (per-environment defaults). |
environment_kwargs | dict | {} | Env-specific kwargs passed to the registry factory; a bare header (null) or any other non-mapping is refused. Merged with core settings in to_env_config(). A key the target environment's constructor chain does not bind raises TypeError; max_turns and reward_terms are refused at parse (set the top-level max_turns / rewards, which validate them). |
environment_kwargs keys are per environment. The ones every environment takes (turn shaping, tool caps, requires_answer, carry_reasoning, max_length_cutoff_recoveries) are in Common knobs, reasoning_effort and reasoning_effort_profiles in Effort levels, tool_budgets in Native Tool-Use, and each environment's own on its page (Code Contests, QA Benchmarks, MCP, SWE).
YAML
environment_type: qa_search
environment_kwargs:
search_backend: duckduckgo
include_python_tools: trueAsyncRolloutConfig
DistributedAsyncEnvironmentalGRPOTrainer: Ray workers, rollout-server connections, weight sync, rollout generation, async prefetch. Source: halo/configs/grpo_environmental.py, where the IS-correction and reasoning-cost fields come from the ISCorrectionArguments and ReasoningCostArguments bundles it mixes in, and the sampling and budget fields from RolloutSamplingArguments and RolloutBudgetArguments in halo/configs/rollout_arguments.py. A field it copies into the actors' RolloutConfig (halo/environments/rollout_config.py) defaults to RolloutConfig's own value. Every numeric field also refuses a bool (true would read as 1) and a non-finite value, beside its own range.
| Parameter | Type | Default | Description |
|---|---|---|---|
rollout_actor_budget | int | 256 | Ray environment actors per node: a local Ray (ray_address unset) splits it over that node's rollout ranks (this // rollout ranks on the node), a shared cluster (ray_address set) treats it as the cluster-wide total and splits it over every rollout rank. A rollout rank is every rank, or each TP/ETP group's leader. It replaces num_rollout_workers (per rank, default 64), which the parser refuses with this remedy: set the budget to the old value times the node's rollout ranks, or keep the old value under ray_address, where it was already the cluster-wide total. A rank gets at least 1 actor, and a split under 4 (or under the round's rows) warns once. >= 1. Async, so not a concurrency limit; size it to the env's blocking per-episode cost: CPU for sandboxed envs (bounded by the sandbox gate), remote latency for search/MCP envs. An actor needs 1 free CPU at placement only (Ray Cluster). |
max_concurrent_rollouts | int | None | None | Per-rank in-flight cap, the throughput throttle; server-pool load is this × rollout ranks ÷ servers per round (under TP/ETP only each group's leader collects). Each round in flight holds its own cap: the prefetch_depth prefetched rounds, and a cold or eval round beside them. Refused below 1 when set; null is the larger of a train round's rows (per_device_train_batch_size × steps_per_generation) and eval_rollout_batch_size, so a round runs in one wave. A value below this rank's actor count is raised to it, and one below a train round warns with the wave count it implies. |
eval_rollout_batch_size | int | None | None | Rows per rank in one evaluation rollout round; rows, not prompts, since the eval sampler has already repeated each prompt num_generations_eval times. Eval rounds run without prefetch, so a round's wall time is its slowest episode and per_device_eval_batch_size-sized rounds idle the servers between them; size it to what the servers sustain: this × rollout ranks requests are in flight at once, and a turn that decodes slower than request_timeout allows fails its episode, so a whole split in one round can oversubscribe the servers that carry the training rounds fine. Keep it a multiple of num_generations_eval, at most max_concurrent_rollouts (refused at start), with dataloader_drop_last off (refused at construction). A short last round is padded to whole batches; the padding is neither rolled out nor scored (evaluation). It bounds the loader's batch, not eval peak memory: the loss forward still chunks per_device_eval_batch_size rows, but the round's widest completion sets the padded width. Refused below 1 when set; null leaves the round at the eval batch (per_device_eval_batch_size), whose geometry is checked at construction the same way. |
ray_address | str | None | None | Ray cluster address (head-node:6379). None = a local instance per rank; the actor budget then counts per node, with it set cluster-wide; setup and sizing on Ray Cluster. |
rollout_backend | "vllm" | "sglang" | "vllm" | Rollout engine for generation and weight sync. sglang supports routing_replay: rollout (serve with --enable-return-routed-experts --moe-runner-backend triton), rejects rollout_max_thinking_tokens at config time and the env's carry_reasoning at trainer construction, refuses the families 0.5.22's loaders cannot take an online update for (SGLangWeightSyncClient.UNSERVABLE_MODEL_TYPES, quoted at trainer construction), and must be served from the NCCL-aligned Dockerfile.sglang image with NCCL_CUMEM_ENABLE=1 (the compose default); see Rollout Servers. |
rollout_server_url | str | "http://localhost:8000" | Primary rollout-server URL (whichever engine rollout_backend selects) for weight sync + generation. |
rollout_connection_timeout | float | 120.0 | Seconds to wait for the rollout server. Refused: non-finite or <= 0. |
rollout_server_configs | list[dict] | None | None | Multi-server configs (each a dict with url and optional group_port / group_host, the weight-transfer master address the serving node dials back to, and node, the label single_group orders servers by). Any other key, a missing url, a group_port outside 0–65535, and a non-default rollout_server_url beside a non-empty list raise. |
rollout_sync_topology | "auto" | "per_server" | "single_group" | "per_server" | How the weight-sync groups span the servers: one NCCL group per server, or one over all of them that broadcasts each chunk once; auto picks single_group for two or more servers that report one served model and one set of weight-loading flags and per_server otherwise, logging its choice. A group needs every server to see every server's GPU (One group over every server). single_group refuses a per-entry group_port, differing group_host values and, at formation, local servers beside remote ones with no group_host; node is refused under per_server and auto. Environmental GRPO only: online GRPO talks to one server. |
rollout_weight_precision | bf16 | fp8_block | bf16 | Precision of the weights the rollout servers hold. fp8_block pushes the attention and MLP projections as 128x128 block FP8 (power-of-two scales), for engine memory and SGLang decode at a slower sync than bf16; the server must serve a checkpoint from export_block_fp8.py with the triton MoE backend. Validated families only; SGLang needs SGLANG_ENABLE_JIT_DEEPGEMM=0; refused beside rollout_adapter_sync and QLoRA (FP8 rollouts). Online and environmental GRPO. |
rollout_adapter_sync | bool | false | Serve a LoRA policy as an adapter beside the frozen base: the first sync pushes the base once, each later one loads the adapter alone under a new name that new episodes select. SGLang only, launched with --enable-lora; attention and dense-MLP linears only. A model or server it cannot serve is refused at startup with the reason (Adapter sync). Environmental GRPO only. |
rollout_group_port | int | 51216 | Trainer-side port of the NCCL weight-sync group, on either rollout backend; 0 takes a kernel-assigned port. With several servers it is the base: a rollout_server_configs entry without its own group_port binds rollout_group_port + index, or its own kernel-assigned port when this is 0. |
sync_weights_every_n_steps | int | 1 | Sync weights every N training steps. Refused: < 1. |
rollout_temperature | float | 0.7 | Rollout generation temperature. Overwrites GRPOConfig.temperature (logged) so log-probs are scored at the sampling temperature; the engine's logprobs must carry it too — vLLM --logprobs-mode processed_logprobs, SGLang's default (leave SGLANG_RETURN_ORIGINAL_LOGPROB unset). The startup probe runs against either engine and refuses raw logprobs. Refused: non-finite or <= 0. |
rollout_top_p | float | 0.95 | Rollout top-p. With a sequence-level ratio (the trajectory geometric band isr_geo_band_*, or OPSM's isr_opsm_delta) and a logprob reference renormalized over the cut (vLLM processed_logprobs) it must be 1.0 — or drop the sequence-level ratio, since every uncertain position otherwise reads as drift. Refused outside (0, 1]. |
rollout_top_k | int | -1 | Rollout top-k; -1 = off. Sent on every request, like rollout_min_p and rollout_repetition_penalty: both engines fill an omitted one from the model's generation_config.json. A cut under a sequence-level ratio is refused against a renormalized reference, as for rollout_top_p. Refused: a bool (true would sample greedily), a non-int, 0 (SGLang rejects it) and below -1. |
rollout_min_p | float | 0.0 | Rollout min-p; 0.0 = off. A cut under a sequence-level ratio is refused as for rollout_top_k. vLLM rejects every request with min_p > 0 on a server running speculative decoding (--speculative-config, MTP). Refused outside [0, 1]. |
rollout_repetition_penalty | float | 1.0 | Rollout repetition penalty; 1.0 = off. The sampling log-probs carry it and the trainer's recomputed ones do not, so a value other than 1.0 is refused at startup under a sequence-level ratio (isr_geo_band_*, isr_opsm_delta). Refused outside (0, 2]. |
rollout_max_tokens | int | 32768 | The per-turn generation budget (one /chat/completions call). The multi-turn trajectory accumulates across turns, bounded by rollout_max_episode_tokens where set and by the served model's context window, not truncated (a trajectory exceeding the window fails). Refused: non-finite or <= 0. |
rollout_max_episode_tokens | int | None | None | The most tokens one episode may sample over all its assistant turns, reasoning and visible output together, recovery turns included; None = only the per-turn cap and max_turns bound it (max_turns × rollout_max_tokens). Enforced by the engine per turn and never stated to the model: a turn's max_tokens is the smaller of its own cap and what the episode has left, its reasoning cap shrinks with it so the turn keeps its answer room (rollout_max_tokens less its reasoning cap, or rollout_max_answer_tokens where smaller), and an episode left with less than that room starts no further turn and ends truncated, priced like a max_turns overflow. Holds on both engines; the eval scripts take it from --training_config. Logged as episode/output_budget_exhausted. Refused: a bool, a non-int, or a value below rollout_max_tokens (one whole turn must fit). See Trajectory length. |
rollout_max_thinking_tokens | int | None | None | Per-turn reasoning-token cap for reasoning models (vLLM thinking_token_budget): caps the chain-of-thought, then forces the answer out of the rest of the turn (its answer room). Every effort level runs under the same cap, and it is never stated to the model. None = rollout_max_tokens less rollout_max_answer_tokens caps reasoning where that is set, else nothing does. Needs a reasoning parser on the server (--reasoning-parser qwen3 for Qwen3.x, the openai_gptoss plugin for gpt-oss): without one vLLM answers thinking_token_budget with a 400 on every request, so rollouts error instead of generating. Refused unless an int in [1, rollout_max_tokens): at or above rollout_max_tokens, which bounds the whole turn, the answer room is 0 and every turn is cut mid-reasoning. Rejected under rollout_backend: sglang for every model: the trainer wires neither of SGLang's budget mechanisms, and harmony models have none server-side. It needs the IS correction where rollout_reasoning_end_token resolves (see train_on_sampled_tokens). See Reasoning cap. |
rollout_max_answer_tokens | int | None | None | The most a turn may generate past its reasoning cap; None = rollout_max_tokens alone bounds the turn. A turn's max_tokens becomes the smaller of rollout_max_tokens and its reasoning cap (rollout_max_thinking_tokens, else rollout_max_tokens less this) plus this, so a turn the engine force-closed keeps this much room and no more; under rollout_max_episode_tokens a turn starts only while the episode has its answer room left (the smaller of this and rollout_max_tokens less the turn's reasoning cap). Never stated to the model. On SGLang, which enforces no reasoning cap, the cap it leaves is dropped from the request and the turn stays bounded by rollout_max_tokens alone. Refused unless an int in [1, rollout_max_tokens): at or above it the bound shrinks no turn. See Answer room. |
rollout_reasoning_end_token | str | "</think>" | The string the server's reasoning parser ends reasoning with, encoded as vLLM encodes it (Qwen3.x, GLM-4, Step-3.7 and LFM-2 </think>, Gemma 4 <channel|>, gpt-oss <|end|><|start|>assistant<|channel|>final<|message|>). Wherever a vLLM reasoning cap can bind, the trainer gives a run of its ids ratio 0 where the engine forced it (every one at sampling log-prob 0) (Objective); a marker the tokenizer does not write (no added token in its encoding) is refused at construction where a vLLM cap can bind, since those closes would stay in the loss, and only warns on a soft-only run. Where it is one token, a turn's reasoning is counted as its sampled ids up to and including it (all of them for a turn cut before closing its reasoning), the count episode/thinking_cap_turns reads; under a marker of several tokens the count and the metric are absent. |
rollout_stop_tokens | list[str] | [] | Special-token strings that end a turn (resolved to the engine's stop_token_ids; a name the tokenizer does not know raises at trainer construction and in the eval scripts). Set to the model's tool-call terminator so a turn stops when the model emits its call; needed for a terminator that is not an eos (e.g. gpt-oss <|call|> under harmony-disabled serving). Empty = only eos stops a turn. |
rollout_chat_template_kwargs | dict | {} | Chat-template variables sent with every rollout request (chat_template_kwargs) and applied to the trainer's own renders, so both sides see one template state. The stock Qwen3.6 template reads preserve_thinking: with the env's carry_reasoning, the carried previous-turn reasoning stays rendered past a user message (a recovery nudge). reasoning_effort is refused here: it is per episode and travels as the request's top-level field (SGLang also gets an exact nested copy, the spelling it lets override that field). A profile key (reasoning_budget, reasoning_price) is refused here too, at trainer construction. |
train_on_sampled_tokens | bool | True | Train on the server's actual sampled token ids (each assistant turn a training row sharing the trajectory advantage) instead of re-tokenizing a chat-template re-render, which eliminates re-tokenization mismatch. On vLLM this needs the server's --return-tokens-as-token-ids (passed by docker-compose.vllm.yml); on SGLang the capture is per-request and needs no server flag. Falls back to a single re-tokenized row for any trajectory with an uncaptured turn. Off, the IS correction is disabled. A run whose vLLM reasoning cap can bind (rollout_max_thinking_tokens, or rollout_max_tokens less rollout_max_answer_tokens) and whose rollout_reasoning_end_token resolves is refused at construction without the correction (this and vllm_importance_sampling_correction), since the reasoning closes the engine forces leave the loss only through its ratio. |
max_train_row_tokens | int | None | None | Longest training row (prompt + completion tokens) a training rank takes; None = no cap beyond the context check. A per-turn row over it is left out of the batch while the episode's other turns still train; a whole-trajectory row over it trains as a zero-weight row. Logged as sampling/rows_over_cap_frac: rows left out over rows left out plus rows that train. A memory bound for the training ranks, below the context check (which rejects rows the served model could not have produced). Refused: a bool, a non-int, a value below 1, or one at or below rollout_max_tokens, since a row is prompt + completion and such a cap would leave out every turn that used its budget. |
drop_degenerate_groups | bool | True | Drop GRPO groups whose completions all settled the same environment reward (grade, shaping and external scores; the trainer's reasoning price and floor excluded: judged with them, a group tied on everything else would train on its reasoning lengths alone). Restores effective batch size on sparse verifiable rewards; logged as sampling/degenerate_group_frac. |
balance_token_mass | bool | False | Scale down the heavier sign of each generation round's advantages so its token-weighted advantage mass (each token weighted by its truncated IS ratio, across all ranks) nets to zero, the negative mass of untrainable turns, which train on a negative advantage only, included. Requires loss_type cispo, dapo or dr_grpo, top_entropy_quantile: 1.0 and no off_policy_mask_threshold (raises otherwise). Logged as advantage/net_token_mass (the trainable rows' share before balancing, also with the knob off), advantage/token_mass_scale and advantage/negative_only_mass (the untrainable rows' share of the trained mass). See Token-mass balance. |
zero_gradient_round_limit | int | None | 50 | Raise after this many train rounds in a row in which no token anywhere carries a policy gradient (every group tied or dropped, or every token masked) while no earlier round of this launch has: raise it for a reward that sparse. Past the first round that trained, a streak this long only warns, as a policy that nearly always or nearly never solves ties for long stretches and still trains. Warns on the first round and logs sampling/zero_gradient_round. Positive; null only logs. |
episode_error_rate_limit | float | None | 0.5 | Raise once a train round's world-wide episode/error_rate (episodes that errored or timed out, which train as masked rows) is over this for episode_error_round_limit rounds in a row: the batch is then mostly the episodes that finished fast, a bias toward short answers, or the rollout path is down. Counted in the checkpoint's sidecar, so a resume keeps the streak. Refused outside [0, 1); null = off. |
episode_error_round_limit | int | 3 | Consecutive train rounds over episode_error_rate_limit that raise. Positive. |
tool_parser_watch_rounds | int | None | 2 | On a tool environment, raise after this many rounds hold the template's rendered tool-call opening in an assistant turn's text while the run has parsed no call anywhere (tool parsers). A turn the engine cut is no evidence. Positive; null turns the watch off. |
truncation_alarm_rate | float | None | 0.25 | Warn when a rollout round's episode/truncation_rate (the share of its episodes ended by the max_turns cap, the episode output budget, or a cut turn past its recoveries) rises over this, and log episode/truncation_alarm (1 over, 0 under) every round; the warning repeats only after the rate drops back under. Refused outside [0, 1); null = off. |
isr_geo_band_min / isr_geo_band_max | float | None | None | Trajectory geometric-mean band: mask a whole trajectory when exp(mean log-ratio) over its corrected tokens leaves the band. Set both, with 0 < min < 1 < max; one without the other raises at parse, as every isr_* bound does. Requires the IS correction (raises otherwise), like every isr_* knob below. Size it above the vLLM↔trainer numerical floor (the trust-region masks); a tighter band masks every step. |
isr_veto_min | float | None | None | Catastrophic-token veto: mask a whole trajectory when any corrected token's raw ratio falls below this. Refused outside (0, 1). |
isr_opsm_delta | float | None | None | Off-policy sequence masking: mask negative-advantage trajectories whose |mean log-ratio| exceeds this many nats; positives never masked. Refused: <= 0. |
isr_engine_reference | bool | False | Re-score every training row on the rollout engine under the just-synced weights and feed the mask stages the engine's current-vs-sampling log-ratio: the trainer↔engine disagreement drops out, the engine's own decode-vs-prefill disagreement stays in the reference; the IS weight stays trainer-vs-sampling. Logs sampling/engine_logratio_mean, sampling/numerics_logratio_mean, sampling/engine_rescore_coverage, sampling/engine_rescore_miss_frac. Requires the IS correction, rollout_temperature / rollout_top_p of 1.0 and rollout_top_k / rollout_min_p / rollout_repetition_penalty off; vLLM re-scores through the completions prompt_logprobs echo, SGLang through /generate with logprob_start_len. The vLLM server needs headroom for the fp32 vocabulary log-softmax that pass materializes; sizing on Rollout Servers (isr_engine_reference headroom). See the trust-region masks. |
isr_engine_rescore_per_server | int | 128 | Re-score requests in flight on each rollout server under isr_engine_reference, summed over every rank: each rank keeps at most its share (this ÷ world size, at least one) in flight on each server, so a server's load does not grow with the world size; below the rank count a startup warning names the load each server still takes. Refused with isr_engine_reference off. A request that misses its deadline reads the trainer's log-ratio and counts in sampling/engine_rescore_miss_frac. |
skip_update_masked_frac | float | None | None | Trust-region circuit breaker: when more than this fraction of IS-corrected trajectories are fully masked by the stages above, or more than this fraction of corrected tokens (the masked trajectories are the long ones), the round's advantages are zeroed and every optimizer step the generation round feeds is skipped: gradients dropped to None, so Adam's momentum cannot step the weights either; the KL term goes with the round. Logged as sampling/update_skipped with the two fractions (sampling/is_masked_traj_frac, sampling/is_masked_token_frac). Refused outside (0, 1]. Requires the IS correction and at least one mask stage (isr_geo_band/isr_veto/isr_opsm); raises otherwise, since nothing would ever be masked. |
score_centering_top_k | int | None | None | Candidates the SGLang rollout server returns per generated token for score centering (arXiv:2609.20807): each token's gradient subtracts their importance-weighted score-function mean, so on a constant reward the drift a sampler-vs-trainer mismatch causes cancels where the candidates hold nearly all of the sampler's mass on the mismatched tokens (all of it at the full vocabulary). None = off; raise it until score_centering/sampler_head_mass_mean is near 1. Refused at construction, and by the script before its preflights, on rollout_backend: vllm, train_on_sampled_tokens: false, a filtered sampler, use_chunked_grpo_logprobs: false, loss_forward_importance_weights, balance_token_mass or isr_engine_reference; at construction also on num_iterations > 1, misaligned accumulation, beta != 0 or importance_sampling_level other than token. See Score centering. |
score_centering_weights | "none" | "tis" | "mis" | "tis" | The importance weight of the centered gradient: none, truncated at vllm_importance_sampling_clip_max, or kept inside [score_centering_mis_low, score_centering_mis_high] and else zero. It replaces the token ratio. Refused while score_centering_top_k is unset. |
score_centering_mis_low / score_centering_mis_high | float | 0.5 / 5.0 | Bounds of the mis weight on the trainer-to-sampler ratio, 0 < low <= high. Refused at a non-default value unless score_centering_weights is mis. |
early_stop_entropy_band | list[float] | None | None | Early stop: end training once TRL's entropy is outside [low, high] on early_stop_patience readings in a row. That metric counts a micro-batch whose loss tokens were all dropped as 0, so under drops it reads below the policy's entropy. Refused unless 0 <= low < high. See Early stop. |
early_stop_logratio_gap | float | None | None | Early stop: end training once the magnitude of sampling/logratio_mean is above this many nats per token on early_stop_patience readings in a row (one per generation round; a step reusing its round carries none and leaves the count). Refused unless finite and positive; raises without the IS correction, which logs the gap. |
early_stop_on_skipped_updates | bool | False | Early stop: end training once the breaker skipped every update on early_stop_patience readings in a row (one generation round each at logging_steps: 1). Raises without skip_update_masked_frac. |
early_stop_patience | int | 3 | Breaching readings in a row an early-stop condition needs; a reading is one training log line, the window's mean under logging_steps > 1. Refused below 1, and at a non-default value when no early-stop condition is set. |
scale_rewards_std_floor | float | 0.0 | Floor (reward units) on the advantage-scaling std divisor: divide by max(std, floor), bounding degenerate-batch noise amplification. 0 = off. Refused: non-finite or negative, and above 0 with scale_rewards: none, which divides by no std. |
routing_replay | "none" | "recompute" | "rollout" | "none" | MoE routing replay. recompute (R2) captures each EP MoE layer's top-k selection in the no-grad logprob-recompute pass and replays it in the update/GC forwards, re-deriving gate weights from live router scores, which leaves the trainer-versus-engine gap alone; it raises in a config that never runs that pass (num_iterations: 1, aligned accumulation, IS correction off, beta: 0). rollout (R3, experimental) replays the rollout engine's per-turn routing through both passes, which narrows that gap (measured); it needs train_on_sampled_tokens and a capture-capable server: vLLM 0.32.0 with --enable-return-routed-experts and --moe-backend triton (the backend weight sync needs), or SGLang 0.5.22 with --enable-return-routed-experts and --moe-runner-backend triton (the triton_kernel/flashinfer runners bypass the capture hook). EP-wrapped MoE only; Gemma4 and Zaya are rejected. |
packed_microbatch_tokens | int | None | None | Pack each training round's rows into micro-batches of up to this many attended tokens per rank, in place of one micro-batch per row; null keeps today's path. The PackedMicrobatchArguments bundle, shared by the environmental and the online script (SDPG excepted). The world runs the same number of micro-batches (at most steps_per_generation, in the window's last slots), and row-mean losses are rescaled so the objective is the per-row one. Refused at construction without use_chunked_grpo_logprobs: true, at per_device_train_batch_size != 1, under a loss_type outside grpo, sapo, dr_grpo, luspo, dapo, cispo, top_entropy_quantile < 1, entropy_coef != 0, length_ordered_microbatches: true (environmental), misaligned accumulation, an evaluation batch other than 1, a family whose rows cross documents, a backend that is not varlen flash, or a budget the EP dispatch cannot carry, and use_sdpg: true (online). See Packed micro-batches. |
length_ordered_microbatches | bool | None | None | Deal each training round's rows into its micro-batches longest first, every rank alike, in place of TRL's per-rank shuffle, and run the round's no-grad forwards (recompute, KL reference) in the same order. Only the row order changes. Unset = on where that holds: accumulation aligned to the generation round, a loss_type other than bnpo, top_entropy_quantile: 1.0, no entropy bonus, and rows forwarded one per backbone call (use_chunked_grpo_logprobs: true, at per_device_train_batch_size: 1 or under FA4); true raises at construction where it does not; false keeps TRL's shuffle. See Memory and Throughput. |
loss_forward_importance_weights | bool | False | Take the importance weight (and the KL tail clamp) from the loss forward's own detached log-probs in place of a separate no-grad pass over the round's rows. true raises at construction on balance_token_mass, an isr_* stage, isr_engine_reference, skip_update_masked_frac, routing_replay: recompute, num_iterations > 1, misaligned accumulation, or a run with no IS correction and beta: 0 (the default false keeps the pass). With an early_stop_logratio_gap the logged log-ratio reads loss tokens only. Composes with packed_microbatch_tokens. See Weight from the loss forward. |
episode_timeout | float | 1200.0 | Deadline (seconds) for one whole rollout episode: generation + tool execution + grading, unlike request_timeout, which bounds a single HTTP call. Pause-aware: the seconds the engines spend paused for a weight sync are credited back, so a sync never times an episode out. A timed-out episode is canceled and counted in episode/error_rate, and a straggler holds its peers at the per-step collective for up to this long. It must stay under the NCCL watchdog (DIST_NCCL_TIMEOUT_MINUTES × 60): a larger value raises when training starts, and ≥80% of it warns. The default sits at two thirds of the 30-min watchdog, leaving ~10 min of margin; raise DIST_NCCL_TIMEOUT_MINUTES before raising it. Refused: non-finite or <= 0. |
enable_prefetch | bool | None | None | Roll the next rounds out while the current one trains (prefetch). Unset = on with two or more rollout servers, off with one; true prefetches on a single server too; false never. The resulting off-policy staleness, prefetch_depth rounds, is corrected by the truncated vLLM importance-sampling ratio built from the batch's sampling logprobs (clamped by TRL's vllm_importance_sampling_clip_max, default 3.0). |
prefetch_depth | int | 1 | Rounds collected ahead of the step that trains them, together on the engines and delivered in submission order: a round trains prefetch_depth rounds after it was submitted, on samples up to that many rounds old (one round per optimizer step at the default steps_per_generation). Raise it when every step misses (async/prefetch_misses) while the servers drain at each round's tail; the next round's episodes fill them. Costs staleness, more rounds in flight across each sync, engine KV for every round in flight, and prefetch_depth cold rounds at the start of a run, each trained twice (Prefetch). Refused: below 1, not an int, or above 1 on a run that does not prefetch (the refusal names enable_prefetch: false, or the opt-in a one-server run left unset). |
use_chunked_grpo_logprobs | bool | False | Compute completion log-probs from the hidden state + a vocab-chunked softmax instead of full [B, T, vocab] logits, bounding the loss-forward peak by chunk size, not B·T·vocab. For large-vocab models on long trajectories. See Chunked log-probs. Off, construction refuses a full-logits plane the device's free memory cannot hold (the logits wall). |
balance_rank_tokens | bool | False | Deal each training round's rows across ranks by attended tokens, every rank keeping its row count, in place of the contiguous slice the rollout hands each. The RankBalanceArguments bundle, online only. Refused under TP/ETP, a loss_type outside grpo, sapo, dr_grpo, luspo, dapo, cispo, top_entropy_quantile < 1, entropy_coef != 0, misaligned accumulation, a single rank; a round with image tensors or any per-row list raises. See Rank balance. |
log_phase_times | bool | False | Log each optimizer step's wall time by phase as phase_time/* (Phase times). Costs a few CUDA events per micro-step and one all-reduce per log. |
reasoning_price_cap | float | 0.1 | Cap of the reasoning price per episode, in reward units, so a long trace cannot outweigh the task reward. The price itself is per effort level: a level's profile (environment_kwargs.reasoning_effort_profiles) charges reasoning_price per 1,000 reasoning tokens an episode spends past its reasoning_budget, summed over its turns, -min(reasoning_price_cap, price × (tokens − budget) / 1000) (Price). Finite and > 0; refused at trainer construction at a non-default value while no level the environment draws sets a positive reasoning_price. Logged as reward/reasoning_price. |
reasoning_floor | float | 0.0 | Weight of the reasoning under-use floor (0 = off): a baseline episode that earned no objective and reasoned, summed over its turns, under its level's reasoning_budget pays -weight × shortfall / budget; a solve pays nothing. The one length term that pays for more reasoning (Floor). An episode pays it or the price, never both, so keep each below what the environment charges for the decisions it prices (a resubmission, in code contests). Refused at trainer construction while a drawn level sets no reasoning_budget above 0, or under a reward that prices no success above zero (no environment term, or one of weight ≤ 0). Logged as reward/reasoning_floor. Finite and >= 0. |
reasoning_terms_in_failed_groups | bool | False | Charge the reasoning price and reasoning_floor in a training group none of whose baseline episodes earned the objective (a positive reward/objective, which a veto zeroes). Off, such a group is charged neither, so the length terms cannot become most of its advantage (Failed groups); eval episodes are always charged. Off with the price on, a reward that prices no success above zero is refused at trainer construction; set while no drawn level prices reasoning and the floor is off, refused as inert. |
model_name | str | None | None | Model name for /v1/chat/completions. null is filled with model_name_or_path at script start, so a request always names a model. |
request_timeout | float | 120.0 | HTTP timeout per rollout-server request (seconds). Pause-aware like episode_timeout: a request in flight across a weight sync is credited the pause, so a sync never expires it (async/requests_expired_in_sync counts the ones that overran even so). One request is one turn, so size it to at least twice the per-turn cap (the turn's total) divided by the per-sequence decode speed at peak concurrency (Sizing a run); a slower turn fails its episode. Refused: non-finite or <= 0. |
max_retries | int | 3 | Max retries of one turn's rollout-server request after a transient failure (exponential backoff); the turn is re-issued, never the episode. It also caps the re-issues of a turn the engine aborts (Engine aborts). Refused: negative. |
retry_base_wait | float | 1.0 | Base wait (seconds) for backoff. Refused: non-finite or negative. The whole retry budget — (max_retries + 1) × request_timeout plus backoff — is only warned about at 80% of the NCCL watchdog, never refused: past it a stuck rollout keeps retrying until the watchdog fires and hangs the per-step barrier instead of giving up. |
Script arguments
CommonScriptArguments
Base shared by all training scripts, mixing in TorchProfilerArguments (enable_torch_profiler, profiler_*), CheckpointArguments and ModelLoadArguments (their flags). Source: halo/configs/script_arguments.py
| Parameter | Type | Default | Description |
|---|---|---|---|
dataset | str | list[str] | "/path/to/dataset" | An org/name[:config][@split] hub id, an s3:// URI, a save_to_disk directory, or a single .jsonl/.json/.parquet/.arrow/.csv file. A list of any of these for multi-dataset training. A bare org/name is a hub id, so a local directory needs ./ or an absolute path, and an org/name that also names a local directory is refused (Dataset path formats). Placeholder default with no validation; an unset value fails at dataset load. |
dataset_ratio | float | None | None | Fraction of each dataset to take, in (0, 1] (0 is refused). A YAML list gives one ratio per dataset entry; that form is YAML-only (the field is annotated float so HfArgumentParser can build, so a list cannot be spelled on the CLI). A ratio below 1 is refused on a pre-processed dataset, whose rows were fixed at preparation. |
test_size | float | None | None | Test split proportion (e.g. 0.05). Leave empty to keep the dataset's own split: set on a dataset that already carries a test split (or, with none, a validation split, read as test), the two are concatenated and re-split at this proportion, so a curated split is not preserved. Ignored with a warning on a sharded or pre-processed dataset, whose split was fixed at preparation time. |
project_name | str | per-script | wandb/clearml project. __post_init__ refuses null and blank, then replaces the "default-project" sentinel with the script's own (e.g. SFT → sft-tuning). |
pad_token / bos_token / eos_token | str | None | None | Override special tokens. An eos_token whose id differs from one a module copied from the config when it was built (Qwen3.8-Flash-Next's n-gram embedding) is refused: the export would record the new id while the model computes with the copy. |
chat_template | str | None | None | Path to a .jinja/.jinja2/.j2 file, or template string. Applied where the tokenizer ships none; one that differs from a shipped template raises without force_chat_template. |
force_chat_template | bool | False | Replace the tokenizer's own template with chat_template, which it needs (set alone, it raises). A VLM run's processor takes it too (Chat templates). SFT on a dataset prepared with --chat_template needs it wherever the checkpoint ships a template, or the load raises (Pre-Processing). |
added_special_tokens | list[str] | None | None | Additional special tokens, added to the tokenizer's existing extra-special set rather than replacing it (transformers' own default replaces, which would drop every control token the checkpoint shipped from all_special_ids and from the exported tokenizer_config.json). Re-listing a token already present is a no-op. Applied before the pad_token/bos_token/eos_token roles, which read an id back. When the tokenizer then holds more ids than the embedding has rows, the embedding (and an untied head) grows to its length; an adapter run saves those rows with its adapters (PEFT saving). DeepSeek-V4, whose hash routers index gate.tid2eid by token id, refuses that grow. |
tokenizer_backend | "hf" | "gigatoken" | "hf" | Text→ids backend; gigatoken is an optional extra whose token IDs are verified identical to hf's at startup. Resolved in configure_run_tokenizer, so every training method honors it; embedding rejects it (SentenceTransformer owns tokenization) — see Dataset Pre-Processing. |
tools_field | str | None | None | Dataset field with tool definitions passed to apply_chat_template. Reward modeling aliases it onto tools, the one column TRL's RewardTrainer templates. Everywhere the render cannot carry it, it is rejected rather than parsed and dropped: outright by KTO and embedding; by async GRPO, whose rollout schema comes from the environment's own tool registry; and on the vision arms of SMPO, DPO and reward modeling, whose pair render templates without tools=. |
unfreeze_layers_patterns / freeze_layers_patterns | list[str] | None | None | Layer-name patterns to unfreeze / freeze (freeze applied after unfreeze). A pattern that matches nothing raises on either knob — unfreeze would leave the model fully frozen, freeze would leave what it named training. They match different things: unfreeze_layers_patterns is fnmatch against full module names, freeze_layers_patterns against full parameter names. Both are refused on an adapter run (LoRA or native expert LoRA), which trains only the adapters. Under pipeline parallelism each stage holds its own layers re-based to index 0, so a layer-indexed pattern (model.layers.3.*) raises; use index-free ones. |
enable_efficiency_metrics / gradient_checkpointing_plan / enable_moe_metrics / moe_balancing / router_balancing_rate / num_full_model_params / enable_torch_profiler / profiler_* | — | — | See Performance & balancing flags. |
max_consecutive_nonfinite_steps | int | 3 | An optimizer step whose global gradient norm is NaN or inf is never applied: its gradients are discarded, weights and optimizer moments stay as they were, and the step counter and LR schedule advance. This many such steps in a row are skipped with a warning (the run logs nonfinite_steps_skipped); the next one raises, as does a non-finite step whose weights are already non-finite. Under AdamWBF16 or Muon the skip happens on the device and the warning or raise follows at the host's late read of the verdict, before the next step's update. 0 raises at the first; negative values are refused at parse time. Under fp16 the guard stands aside only where accelerate's optimizer wrapper steps through the loss scaler (an accelerate-launched DDP or FSDP run), which skips overflow steps itself; on the toolkit's no-wrap path (torchrun FSDP2, EP, TP, ETP, PP) it stays armed. See Trainer Architecture. |
report_mfu_diagnostics | bool | False | Add MFU / S-MFU / achieved-TFLOPS to the headline log. Needs enable_efficiency_metrics (refused without it), whose callback computes them every step regardless; this only controls headline visibility. tokens/s/GPU is the reported throughput metric: MFU's denominator (peak device FLOPS) shifts with GPU, dtype and kernel, and its MoE numerator is ambiguous, so it makes a misleading headline. |
save_completions | bool | True | GRPO family (online / async): write per-step completions and trajectories to <output_dir>/completions/ (+ a wandb completions table): completions_<step>.parquet for online GRPO, a completions_<step>/ directory of one rank<NN>.parquet part per rank for async GRPO (Saved trajectories). Text is rendered from detokenized message content. Independent of TRL's console-only log_completions; ignored by non-generating trainers (SFT, offline GRPO). |
log_decoded_samples | bool | False | Write the first few decoded train/eval samples (skip_special_tokens=False) to <output_dir>/log/{train,eval}_sample.txt. Written on the FS-aware save rank (same gate as run.log: global rank 0 on a shared output FS, each node's local rank 0 when DIST_OUTPUT_SHARED_FILESYSTEM — or the umbrella it falls back to — is 0); datasets without input_ids are skipped. KTO, reward modeling and embedding refuse it. |
SFTScriptArguments
Source: halo/configs/sft.py
| Parameter | Type | Default | Description |
|---|---|---|---|
conversation_field | str | "prompt" | Dataset field with conversations; a null or blank value is refused. |
images_field | str | None | None | VLM only: image column (HF Image, single or list) injected into the first user turn — hub datasets that keep images outside the conversation (FineVision/the_cauldron/Docmatix) train without preprocessing. Setting it declares the run VLM on a multimodal checkpoint, before the dataset is probed (declaration rules). |
freeze_vision_tower | bool | False | Freeze a multimodal checkpoint's vision and audio towers and their projectors, training the text tower on image data. Refused with adapters and on a checkpoint with no tower (Vision-language models). Mixed image and text rows need it only under accelerate-managed FSDP2, or beside a frozen input embedding with gradient checkpointing off (Shard groups). |
min_pixels / max_pixels | int | None | None | Image run: bounds on the pixels per image, for a processor that resizes by pixel count; others refuse them, as does a pre-processed dataset. Exported with the processor. |
system_prompt | str | None | None | System prompt if none in dialogue. |
train_on_completions_only | bool | True | Train only on assistant completions, masking prompt tokens. |
train_on_last_assistant_only | bool | False | Train only on the last assistant message. Requires train_on_completions_only=True. |
tree_packing | bool | False | Pack rows that share a prefix as trees: the prefix is stored once and each row's remainder attends it and itself only. Needs packing and tree_share_markers; GPT-OSS, Qwen3 and Qwen3.5/3.6 on FA4 or FA2, GLM-5 Next on SDPA. See Tree Packing. |
tree_share_markers | list[str] | [] | Tree packing: rows whose message before the final turn is identical up to and including one of these strings are examined together for a shared prefix. |
tree_share_requires | str | None | None | Tree packing: a row is shared only if this text occurs before its marker. |
tree_full_attention | "split" | "gather" | None | None | Tree packing, full-attention layers: split (FA4, except its head_dim-256 kernel) or gather (FA4 or FA2). Unset takes split where the model's backend runs it, else gather. Refused on GLM-5 Next, whose sparse-attention layers attend in their own forward. |
mtp_loss_weight | float | 0.0 | Weight of the MTP head's cross-entropy beside the next-token loss. Above 0 builds the family's released head and trains it; 0 builds nothing, and the other mtp_* fields are refused off their defaults. |
mtp_loss_type | "hard" | "soft" | "hard" | hard fits the gold token two ahead. soft is Soft-MTP: it keeps the loss value and routes the head's gradient to the trunk's next-token distribution. |
mtp_train_scope | "joint" | "detached" | "head_only" | "joint" | What the head's term reaches: head, trunk, embedding and output embedding; the head alone; or the head alone over a frozen trunk. |
mtp_head_source | str | None | None | Hub id or local directory to read the head from when the model's own is stale or absent. Ignored by a resume from a checkpoint, which keeps its trained head; refused under init_from_scratch. |
mtp_min_acceptance_gain | float | None | 0.0 | head_only: the least gain over its first held-out reading a finished fit must show on both acceptance metrics, or the run exits non-zero naming the checkpoint that holds the rejected head. null turns the gate off; refused off head_only. |
allow_cross_document_linear_attention | bool | False | Train packing / padding_free rows on a model whose linear-attention layers receive no document boundary (Ring's Lightning Attention-2), or whose short convolutions receive none (Inkling); their state then carries from each document of a row into the next. Off, such a run is refused; under pipeline parallelism it is refused either way (PP packing). A run making no such row (no packing, padding_free or packed pre-processed corpus) refuses the opt-in. GLM-5 Next's flat rows stay refused: its DSA layers read only the padding mask on the dense backends it runs (Document isolation). Refused under tree_packing, whose plan segments those layers. |
generate_eval_examples | bool | False | Generate text examples during eval. |
assistant_message_template | str | None | None | The rendered assistant-turn prefix of the model's chat template; required when train_on_completions_only is on, refused when it is off. |
num_eval_examples | int | 50 | Examples to generate during eval; refused off its default with generate_eval_examples off. |
model_supports_system_role | bool | True | If False, system prompt is prepended to the first user message. |
interleaved_thinking | bool | False | Pass clear_thinking=False to apply_chat_template. Needed for templates with a clear_thinking switch (e.g. GLM) when the rollout backend kept historical assistant reasoning — the template default otherwise strips <think>…</think> from history. Text-only: the VLM path raises. |
OnlineGRPOScriptArguments
RLVR Online GRPO. Source: halo/configs/grpo_online.py
| Parameter | Type | Default | Description |
|---|---|---|---|
max_prompt_length | int | None | None | Prompt budget applied as a dataset filter: rows above it are dropped, never truncated. null = no filtering. |
prompt_field | str | "prompt" | Field with the prompt (string or conversation list). |
answer_field | str | "answer" | Field with the ground-truth answer. |
system_prompt | str | None | None | System prompt to prepend. |
reasoning_effort | "low" | "medium" | "high" | "random" | None | None | Chat-template reasoning-effort steer: low/medium/high, random (sampled per prompt), or None. Passed via chat_template_kwargs; needs a template that reads reasoning_effort (e.g. gpt-oss harmony). |
rewards | list[dict] | [{source: accuracy}] | Reward terms, each {source, name?, weight?, exponent?, ...}; sources accuracy, format, judge (requirements only; one listing checks is refused), reward_model; the external terms take view (final / full / judge-only digest) and on_error (invalid / neutral). One TRL reward function per term. YAML-only. See Reward Terms. |
use_chunked_grpo_logprobs | bool | False | Vocab-chunked completion log-probs instead of full [B, T, vocab] logits, for large-vocab models on long completions. See Online GRPO. Off, construction refuses a full-logits plane the device's free memory cannot hold; the check is skipped with max_completion_length unset (the logits wall). |
packed_microbatch_tokens | int | None | None | As on the environmental trainer (Packed micro-batches): TRL's old-policy and reference passes and the loss forward run the round's token-budget bins as one flattened varlen call each. Needs use_chunked_grpo_logprobs: true; refused with use_sdpg: true. |
loss_forward_importance_weights | bool | False | Build the vLLM importance-sampling ratio (any vllm_importance_sampling_mode) and the KL tail clamp from the loss forward's detached log-probs in place of TRL's no-grad old-policy pass. true raises at construction on balance_token_mass, num_iterations > 1, misaligned accumulation, or a run with vllm_importance_sampling_correction off. Composes with packed_microbatch_tokens and use_sdpg. See Weight from the loss forward. |
log_phase_times | bool | False | Log each optimizer step's wall time by phase as phase_time/* (Phase times). Costs a few CUDA events per micro-step and one all-reduce per log. |
use_rlrr | bool | False | RLRR relative-reward shaping: advantages come from intra-group rankings instead of the group mean/std. Refused together with scale_rewards_std_floor > 0, which RLRR never divides by, or drop_degenerate_groups, whose raw-reward equality test would mask the all-correct groups RLRR ranks by length. |
use_sdpg | bool | False | Swap in DistributedSDPGTrainer: the GRPO loss plus a privileged-teacher reverse-KL OPD term. Text-only. See SDPG. |
opd_positive_advantage_only | bool | True | Charge the OPD term on positive-advantage tokens only. false with use_sdpg: false is refused. |
The tuning fields behind the two toggles — rlrr_* (RLRRArguments, halo/configs/grpo.py) and the SDPG set (SDPGArguments plus opd_positive_advantage_only) — are refused at a non-default value when their toggle is off rather than ignored; both sets are additionally range-validated whether or not their toggle is on (their bounds: RLRRArguments, SDPGArguments).
It also inherits AdvantageShapingArguments (halo/configs/grpo.py): scale_rewards_std_floor, drop_degenerate_groups and balance_token_mass, with the same semantics as the AsyncRolloutConfig rows, except that drop_degenerate_groups defaults false here (it is true in async GRPO, whose sparse verifiable reward makes dead groups dominate the batch) and judges a group on the total weighted reward, since this arm's terms carry no shaping.
rollout_weight_precision (RolloutWeightPrecisionArguments) is an online field too, as in its AsyncRolloutConfig row.
It inherits GRPOEarlyStopArguments as well: early_stop_entropy_band, early_stop_logratio_gap and early_stop_patience, as in the AsyncRolloutConfig rows, with the gap read from TRL's sampling/sampling_logp_difference/mean (refused without vllm_importance_sampling_correction). The skipped-update condition is async-only: this trainer has no trust-region breaker.
RLRRArguments
The rlrr_* tunables of RLRR relative-reward shaping (arXiv:2601.23058), read under use_rlrr: true. Source: halo/configs/grpo.py
| Parameter | Type | Default | Description |
|---|---|---|---|
rlrr_mode | "hrr" | "prr" | "hrr" | Hybrid rank correction (Eq. 3) or pure relative (Eq. 4). |
rlrr_tau | float | 0.1 | HRR rank-correction magnitude; finite, > 0. Too high dilutes the rule reward. |
rlrr_lambda | float | 2048.0 | Length-bin granularity for re-ranking: correct responses bucket by floor(len / λ); finite, > 0. |
rlrr_xi_pos / rlrr_xi_neg | float | 1.0e-3 / -1.0e-3 | Advantage cap for incorrect responses and floor for correct ones (Eq. 5 clip); xi_neg > xi_pos is refused. |
rlrr_std_normalize | bool | False | Divide the centered advantage by the group std; false keeps the Dr. GRPO normalizer. |
rlrr_length_rerank | bool | True | The length-bin tie-break in hierarchical re-ranking (Eq. 6). |
rlrr_correctness_clip | bool | True | Correctness-aware advantage clipping (Eq. 5); disable for pure PRR with no gold labels. |
rlrr_correctness_threshold | float | 0.5 | A response is correct iff its raw reward is at least this, the only correctness signal. |
SDPGArguments
The privileged-teacher OPD term, shared by online SDPG (on OnlineGRPOScriptArguments, read only under use_sdpg: true) and self-distillation. Source: halo/configs/distillation.py
| Parameter | Type | Default | Description |
|---|---|---|---|
sdpg_hint_template | str | "\n[Hint] The correct answer is: {answer}. Do NOT state that you were given the answer.\n" | str.format template the teacher forward sees after its prompt. {answer} fills on both arms, {solution} on self-distillation only; any other placeholder is refused at parse time. |
sdpg_loss | "reverse_kl" | "forward_kl" | "unnormalized_kl" | "jensen_shannon" | "reverse_kl" | OPD loss. |
sdpg_jsd_beta | float | 0.5 | β of jensen_shannon, in [0, 1] (Generalized JSD). Refused off its default under any other sdpg_loss. |
sdpg_temperature | float | 1.0 | OPD softmax temperature; finite, > 0. |
sdpg_beta_base | float | 1.0 | Base OPD coefficient; finite, >= 0. 0 drops the term. |
sdpg_beta_warmup_steps / sdpg_beta_decay_steps | int | 0 / 0 | Steps over which beta ramps from 0 to sdpg_beta_base, and the final steps over which it decays back to 0; >= 0. |
SelfDistillScriptArguments
The conversation-rendering and vision fields of SFTScriptArguments (no train_on_last_assistant_only, tree_*, allow_cross_document_linear_attention or generation-eval key) plus SDPGArguments and the fields below. Source: halo/configs/distillation.py
| Parameter | Type | Default | Description |
|---|---|---|---|
sdpg_answer_field / privileged_solution_field | str | None | "answer" / "solution" | Columns filling the hint's {answer} and {solution}; a slot the template names needs its column while sdpg_beta_base > 0. At sdpg_beta_base: 0 the script refuses them off their defaults, as it does confidence_weight_opd, opd_exclude_eos and the sdpg_* knobs (sdpg_temperature stays legal beside reference_kl_coef > 0, whose anchor reads it). |
opd_exclude_eos | bool | True | Drop EOS/stop tokens from OPD but not from SFT. |
reference_kl_coef | float | 0.0 | Alpha on a frozen-reference KL anchor; finite, >= 0. 0 loads no reference. |
reference_kl_loss | "unnormalized_kl" | "reverse_kl" | "forward_kl" | "unnormalized_kl" | The anchor's divergence, evaluated at sdpg_temperature. |
reference_model_name_or_path | str | None | None | The anchor model; None = the student's init weights. |
confidence_field / confidence_power | str | None / float | None / 4.0 | Per-sample weight conf**p from this column, each confidence in [0, 1]; p finite, > 0. |
confidence_weight_opd | bool | True | Apply that weight to OPD as well as SFT. |
EnvironmentalGRPOScriptArguments
Source: halo/configs/grpo_environmental.py
| Parameter | Type | Default | Description |
|---|---|---|---|
max_prompt_length | int | None | None | Prompt budget applied as a dataset filter: rows above it are dropped, never truncated. null = no filtering. |
prompt_field | str | "prompt" | Field with the prompt or question. |
answer_field | str | None | "answer" | Field with the expected answer, carried into the row as answer when the column exists. Whether an answer is needed is the environment's requires_answer; a value other than the default must name a real column or startup raises. |
context_fields | list[str] | None | None | Additional fields passed as context to the environment. |
There is no system_prompt: the environment builds the rollout conversation from its own system prompt plus the dataset's user turn, so a config setting the key raises; set it with environment_kwargs: {system_prompt: ...} (Async GRPO). The ReAct presets (react_math, react_search) hardcode theirs and refuse the kwarg.
Other script arguments
These extend CommonScriptArguments with minimal additions:
| Class | Source | Extra Fields |
|---|---|---|
SMPOScriptArguments | halo/configs/preference.py | generate_eval_examples (True), num_eval_examples (50), and the vision fields freeze_vision_tower, min_pixels, max_pixels, as on DPO, KTO and reward below |
DPOScriptArguments | halo/configs/preference.py | generate_eval_examples (True), num_eval_examples (50), generation_max_prompt_length (512) — bounds the eval-time generation dataset only, refused off its default with generate_eval_examples off; DPO has no training-side prompt cap — and images_field (None) — VLM only: image column renamed to the images spelling TRL's vision probe reads, ahead of the dispatch |
OfflineGRPOScriptArguments | halo/configs/grpo_offline.py | generate_eval_examples (True), num_eval_examples (50) |
KTOScriptArguments | halo/configs/preference.py | completion_field ("completion"), label_field ("label") — KTO uses an unpaired {prompt, completion, label} dataset — and images_field (None) — VLM only: renamed to TRL's images spelling; the vision path refuses precompute_ref_log_probs and paired columns |
RewardScriptArguments | halo/configs/reward.py | images_field (None) — VLM only: image column, renamed to images and merged into the shared prompt conversation; declares the run VLM on a score-headed multimodal checkpoint (reward VLMs) |
ClassificationScriptArguments | halo/configs/classification.py | text_field (None) — raw-text column wrapped as one user turn when the dataset has no prompt conversation (e.g. text/label sets like imdb) |
EmbeddingScriptArguments | halo/configs/embedding.py | None. The script refuses inherited knobs it has no stage for: pad_token, bos_token, eos_token, chat_template, force_chat_template, added_special_tokens, unfreeze_layers_patterns, freeze_layers_patterns, tools_field, log_decoded_samples, text_only_model and a non-default tokenizer_backend |
YAML shape
A config is the method's fields plus any TrainingArguments field. bf16: true,
use_liger_kernel: true, logging_nan_inf_filter: false and, with pinned batches,
accelerator_config.non_blocking: true are toolkit defaults and need not be written.
YAML
model_name_or_path: "meta-llama/Llama-3.1-8B-Instruct"
dataset: "s3://bucket/sft-dataset"
conversation_field: "conversation"
train_on_completions_only: true
assistant_message_template: "<|start_header_id|>assistant<|end_header_id|>\n\n"
max_length: 16000
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 0.00004
gradient_checkpointing: true
output_dir: "checkpoints/sft-run"Working configs for every method and model family live under examples/<method>/<family>/; each
method page links its own.
Parsing
HaloArgumentParser (halo/configs/parser.py) reads the YAML (argv[0] when it ends in .yaml/.yml) or a flags-only command line, casts the --key=value overrides, applies the toolkit defaults above, then builds each config dataclass once from the merged values. Every __post_init__ (transformers', TRL's, the toolkit's) therefore derives its state from, and validates, the final configuration: --gradient_accumulation_steps=1 over a YAML's 8 also re-derives GRPO's steps_per_generation and generation_batch_size.
- A toolkit default applies only where neither the YAML nor the CLI wrote the field, and reaches every config declaring it before that config is built, so
TrainingArgumentsderivesmixed_precisionfrombf16. Thebf16default yields to an explicitly-enabledfp16. - Every key no config declares raises and names the field. Nothing is migrated or stripped, ecosystem spellings included.
- A
Literal[...]field rejects an out-of-set value (advantage_method), in YAML and on the CLI alike. Mixed unions (float | Literal["auto"]) are not validated. - A string on a bool field is rejected: YAML 1.2 reads the 1.1 spellings
yes/no/on/offas truthy strings, which would silently invertpacking: no.
Every training script parses through parse_script_args (halo/entrypoints/script.py), which adds four effects around the pure parser:
init_distributed()runs before any dataclass is built. ConstructingTrainingArgumentstouchesself.device, which would let accelerate create the default process group withoutdevice_id; that binding lost costs anew_groupper mesh dimension instead of onencclCommSplit.output_direxpands strftime directives (checkpoints/sft-%Y-%m-%dT%H-%M-%S) from one timestamp, broadcast from rank 0 so every node agrees.%%collapses to%; any other%(sft-100%-data) survives byte-identical. No other field expands them.- The
run.logtee into thatoutput_dirstarts (Launch Recipes). - A
full_determinismconfig turns transformers'FLASH_ATTENTION_DETERMINISTICon before anything picks an attention backend: SFT plans its rows before the run setup, and HF's Trainer sets the variable only at construction, after the load.plan_row_attentionrefuses afull_determinismconfig the switch has not reached, such as one built withoutparse_script_args.
CLI overrides
--param=value overrides any field, dashed spellings normalized to the underscore name. The = is required: --key value raises "CLI overrides must be in --key=value form". A repeated override (--learning_rate=1e-5 --learning_rate=2e-5) raises instead of keeping the last. An override matching no field of the script's config dataclasses raises, as an unknown YAML key does.
--field=None,--field=nulland--field=noneclear an Optional field, container unions included. Three carve-outs: aLiteralwhose choices include"none"gets the string (--moe_balancing=none); a field whose default spells "no value" as a string of its own gets that string (--report_to=nonereaches transformers as"none", the way to silence logging on a smoke run, whereNonewould become[None], which the Trainer refuses); an optional bool refuses the spelling (--bf16=noneraises).- A field with no confident string cast stays in the YAML: dict-typed fields, lists of containers (
rollout_server_configs: list[dict]), and unions with a container member (dataset: str | list[str],--report_to=wandb). - Without a YAML the whole config is argparse flags:
--key valueworks, an unknown flag is a usage error, each flag reaches every config dataclass that declares it (pad_tokensits on the script args and on TRL's config), and an unset field keeps each dataclass's own default.
Source on GitHubUpdated 18 November 2026