Scripts Reference
Catalog of toolkit scripts by category. For config fields see Configuration; for parallelism flags see ParallelismConfig.
Every entry script answers python <script> --help with its full flag list, including the CLI overrides for any YAML field.
The halo CLI (halo/cli.py) runs them by name: halo launch <method> <config> for scripts/training/, halo run <tool> for every other scripts/ subtree except benchmarks/, whose harnesses (Halo's and other frameworks') are run by their own launch scripts, each in its framework's image (--list on either enumerates them). halo launch follows its flags, never the config: accelerate launch with --accelerate <cfg>, torchrun when --nproc/-n > 1, plain python otherwise, so halo launch sft <ep-config>.yaml without -n runs single-process. --numa-binding <mode> (torchrun's modes) goes on the torchrun line ahead of the script and is refused for an accelerate launch or python launch, which have no NUMA binding. halo run always runs one python process (every tool is single-process) and owns only --list, --dry_run and --root, so -n, -p and a tool's own --port reach the tool. Both exec the script with --root first on PYTHONPATH, so a second checkout runs on its own halo and scripts.
Run scripts from the repo root with the repo root importable. The wheel installs halo only (packages = ["halo"]), and python scripts/<subtree>/<tool>.py puts the script's directory on sys.path, not the repo root. So every CLI that imports a scripts.* helper needs scripts importable as well as halo.
Both training images set PYTHONPATH=/workspace and the Makefile passes it into every container it starts. A host run needs PYTHONPATH=. from the repo root.
A tool script parses its flags and calls its driver in halo/, which holds the logic; the tests exercise the driver and reach a script only through its parser or main(). The checkpoint tools' drivers are in halo/checkpoint/tools/: ep_shard_merge.py, pp_stage_merge.py, model_merge.py, adapter_merge.py, bf16_conversion.py, lowp_export.py, sink_reset.py, expert_unfuse.py, vision_reattach.py, vocab_patch.py, block_fp8_export.py, fp8_dequant.py (the GLM-5 and Mistral 3/4 converters) and deepseek_v4_dequant.py. The dataset tools' drivers are in halo/data/, the generation, scoring and environment tools' in halo/entrypoints/, and the code-contests test compaction is halo/environments/tasks/coding/suite_compaction.py. Four kinds of script keep their logic. A training script assembles its run around the trainer, its driver. The two Gradio playgrounds are the app. nvlink_health.py and the scripts/docs/ checks import no halo module, so they run on the bare host. trace_report.py hands its traces to TraceLens's report generators. A script configures logging when it runs, never at import, so importing one changes no logging state: a tool's main() calls configure_cli_logging, the three async CLIs through run_async_cli, and a training script calls configure_training_logging as it parses its arguments; regrade_trajectories.py and the two Gradio apps leave the WARNING root.
The scripts.* flag helpers are the shared flag surfaces:
The generation CLIs' runs live in halo/entrypoints/: resumable_generation.py (the S3 prompt dataset and its resume, the merged push, the empty-run refusal, the batched generation run and run_async_cli), reward_scoring.py (the reward-model load, built on a CUDA device through apply_family_patches as a training load is and on CPU with the backend as given, and refused when the backend drops the live sinks a GptOss checkpoint holds, plus the batched scorer and the score.py run) and rejection_sampling.py (the rejection_sample.py run).
Multi-word flags use underscores (--base_url), matching the YAML field names a training flag overrides; a training script also takes the dashed spelling of an override (CLI overrides). A tool's on/off flag is negated by argparse's --no- prefix (--no-trust_remote_code), and a tool flag that is itself a negative keeps underscores (--no_overwrite). A training script's flags-only command line also takes transformers' --no_<field> for a field that defaults to true (--no_gradient_checkpointing); with a YAML, override it as --<field>=false. Beyond argparse's -h, short options exist only on s3.py (-b, -s, -v, -q, -r, -n, -y), gradio_openai_chatbot.py (-m) and halo launch (-n, -a, -p). One concept has one spelling across the tools: --output_dir for an output directory, --output_path for an output file, --dataset / --split / --config for a dataset source, --model_id for a model that may be a Hub repo (the reward-model scorers prefix it, --rm_model_id, beside the served --model; prepare_dataset.py's flags are its config's fields, so it spells --model_name_or_path), --max_tokens for a generation cap and --max_workers for request or episode concurrency.
The checkpoint tools under after_training/ and before_training/ take one source/destination pair: --input_dir → --output_dir for a local checkpoint directory, --model_id → --output_dir where the source may also be a Hub repo (patch_vocab.py, convert_deepseek_v4_bf16.py, convert_mistral4_bf16.py, convert_glm5_bf16.py, reset_sinks.py). reattach_vision_tower.py takes both: --input_dir for the text-only export, --model_id for the multimodal base. A Hub --model_id is fetched without its vendor weight dumps (gpt-oss's original/ and metal/), which no tool reads or carries into its output.
Three tools keep a differently-shaped source because it is a different thing: merge_models.py --models (N inputs), merge_peft_adapters.py --adapter_dir (an adapter, not a checkpoint), and prepare_dataset.py --input/--output (dataset URIs — s3://, hf:// or a local path).
merge_peft_adapters.py and convert_to_bf16.py select the head with one --task (add_task_arg): causal_lm, classification, and base on the conversion alone, since an adapter has no bare-backbone task.
Launcher choice: Two launchers.
Training scripts
Every training entry script runs the shared skeleton in halo/entrypoints/: parse_script_args (distributed init, the YAML/CLI parse, the output_dir expansion, the run.log tee and the full_determinism attention switch) → init_training_script (parallelism config, run setup, resume resolution, returning the run's ScriptContext) → DP-shard-aware dataset loading and the modality verdict (data_setup.py) → the policy load and window pins (model_setup.py) → configure_balancing_and_build_callbacks → run_trainer, which emits the canonical start log (mode, EP/CP/TP/ETP/PP/DP sizes and the attention backend the policy was built with) and reorders integration callbacks last so moe/* and efficiency keys reach W&B. with_distributed_teardown(main)() tears the process group down after a finished or early-stopped run.
Which parallelism axes each trainer takes: Trainer compatibility. Launch commands are on each method's page.
Inference & generation scripts
In openai_batched_generation.py, --dataset / --output_path name S3 datasets. A full
s3://<bucket>/<key> URI is used as given; a bare key is placed by build_s3_uri under --subfolder
(default datasets, None to skip) in HALO_S3_DEFAULT_BUCKET, and raises naming its flag while that
variable is unset.
score.py and rejection_sample.py read --dataset (a local JSONL) and append to --output_dir. --follow_up_prompt_field
names follow-up turns sent after the first answer, and --correct_answer_field the ground truth shown to the reward
model. That model loads from --rm_model_id with --rm_attn_implementation (default sdpa) and drops, counted, a
conversation over --rm_max_seq_len tokens (16000). rejection_sample.py adds --n_hypos hypotheses per prompt (5,
at least 2) scored --rm_max_batch_size per forward (8). gradio_openai_chatbot.py --stop_token_ids takes a
comma-separated list passed to vLLM.
The three async CLIs — openai_batched_generation.py, rejection_sample.py, score.py
— run under a shared SIGINT/SIGTERM handler
(run_async_cli) that exits 128 + signo, the shell's own convention for a signalled process.
Progress is stored for resume (generation through parallel_openai_requests and its JSONL resume store under $HALO_DATA_ROOT/inference/resume, halo/inference/batch_requests.py; the two reward-model CLIs in their own JSONL output), so re-running resumes; the non-zero exit is what stops a wrapper script or
&& chain from consuming a partial output dataset as a finished one.
A run that produced no usable row raises rather than writing an empty result (reject_empty_results), so a dead endpoint or a
wrong --model cannot republish the resumed rows as a finished job. Each CLI's summary line
names its per-reason drop counts (first-response failures, degenerate skips).
Every endpoint these CLIs talk to is OpenAI-compatible (--base_url / --api_key),
reached through the one shared client (build_openai_client, halo/inference/openai_client.py) with the toolkit's retry policy;
--base_url defaults to http://localhost:8000/v1 and --api_key to VLLM_API_KEY, then OPENAI_API_KEY, then the
EMPTY placeholder (halo/inference/endpoints.py). There
is no separate Azure mode — point --base_url at the deployment's OpenAI-compatible route like
any other endpoint.
Bash
python scripts/inference/generation/openai_batched_generation.py \
--model my-model --dataset s3://your-bucket/datasets/prompts --output_path s3://your-bucket/datasets/responses
python scripts/inference/reward_model/score.py \
--model my-model --dataset data/prompts.jsonl --rm_model_id path/to/reward-modelPost-training scripts
Bash
python scripts/after_training/merge_peft_adapters.py \
--adapter_dir /models/sft-llama-lora --output_dir /models/sft-llama-merged
python scripts/after_training/merge_ep_shards.py \
--input_dir /models/ep-checkpoint --output_dir /models/merged-modelInput guards
Every tool here refuses an input it cannot express, rather than writing a plausible-looking result:
Per-rank EP-sharded input (
metadata.format=ep_sharded) is rejected by all of them exceptmerge_ep_shards.py, which exists to consume it. Merge first, or re-save gathered.Such a save reuses the ordinary index filename while each expert tensor is one rank's partial slice under a
.shard_Nkey. Afrom_pretrained-based tool would see the real expert keys as missing, which transformers resolves by randomly initializing them (warning only), and then save that over the source.The refusal does not need the index: a directory whose shards carry
.shard_Nkeys with no index (a save killed before the index write) is caught by a header-only peek. Thescripts/before_training/bf16 converters refuse it on the same terms, since all of them also accept a local directory.In-place conversion (
--output_direqual to any input directory) is rejected by every tool that streams from the source while writing: the write deletes the weight files it does not overwrite and so destroys the source mid-read. A refused conversion creates no output directory.That is
merge_ep_shards.py,merge_pp_stages.py(on every node directory),unfuse_moe_experts.py,quantize_to_lowp.py,merge_models.py,merge_peft_adapters.py,convert_to_bf16.py,reattach_vision_tower.py(on both of its sources),convert_deepseek_v4_bf16.py,convert_mistral4_bf16.py,convert_glm5_bf16.py, andpatch_vocab.py, whose save would overwrite the source's shards in place with no undo, and whose post-save sweep then deletes every source shard the save did not rewrite.prepare_dataset.pyis not one of them: it bakes into a staging sibling of its--output(a temp directory for S3 or the Hub) and publishes with a swap once the bake is done, so its--outputis guarded by--overwrite, not by this refusal.reset_sinks.pyis the exception: it stages and verifies every shard it rewrites before any replaces its target, so in-place is safe there, but never implicit. It takes--in_place; an--output_diraimed at the input is refused like the rest.Omitting both is an error on a write (a
--dry_runonly reads) rather than a silent rewrite of the only copy. A fresh-dir run sweeps leftovers like the others.Remote code. Every checkpoint tool that loads a checkpoint's own modeling/config/tokenizer code takes one
--trust_remote_code, spelled once (add_trust_remote_code_arginscripts/cli_arguments.py):merge_peft_adapters.py,merge_models.py,convert_to_bf16.py,reattach_vision_tower.py,patch_vocab.py,prepare_dataset.py,convert_deepseek_v4_bf16.py, the reward-model scoring CLIs (score.py,rejection_sample.py) throughscripts/inference/reward_model/reward_model_arguments.py, anddeduplicate_dataset.py.The default follows the input source. A local checkpoint or merge input (
--input_dir,merge_models.py's--models,--base_modeland--tokenizer_source, all local directories, a Hub id refused) defaults on: the remote-code families in the roster (Bailing/Ling, Laguna) do not load without it, and the operator already produced that artifact.A source that may be a Hub repo (the
--model_idofpatch_vocab.py,convert_deepseek_v4_bf16.py,reattach_vision_tower.pyanddeduplicate_dataset.py, the scorers'--rm_model_id, andprepare_dataset.py's--model_name_or_path) defaults off: a freshly downloaded third-party repo must not execute its own code merely because a tool was pointed at it.An adapter (
merge_peft_adapters.py's--adapter_dir,convert_to_bf16.py --peft) follows the base itsbase_model_name_or_pathnames: on for a local directory, off for a Hub id, so an adapter on a Hub base of a remote-code family needs--trust_remote_code. Everywhere the opposite is one flag away (--trust_remote_code/--no-trust_remote_code).The tools that never enable remote code expose no such flag —
merge_ep_shards.py,merge_pp_stages.py(it copies files),unfuse_moe_experts.py,quantize_to_lowp.py,reset_sinks.py,convert_mistral4_bf16.pyandconvert_glm5_bf16.pystream safetensors and rewriteconfig.jsonas JSON, so no checkpoint code is ever imported.Uninitialized buffers. A tool whose loaded model is handed to a forward (reward scoring, the dedup embeddings,
convert_to_bf16.py --check_inference) callsfinalize_loaded_modelafter placement, since transformers 5 re-materializes non-persistent buffers as uninitialized memory (buffers).The load-only conversion tools do not:
state_dictomits those buffers, so an unrepaired one reaches neither a number nor the written checkpoint.tests/cpu/model_loading/test_load_finalization.pypins each new tool loader, in a script or itshalo/driver, into one of the two groups.Shard cap and Hub source. Two more flags are spelled once each, beside
add_trust_remote_code_arg.add_max_shard_size_argwrites--max_shard_size(defaultDEFAULT_MAX_SHARD_SIZE,5GB) for every tool that re-shards its safetensors output;quantize_to_lowp.pyandreset_sinks.pymirror the input's shard layout one-for-one and take no cap.add_hub_source_argswrites--model_idplus the--revisionpin for every tool whose source may be a Hub repo.patch_vocab.pytakes--model_idwithout--revision: it threads none, and an advertised pin a tool ignores would silently convert whatever the Hub's default branch holds that day. Pin such a source by downloading it first and pointing--model_idat the directory.Wrong model family.
unfuse_moe_experts.pyresolves the family from the checkpoint'smodel_typeand emits the projection names it declares (LFM-2 readsw1/w3/w2, not GLM-4'sgate_proj/up_proj/down_proj). A family whose checkpoint is not per-expert at all is refused rather than written under a guessed triple.Both checks run after the sharded-input refusal and after the already-per-expert copy-through, so those keep their own diagnosis.
merge_models.pyinputs. Asymmetric key sets (past a Hub base's keys the shipped model never reads), a key stored at different shapes, a--tokenizer_sourcethat is not a local directory and a tokenizer-less one are refused (Model Merging).
Preparation scripts
Bash
python scripts/before_training/prepare_dataset.py \
--input "s3://bucket/raw/dataset" --output "s3://bucket/preprocessed/dataset" \
--model_name_or_path "Qwen/Qwen3-8B" --max_length 8192 --test_size 0.01 --num_shards 64 \
--pack_sequences --assistant_message_template $'<|im_start|>assistant\n'
python scripts/before_training/patch_vocab.py \
--model_id Zyphra/ZAYA1-8B --output_dir <large-volume>/models/ZAYA1-8B-patched \
--patterns '["<tool_call>", "<tool_result>"]'
# GptOss sink reset: --reset_sinks
# remote-code family (Bailing/Ling, Laguna): add --trust_remote_code — a Hub source is opt-in herepatch_vocab.py only grows the embedding (added tokens reuse existing padding rows), never shrinks it: models like GPT-OSS ship a vocab padded past len(tokenizer), and shrinking would drop the high special tokens (harmony <|return|>/EOS, …) and break generation. A patched checkpoint therefore serves like its base, except a GPT-OSS patched with --reset_sinks: served sinks-off, the pretrained model degenerates (GPT-OSS). The added tokens start at the mean embedding and mean nothing until trained.
deduplicate_dataset.py embeds --text_field with --model_id (default sentence-transformers/all-MiniLM-L6-v2,
capped at --model_max_length tokens, --batch_size per forward on --device), or reads --load_embeddings <file.npy>
instead; --save_embeddings [PATH] keeps what it computed (default <output_path>.embeddings.npy) and is refused
beside --load_embeddings. A row whose cosine similarity to an earlier one exceeds --similarity_threshold (0.95)
is dropped, over --dedup_steps reshuffled passes (3) of --dedup_batch_size rows per search batch (100000) on
--max_workers threads. --sample_size N runs on a seeded sample and --keep_fields whitelists the output columns.
Profiling & benchmarks
Throughput benchmarks live under tests/gpu/profiling/, grouped by what they measure:
run_baseline_benchmarks.sh in the same directory runs the rows make bench holds to tests/baselines/bench/
(Golden performance baselines).
Bash
make bench BAND=1
torchrun --nproc_per_node=8 tests/gpu/profiling/benchmark_smpo_ep.py --ep 8 --cp 2Docs scripts
Related pages
- Checkpoints & Resume
- SFT Dataset Pre-Processing — guide to
prepare_dataset.py - Configuration Reference — YAML options for training scripts
Source on GitHubUpdated 18 November 2026