Contributing

Every command goes through make: contributors, CI and the agent skills call the same targets, and the Docker invocation lives once in the Makefile (make help lists them).

Ground rules

  • The host has no usable Python. PyTorch, DeepEP and Flash Attention live only in the images, so anything that runs, runs inside one, where python, torchrun, pytest and ruff are on PATH (uv installs into the system interpreter; no venv).
  • make lint, make format, make precommit and make docs are the host-runnable gates. ruff runs through uvx ruff@$(RUFF_VERSION) at uv.lock's pin (else the ruff on PATH); the docs target needs stdlib Python 3.12. Tests and benchmarks run in the image.
  • Credentials live in the repo-root .env (cp .env.example .env): the GPU make targets pass it with --env-file and fail without the file.
  • Markdown lives in agent-docs/, human-docs/ and skills/, plain GitHub markdown; the root files (README, CONTRIBUTING, SECURITY, CODE_OF_CONDUCT, AGENTS.md, CLAUDE.md, .github/*) are the exception, and scratch markdown goes to /tmp.
  • Large outputs go to a verified large volume. The make targets put the caches and TMPDIR under HALO_SCRATCH, which resolves to the local mount other than / with the most free space when unset; confirm it with findmnt / df -h or export it (Docker → Standard training launch).

The gate (read first)

Every change starts as an accepted issue and a maintainer's /approve @you on it, before any PR; an unapproved PR is closed automatically. The rules are in CONTRIBUTING.md, the workflow mechanics in Continuous Integration.

Dev setup

Bash

# Outside contributors: fork on GitHub first — there is no push access to whitecircle/halo.
git clone https://github.com/<your-handle>/halo.git   # maintainers: whitecircle/halo
cd halo

cp .env.example .env          # the GPU make targets pass it with --env-file and fail without it
# Either pull the prebuilt image under the name the make targets use ...
docker pull public.ecr.aws/whitecircle/halo:blackwell && docker tag public.ecr.aws/whitecircle/halo:blackwell halo:blackwell
# ... or build it: B200 (SM100) / B300 (SM103); build-hopper for H100/H200 (SM90)
make build-blackwell
make check-lock               # check uv.lock installs into the image
make seed-hf-cache            # configs + tokenizers the CPU tests read into HF_CACHE (no weights)
make test-cpu                 # sanity check; needs Docker, not a GPU

The CPU tier needs the image's Triton build: on the PyPI triton==3.6.0 a bare uv.lock install gets, the interpreted tests of kernels with a runtime loop bound fail, and tests/cpu/kernels/test_triton_interpreter.py names the cause in one failure.

Point any run target at another image with IMAGE= (make test-cpu IMAGE=halo:hopper); the build-* and push-* targets name their own tags. What each image carries and how it builds: Docker.

Dev loop

StepCommandWhere it runs
Formatmake formathost (ruff format)
Lintmake linthost (ruff check)
Lint + format-checkmake precommithost
Same gates on every commit (optional)pipx install pre-commit && pre-commit installhost (.pre-commit-config.yaml)
CPU testsmake test-cpuimage (no GPUs)
Core GPU tests (pre-merge, GPU changes)make test-gpu-coreimage (-m "gpu and core")
Full GPU tests (heavy tier)make test-gpu-fullimage (-m gpu)
vLLM-server GPU testsmake test-gpu-vllmimage (-m "gpu and vllm_server and ($SERVER_TIER)")
SGLang-server GPU testsmake test-gpu-sglangimage (-m "gpu and sglang_server and ($SERVER_TIER)")
Throughput and peak memory hold to the committed baselinesmake bench BAND=1image (8 GPUs)
A structural change only moves codemake verify-move BASE=<sha>image (no GPUs)
Every CPU saver writes what it wrote at BASEmake verify-artifacts BASE=<sha>image (no GPUs)
EP forward, backward and gradients match BASE bit for bitmake verify-ep BASE=<sha> [TIER=gpu]image (TIER=gpu: 4 GPUs)
Ulysses CP attention and model match BASEmake verify-cp BASE=<sha> [TIER=gpu]image (TIER=gpu: 2 GPUs)
The efficiency callback reports what it reported at BASEmake verify-efficiency BASE=<sha>image (no GPUs)
Docs links, anchors and code referencesmake docshost (scripts/docs/check_links.sh)

The server tiers need their compose server already serving on a GPU outside TRAINER_CUDA_DEVICES (default 0,1,2,3,4,5,6, which the target pins as the trainer's CUDA_VISIBLE_DEVICES): weight sync is an NCCL broadcast, and a rank cannot broadcast to itself. Without EFA=1 the targets force the no-fabric recipe both compose bases default to; EFA=1 passes /dev/infiniband and the fabric variables instead, matching a server started with its EFA overlay (Rollout Servers → Servers on other nodes). The SGLang server needs only cuMem parity on top, the compose default (NCCL transport).

Each server tier runs one pass per served model: the tests broadcast the trainer's own weights into the served model and assert the served policy moved, so server and trainer hold the same checkpoint. SERVER_TIER defaults to not moe (serve Qwen/Qwen3-0.6B); the MoE passes restart the vLLM server on Qwen/Qwen3-30B-A3B-Instruct-2507 for SERVER_TIER='moe and not gptoss' and on unsloth/gpt-oss-20b-BF16 for SERVER_TIER='moe and gptoss', while SGLang's MoE tests are gpt-oss only (SERVER_TIER=moe). The vLLM server needs VLLM_REASONING_PARSER=qwen3: the benchmarks send a per-effort CoT budget, which draws a 400 without a reasoning parser (the target's health guard prints the compose line).

The env-GRPO rows train on a native tool call, so the vLLM server also needs the served family's tool parser (Tool parsers are per family). The compose default, hermes, reads JSON <tool_call> calls only. The gpt-oss pass serves the image's parser plugins instead (the Makefile's VLLM_GPTOSS_PARSERS, which the health guard prints into its gpt-oss compose line), and they replace the qwen3 reasoning parser. A per-family pass (HALO_TEST_ENV_GRPO_MODEL) takes its family's row of that table. Under a wrong parser every call stays text, so each env-GRPO row on the draw task (all but the one-process react_math rows, which send no tool schema) raises by its second round. SGLang's default parser (auto) reads it off the chat template where it can and disables it elsewhere: the Qwen3-30B pass needs SGLANG_TOOL_PARSER=qwen25 (Tool parsers on SGLang).

Bash

make train CONFIG=examples/sft/qwen3/qwen3-4b-ultrachat.yaml NPROC=8
make train CONFIG=examples/sft/gptoss/gptoss-20b-multinode-ep.yaml \
  NPROC=8 EXTRA="--expert_parallel_size=8 --use_grouped_gemm=true"
make train METHOD=preference/smpo CONFIG=<config>    # METHOD: any `halo launch --list` entry; default sft

make clean prunes wandb/, the ruff and pytest caches and the $HALO_SCRATCH test scratch, and leaves checkpoints/, where every example writes its run.

Code style

  • ruff, line length 119. Selected: E, F, I, W, UP, B, C4, SIM, T20, PLC0415, D200, D202, D210, D300, D419, ERA001; make lint and the blocking CI job enforce the full set, CI with ruff format --check (Continuous Integration).
  • No inline imports (PLC0415), in tests/ as in halo/. A function-level import is reserved for a genuinely optional or arch-specific dependency (flash_attn, deep_ep, deep_gemm); a circular import is never that exception.
  • One docstring format (D200, D202, D210, D300, D419): triple double quotes, a one-line docstring on one line, no padding inside the quotes, no blank line after a function's docstring, never empty.
  • No commented-out code (ERA001); a note that merely parses as Python (# Default: n * 4) is reworded as prose. No stray print() in halo/ (T20); tests/** relaxes E402, T20, E741, B007, B023 and SIM117, scripts/** relaxes T20.
  • Reuse over re-implementation. New trainers extend DistributedTrainerMixin; new parallelism layers reuse the EP-layer base hooks (Trainer Architecture).
  • Naming, named constants, module order, comments and docstrings follow the Code style section of CLAUDE.md.
  • A field help is short. A config or script-argument help gives the effect and what the default means in one or two sentences, optionally closed by a See agent-docs/<page>#<heading>. pointer the reference check resolves; rationale and measurements go to the Configuration Reference or the method's page.

AI-assisted contributions

AI-assisted contributions are welcome on the terms in CONTRIBUTING.md; CLAUDE.md and AGENTS.md bind agents and humans alike.

Tests

CPU tests run under pytest. GPU tests are scripts launched through a manifest (tests/gpu/manifest.py), each row under torchrun, or as one process that starts its own ranks for a launcher="python" row (a halo launch driver, such as trainers/other/test_entry_scripts.py). Most of the conventions below are enforced by tests/cpu/conventions/.

  • Test behavior, not implementation. Assert loss finiteness, loss decrease, output shapes and the rejection of unsupported configs, never private internals.
  • A CPU test file is a plain pytest module ending in if __name__ == "__main__": raise SystemExit(pytest.main([__file__, "-v"])), so a standalone run executes exactly what pytest collects; tests/cpu/conventions/test_test_conventions.py refuses any other entry, a hand-listed runner or a printed pass/fail summary.
  • The cpu marker is applied by path. tests/conftest.py marks every item under tests/cpu/, so no file re-declares it in pytestmark.
  • Imports resolve from the image, not from sys.path surgery. PYTHONPATH=/workspace is baked in and tests/conftest.py puts the repo root on the path, so from halo... / from tests.common... work under pytest and a standalone torchrun. A tool's driver lives in halo/ and a test imports it from there (from halo.checkpoint.tools.model_merge import merge_models); scripts/ is a namespace package, so a test that drives a CLI's own parser imports it by dotted path (from scripts.environments.inference.eval_code_contests import parse_args), and tests.common.repo.load_script_module loads a file by path where a test needs its own module object or the file sits outside the packages (docker/).
  • Every first-party reference resolves without running. tests/cpu/conventions/test_references_resolve.py statically resolves each first-party import, patch / monkeypatch target, attribute chain on an imported module or class, dotted name and repo-root path in the Python and text files under halo/, scripts/, tests/, docker/ and .github/, the image builds, Makefile, pyproject.toml, .env.example, .pre-commit-config.yaml and the recipes, so a moved module or a dead patch target fails the CPU tier even where only a GPU script names it. An attribute a third-party base supplies (Function.apply) is listed per base in _EXTERNAL_BASE_ATTRIBUTES (scripts/docs/references.py); text that only reads as a reference is exempted per file in the test's _NOT_REFERENCES. The sweep also holds each reference to the binding it means: an import names the module that defines what it takes (a re-binding module is a second path that keeps a moved symbol reachable), and a patch of a module's own top-level name must target a name some function reads when it runs, or it intercepts nothing.
  • A message names symbols, not source paths. tests/cpu/conventions/test_messages_name_symbols.py fails a string in halo/ or scripts/ that names a package file by path: an error, warning or field help names the function, class or constant its reader acts on. Docstrings are exempt, and a scripts/ tool a user runs keeps its path.
  • A move rewrites every reference in the same change: the imports (a from line split per destination), the patch / monkeypatch / caplog / sys.modules / import_module strings and the repo-root paths, leaving no re-export. Splice the source text at the AST nodes' offsets, or use libcst in a throwaway environment (uv run --no-project --with libcst <script>, which leaves uv.lock alone); never regenerate a file with ast.unparse, which drops its comments. make verify-move BASE=<sha> proves the move changed no code (Move kit).
  • The import layering is read off the import graph. tests/cpu/conventions/test_import_layering.py holds the layer table (_LAYERS: a unit imports only units of lower rows, cli imports nothing from the package, the tool-safe units reach no process group, parallelism or rollout sync, one lazy deep_ep importer, no module cycle), refuses a raw toolkit env read, and asserts _LAYERS equals the table Architecture renders. TYPE_CHECKING imports are exempt.
  • Convention tests also run on the hosted runner. The conventions job in .github/workflows/lint.yml runs the files it lists on the standard library and pytest alone, so such a test reads the source with ast and takes repo helpers from tests.common.repo; a listed test that runs a probe script names the files the probe collects in _PROBE_COLLECTS (tests/cpu/conventions/test_hosted_conventions_are_stdlib.py).
  • Remote code loads into the test process's own module cache. tests/cpu/conftest.py points HF_MODULES_CACHE at a scratch dir per process before transformers is imported. Do not redirect the cache inside a test around a trust_remote_code load: transformers puts each cache it copies into on sys.path and never removes it, so the transformers_modules package can stay bound to the test's temp dir and break every later local remote-code load in that worker.
  • An oracle a test loads itself is finalized as production loads are. finalize_loaded_model runs once its weights sit on their device: transformers 5 leaves a remote-code family's rotary table as uninitialized memory, and a bare oracle reads that noise as the parallel run's error. tests/cpu/model_loading/test_load_finalization.py fails a load by a family's own trust_remote_code in a module that runs a forward without it.
  • Deterministic, seeded data. GPU tests generate synthetic problems from a fixed seed; only rank 0 prints. A seed alone does not fix a chat-templated batch, because strftime_now stamps the live date (gpt-oss's harmony system message), so build fixed batches through tests.common.ep_reference.fixed_chat_batch, which pins CHAT_TEMPLATE_NOW.
  • GPU tests register in the manifest. Drop the script under tests/gpu/ and add one TestSpec (fields); a script missing from the manifest fails collection, as does a bench*.py in neither the manifest nor _UNMANIFESTED_BENCHMARKS.
  • Run GPU tests through pytest: make test-gpu-core, or a marker expression over the directory (pytest -m "gpu and ep" tests/gpu). Only the launcher entry points (LAUNCHER_ENTRYPOINTS) are collected under tests/gpu/, so no collection imports a GPU script, and no CPU test or tests/common helper reaches one (a fixture both tiers need lives in tests/common/). The launcher allocates a free --master_port per launch from tests/common/ports.py (a pool from 20000 to the kernel's ephemeral floor, one slice per xdist worker) and points TMPDIR at a per-run dir under pytest's basetemp; never hardcode either.
  • Scratch goes through the launcher's TMPDIR: setup_cache_dirs for per-rank dirs, shared_scratch_dir (tests/common/distributed.py) for a checkpoint rank 0 writes and the peers read. Tagged dirs (SCRATCH_DIR_TAG) older than 6 h are removed at session start; checkpoint locations belong in tests/common/models.py, never a literal /mnt/....
  • Use the gpu_test_main harness (tests/common/harness.py) and build checks from the shared helpers; return ctx.metrics(trainer) so the result line carries tokens/s/GPU and peak memory (skills/write-tests/harness.md). A script whose lifecycle the harness cannot express is listed in _OWN_LIFECYCLE (tests/cpu/conventions/test_gpu_harness_conventions.py) with its reason. Shared helpers in tests/common/ keep one concern per module, with no grab-bag word in a module name (tests/cpu/conventions/test_module_names.py).
  • Cover the matrix and the rejections. Markers select the tier (core or full), the world size (1gpu 2gpu 4gpu 8gpu, matching the row's nproc) and capabilities (ep cp tp etp pp hsdp vlm lora moe plus the model family); vllm_server / sglang_server need a live container and are always full. A new parallelism mode needs a correctness test per supported combination and a test that the unsupported ones are rejected.

Test knobs

The GPU launcher (tests/gpu/conftest.py) reads its knobs with raw os.environ, since importing halo/env.py would pull torch into the launcher, so each is matched as a literal string:

VarMeaning
VLLM_SERVER_URL / SGLANG_SERVER_URLThe live server the vllm_server / sglang_server tiers drive (default http://localhost:8000 / :30000); set by the shipped infrastructure (Makefile, docker-compose.vllm.yml), hence no HALO_TEST_ prefix
HALO_TEST_LAUNCH_IDSet by the launcher on every launch so the orphan sweep finds surviving workers in /proc; do not export it
HALO_TEST_NPROCSet by the launcher for a launcher="python" row: the rank count its script starts (halo launch -n)
HALO_TEST_REQUIRE_SERVERThe engine name (vllm / sglang, set by the make server tiers) whose tests must not skip: an unreachable endpoint raises a UsageError instead

Test scripts read their own knobs through halo/env.py, prefixed HALO_TEST_ so a stray export cannot collide with them (the one exception is the server-side VLLM_MODEL below). Cross-suite knobs:

VarMeaning
HALO_TEST_MODELCheckpoint of the suites that sweep architectures: the expert-LoRA, router modules_to_save, mixed merged-save and expert-LoRA reference suites (default gpt-oss-20b) and the dense self-distillation text suite (default Qwen3-0.6B)
HALO_TEST_EP / HALO_TEST_CP / HALO_TEST_TP / HALO_TEST_ETPParallel size a sweep-capable suite builds its ParallelismConfig with (unset: the suite's default); the suffix spells the axis as the config does (ep_size / cp_size / tp_size / expert_tp_size)
HALO_TEST_ATTN / HALO_TEST_GC / HALO_TEST_REVISIONAttention implementation, gradient checkpointing (default on) and hub revision for the suites that sweep them
HALO_TEST_REQUIRE_HUB_CACHEA CPU test whose Hub repo is missing from the local cache fails instead of skipping (Hub seed); gated repos and local paths still skip. Load through the tests/common/tokenizers.py helpers and name the repo in tests/common/models.py
HALO_TEST_OFFGRPO_PARALLELtp (default, dense Qwen3) or ep (gpt-oss MoE) leg of trainers/grpo/test_offline_grpo_tp_resume.py
HALO_TEST_MAX_STEPS, HALO_TEST_BATCH_SIZE, HALO_TEST_GRAD_ACCUM, HALO_TEST_NUM_GENERATIONS, HALO_TEST_NUM_WORKERS, HALO_TEST_MAX_CONCURRENT, HALO_TEST_ROLLOUT_MAX_TOKENS, HALO_TEST_MAX_COMPLETIONStep count and rollout sizing of trainers/grpo/test_environmental_grpo_benchmarks.py, sized for one vLLM server
HALO_TEST_BENCHMARK_DATASETS_CACHEA writable datasets cache holding openai/gsm8k (main), basicv8vc/SimpleQA, TIGER-Lab/MMLU-Pro and deepmind/code_contests, which the same suite reads instead of the empty per-run cache, so it runs offline. The make tiers do not forward it: pass EXTRA_DOCKER_ENV="-e HALO_TEST_BENCHMARK_DATASETS_CACHE=..." with a path under the mounted $(HALO_SCRATCH)
HALO_TEST_VLLM_GROUP_PORT / HALO_TEST_SGLANG_GROUP_PORTPin the trainer-side group port the rollout e2e suites bind; unset, each run draws one with free_port() below the ephemeral range. The online-GRPO rows offset a pinned port by the row's index, a resume row rebinds for phase 2, and the re-init suite every cycle
HALO_TEST_SGLANG_SERVER_URLS / HALO_TEST_SYNC_SERVER_CONTROLThe SGLang servers of the single_group weight-sync rows (trainers/grpo/test_single_group_sync.py, --backend sglang --servers N; the vLLM rows read HALO_TEST_VLLM_SERVER_URLS), each serving Qwen3-0.6B; a row skips with fewer URLs than --servers. trainers/grpo/test_single_group_failure.py also needs an executable in HALO_TEST_SYNC_SERVER_CONTROL, called <it> kill <index>, <it> restart <index> or <it> restart-all and returning once the servers it touched serve again; it skips without one
HALO_TEST_VLLM_SERVER_URLSComma-separated vLLM endpoints of the one-process (*_world1) rows of trainers/grpo/test_online_grpo_vllm_e2e.py, each serving the same checkpoint on its own GPU (default: HALO_TEST_VLLM_DENSE_SERVER_URL). An --trainer environmental row pushes through a WeightSyncPool over the first --servers of them and asserts the policy moved on every one; --servers 2 skips with fewer URLs. The two-rank --servers row of trainers/grpo/test_env_grpo_e2e.py reads the same list (default: VLLM_SERVER_URL), each URL serving that row's checkpoint
VLLM_MODELCheckpoint the running vLLM server serves, read by the rollout benchmark and not forwarded by the make tiers (put it in .env or EXTRA_DOCKER_ENV). It must be the model the test trains: unset, the benchmark falls back to Qwen3-0.6B

Suites that pin one family or one phase add HALO_TEST_<SUITE>_* knobs, named by the manifest entry and the script:

VarMeaning
HALO_TEST_<SUITE>_MODELSwap the checkpoint a family or phase suite loads; the per-suite spelling keeps a global export from pointing a family test at another family. Per family (HALO_TEST_ZAYA_MODEL, HALO_TEST_GLM4_MODEL, HALO_TEST_GEMMA4_MODEL, HALO_TEST_QWEN3_5_MODEL, …) and per phase: HALO_TEST_EP_RT_MODEL (EP round-trip, default a local Gemma4-26B-A4B checkpoint), HALO_TEST_EP_CP_RT_MODEL (EP+CP round-trip, local gpt-oss-20b), HALO_TEST_EP1_KNOB_MODEL (ep1 weight sync, default gpt-oss), HALO_TEST_RESUME_MODEL / HALO_TEST_RESUME_EP_MODEL (SFT resume: dense Qwen3-0.6B, and the MoE the ep mode needs), HALO_TEST_STEP3P7_MODEL (the Step-3.7 vLLM sync suite serves its own --write-checkpoint tree, so it has no default and skips without one) and HALO_TEST_GLM5_NEXT_MODEL (the same for GLM-5 Next's test_weight_push_exact.py rows), HALO_TEST_QWEN4_EXP_MODEL (the Qwen3.8-Flash-Next real-checkpoint suite, tests/gpu/models/test_qwen4_exp_real_checkpoint.py: a repo id it reads from the local Hub cache only, or a directory), HALO_TEST_NEMOTRON_MODEL and HALO_TEST_NEMOTRON_LIGHTNING_MODEL (the Nemotron-H release suite, tests/gpu/models/test_nemotron_h_real_checkpoint.py, and the MTP real and serving rows, one per --release, default that Hub release). A suite whose local default is absent skips
HALO_TEST_MTP_REAL_MODEL / HALO_TEST_MTP_REAL_STALE_HEADThe trunk (default Qwen/Qwen3.6-35B-A3B) and the checkpoint holding the stale head (default Qwen/Qwen3.5-35B-A3B) of tests/gpu/trainers/sft/test_sft_mtp_head_real.py and the trunk of test_sft_mtp_head_cost.py. Set to anything else, a row records its numbers and holds no bar read off the defaults
HALO_TEST_ZAYA_EP_STEPS / HALO_TEST_ZAYA_EP_SEQSteps (default 4, asserted) and max_length (default 512) of the ZAYA1-8B EP suite (tests/gpu/parallelism/ep/test_zaya_ep.py)
HALO_TEST_ZAYA_FSDP_STEPSSteps (default 4) of the ZAYA1-8B FSDP2 suite (tests/gpu/trainers/sft/test_zaya_fsdp.py)
HALO_TEST_EP_RT_ATTN / HALO_TEST_EP_RT_SCOPE / HALO_TEST_EP_RT_EXPERT_TP / HALO_TEST_EP_RT_KEEPEP save/reload round-trip (tests/gpu/parallelism/ep/test_ep_save_reload_roundtrip.py): attention (default sdpa), ep_scope (default auto; node / global pick the node-local or cross-node gathered save), expert_tp_size (default 1; at HALO_TEST_EP=1 this is pure ETP), and _KEEP to leave the checkpoint on disk
HALO_TEST_EP_CP_RT_ATTN / HALO_TEST_EP_CP_RT_KEEPThe same two for the EP+CP round-trip (tests/gpu/parallelism/combined/test_ep_cp_save_reload_roundtrip.py; attention defaults to flash_attention_2)
HALO_TEST_EP1_KNOB_ATTN / HALO_TEST_EP1_KNOB_LAZYep1 fsdp_shard_ep1_experts weight-sync suite (tests/gpu/parallelism/ep/test_ep1_knob_weight_sync.py): attention (default flash_attention_2) and ep_lazy_loading (default on; =0 loads through from_pretrained + EP patching)
HALO_TEST_EP_FORWARD_DELAY_MSMilliseconds every EP layer of tests/gpu/profiling/benchmark_sft_ep.py idles the GPU before each forward (default 0): the bench band's negative control
HALO_TEST_RESUME_FSDP_RESHARDfsdp_reshard_after_forward (default off) for the fsdp, fsdp_base and cp modes of tests/gpu/trainers/sft/test_sft_checkpoint_resume.py (TP+DP+FULL_SHARD is config-rejected)
HALO_TEST_VLLM_DENSE_SERVER_URLEndpoint of the dense and one-process rows of trainers/grpo/test_online_grpo_vllm_e2e.py and of the vLLM re-init and pool-reconnect suites; the MoE rows train another checkpoint, so one VLLM_SERVER_URL cannot carry both (unset: VLLM_SERVER_URL, then http://localhost:8010)
HALO_TEST_ONLINE_GRPO_MOE_MODEL / HALO_TEST_ONLINE_GRPO_DENSE_MODELCheckpoints of that e2e: MoE rows Qwen/Qwen3-30B-A3B-Instruct-2507 (EP/ETP), dense rows Qwen/Qwen3-0.6B (TP, FSDP2-DP, one-process); the server must serve the same one
HALO_TEST_ENV_GRPO_MODEL / HALO_TEST_ENV_GRPO_MAX_STEPSCheckpoint (default Qwen/Qwen3-30B-A3B-Instruct-2507) and steps (default 2; a --prefetch-depth N row runs N + 2: its N cold rounds, then two that pop a prefetched one) of the two-rank vLLM rows of the Environmental-GRPO e2e (tests/common/env_grpo_e2e.py)
HALO_TEST_ENV_GRPO_SGLANG_MODEL / HALO_TEST_ENV_GRPO_4GPU_MODELCheckpoints of the SGLang rows and of the four-rank vLLM rows (EP+ETP, EP+TP, ep4) of trainers/grpo/test_env_grpo_e2e.py, both default unsloth/gpt-oss-20b-BF16
HALO_TEST_ENV_GRPO_ATTN_IMPL / HALO_TEST_ENV_GRPO_LORA_TARGETS / HALO_TEST_ENV_GRPO_TRUST_REMOTE_CODEPer-family overrides for that e2e: the policy's attn_implementation (unset: auto-selection; sdpa for a remote-code family without FA4) and a comma-separated lora_target_modules for the --peft lora rows (unset: the attention projections read off the index; needed where they do not end in _proj, such as Ling's query_key_value,dense) The last (default on) is off for Step-3.7, whose auto_map is for the server: its trainer takes the native class.
HALO_TEST_PUSH_EXACT_FRESH / HALO_TEST_PUSH_EXACT_MODELtrainers/grpo/test_weight_push_exact.py: the first states the server has taken no other suite's push, without which the suite skips, since a server another push moved cannot be told from one this push moved; the second is a local path standing in for the row's checkpoint on both sides (the script header has the server launch)
HALO_TEST_VLLM_REINIT_MODEL / HALO_TEST_VLLM_REINIT_CYCLES / HALO_TEST_VLLM_SERVER_GPUCheckpoint (default Qwen/Qwen3-0.6B), connect/sync/disconnect cycles (default 12, enough for a stranded communicator to overshoot the 1 GiB growth budget) and the server's nvidia-smi index (unset: every GPU the trainer does not own) for trainers/grpo/test_weight_transfer_reinit.py --backend vllm; the pool-reconnect suite reads the same checkpoint knob
HALO_TEST_SGLANG_REINIT_MODEL / HALO_TEST_SGLANG_REINIT_CYCLES / HALO_TEST_SGLANG_SERVER_GPUThe same three for --backend sglang, against SGLANG_SERVER_URL

The local checkpoints the per-suite overrides default to (Gemma4-26B-A4B for HALO_TEST_GEMMA4_MODEL and HALO_TEST_EP_RT_MODEL, gpt-oss-20b for HALO_TEST_EP_CP_RT_MODEL) are patch_vocab.py outputs at $HALO_DATA_ROOT/models/<repo name>-patched (patched_checkpoint_dir in tests/common/models.py), under $(HALO_SCRATCH)/models/ in the make tiers. The suites need a local directory, not added tokens, so the tool runs without --patterns:

Bash

python scripts/before_training/patch_vocab.py --model_id google/gemma-4-26B-A4B-it \
    --output_dir "${HALO_DATA_ROOT:?}/models/gemma-4-26B-A4B-it-patched"
python scripts/before_training/patch_vocab.py --model_id unsloth/gpt-oss-20b-BF16 \
    --output_dir "${HALO_DATA_ROOT:?}/models/gpt-oss-20b-BF16-patched"

Manifest rows and tiers

A spec launches its args_matrix entries under its nproc, markers, timeout and flaky; a Row entry sets its own launch values, so one script runs at several widths, tiers or engines, each row holding gpu, one tier and the Ngpu tag of its own nproc (tests/cpu/conventions/test_gpu_harness_conventions.py). A leg the script can name becomes a CLI flag with one row per leg, so each gets its own pytest node and verdict (--trainer / --mode on trainers/grpo/test_online_grpo_vllm_e2e.py; --backend vllm / sglang on trainers/grpo/test_env_grpo_e2e.py; one --mode per parallel shape on trainers/sft/test_sft_families.py and trainers/sft/test_sft_gptoss_modes.py; --ep-loading lazy / eager on trainers/grpo/test_offline_grpo_reference.py and test_offline_grpo_ep_cp.py, pinned by tests/cpu/conventions/test_offline_grpo_ep_loading_rows.py). Rows that keep the spec's markers share their union, so -m "gpu and cp" also selects the non-CP rows of such an entry. A leg that only swaps the checkpoint stays an env override (HALO_TEST_<SUITE>_MODEL): a per-family pass is a run of that row with the override set.

core is the pre-merge gate, and small-and-fast (≤2 GPUs, tiny model) is its intent; size the host from the manifest, not from that intent. Most manifest entries carry a core row; within the tier some rows need 4 GPUs, many declare 1500–2400 s timeouts, and a large minority load a real multi-billion-parameter checkpoint (gpt-oss-20b, ZAYA1-8B, Qwen3-30B-A3B, Qwen3.5-2B, Qwen3-VL-2B and three Ling/Ring checkpoints), so the summed worst-case timeouts pass 100 hours. The manifest is the only place exact counts live. A big-checkpoint entry stays core only as a correctness gate, a comparison against an independent reference (parallelism/ep/test_ep_modes_vs_reference.py: gpt-oss and Qwen3-MoE parallel modes vs the dense reference); smokes on the same checkpoint (trainers/sft/test_sft_gptoss_modes.py, trainers/lora/test_lora_mixed_merged_save.py) are full.

Run make test-gpu-core deliberately, not as a quick check, and mark a new heavy or many-GPU test full. gpu-tests.yml budgets the tier 6 h; a new core entry that pushes past that belongs in full.

A test must not be AI-slop

A test earns its place only if it fails when the behavior breaks: mutate the function under test (flip a sign, drop a clause, return the input) and a real test goes red. These stay green:

Anti-patternWhat to do instead
Tautology — assert True, assert x == x, or asserting a value the test just setAssert the computed output against an independently known expected value
Smoke-only — call it, assert it "did not raise"Assert the return value / shape / loss
Mock the thing under test, then assert the mock returned what you told it toMock only the boundary (network, GPU, clock); call the real function
Vacuous assertion — assert out is not None, assert len(out) >= 0Assert the exact value, correct length, or a tight bound with a reason
Test the framework — assert isinstance(cfg, ParallelismConfig)Assert the property your code computes (cfg.data_parallel_size == 4)
Swallow-and-pass — try: real_call() except Exception: passLet it raise, or pytest.raises(ValueError, match=...)
Golden with no meaning — a literal pasted from one runDerive the expected value from the math; if it is a true golden, say why

tests/cpu/parallelism/test_parallelism_config.py is the voice to copy. Cover the edge cases (empty input, the unsupported combination, the degenerate near-zero case): that is where regressions hide.

Golden performance baselines

make bench runs the rows of tests/gpu/profiling/run_baseline_benchmarks.sh on one 8-GPU node, each a torchrun launch at seq 4096, batch 1, with gradient checkpointing and the production attention and grouped-GEMM defaults:

RowsScriptKeys
Qwen3-8B under FSDP2, CP2, TP2benchmark_sft_dense.pysft_dense_qwen3-8b_{fsdp,cp2,tp2}_s4096
gpt-oss-20b under EP1, EP2, EP8, EP8 × TP2, EP4 × ETP2benchmark_sft_ep.pysft_ep_gpt-oss-20b_{ep1,ep2,ep8,ep8_tp2,ep4_etp2}_s4096
the EP8 row's gathered save (--measure_save)benchmark_sft_ep.pysft_ep_gpt-oss-20b_ep8_gathered_save

Every rank prints one __HALO_BENCH__ line per key (tests/common/reporting.py). The key names the golden tests/baselines/bench/<key>.json: the commit, image, date, world size and command it was measured with, and the metrics of at least five repeats. make bench BAND=1 holds the run to them (tests/common/equivalence/bench_band.py, evidence in evidence/bench_band-<revision>.json); make bench-band re-checks the logs make bench left at HEAD without GPUs.

  • The band. Each gated metric's band is the mean of its baseline repeats, give or take the larger of a margin and twice their spread. tokens_per_second (mean over ranks, 2%) and save_seconds (6%) must stay inside it on either side: a faster run fails too, so a speed-up re-records its baseline. peak_allocated_gb (0.5%) and save_peak_rss_gib (1%, the save's peak resident host memory), each the highest over ranks, must not rise above the upper edge, which is never below the highest repeat. MFU, S-MFU and TFLOPS stay under metrics.diagnostics, never gated.
  • The corpus. Every row trains on create_benchmark_dataset (tests/common/datasets.py): seeded uniform token ids no model can fit, so an MoE row's routing, and the hottest EP rank's step time and peak, stay put (EP8, EP8 × TP2 and EP4 × ETP2 repeat within 1% in throughput and 0.02 GB in peak; on a corpus the model fits, EP8 moved 4% and 1.2 GB between runs).
  • A quiet host. A row sharing the host with another job's CPU or network load runs 4–17% slow; recording refuses that repeat and a band run that meets one fails.
  • Flags. Each training line carries its rank's mean and median step time (mean_step_time_ms, median_step_time_ms; the band reads rank 0's). A repeat whose mean is more than 2% off its median is flagged, never failed: re-run it before reading its throughput as a regression.
  • Same command. A key compares only with a baseline measured by the same command on as many ranks of the same GPU model; a measured key with no baseline, and a baseline the run did not measure, fail.
  • Another machine. A baseline can fail on another machine of its type with no code effect: gpt-oss-20b EP1 read 2.4% under its baseline on a second B300 machine, at the baseline's commit too. Split a miss with an A/B of that commit and HEAD on the machine that missed.

make bench RECORD=1 runs the rows five times on a clean tree and writes the baselines only when no repeat is outlying (a rate or duration outside the other repeats' band widened to four times their spread, a peak only above its upper edge), removing any baseline the run did not measure; commit the files it writes. The negative control, make bench BAND=1 EXTRA_DOCKER_ENV="-e HALO_TEST_EP_FORWARD_DELAY_MS=2", must fail every EP row's tokens_per_second and pass the dense rows and the save. EP4 runs with ETP2 because pure ep_size > 2 with ep_group_size < nvlink_domain_size is rejected at config time (Expert Parallelism).

Goldens

Each golden is a JSON file under tests/common/goldens/, written by the generator module beside it and checked against today's code by a CPU test. A refactor leaves it unchanged. Regenerate one only for a deliberate change, in a commit that says so:

Bash

python -m tests.common.goldens.<name>          # exits non-zero when the committed file is stale
python -m tests.common.goldens.<name> --regen  # rewrites it from today's code
Golden (<name>)PinsChecked by
rank_layoutFor every parallel shape of the grid: the EP, dispatch, expert-TP, expert-replica, DP-scope and CP groups and the pipeline chains, each rank's coordinates, and the EPConfig flags that decide where expert gradients sync (not the TP / FSDP DeviceMesh dimensions)tests/cpu/parallelism/test_grid_goldens.py
ep_group_orderThe new_group calls EPConfig makes, in order with their keyword arguments, which call built each group a rank keeps, and each rank's ep_group_idx and expert_replica_rankstests/cpu/parallelism/test_grid_goldens.py
load_collective_traceWhat load_distributed_model issues on ranks of a 4-rank job on two 2-GPU nodes, for every branch of its loader ladder from a local directory and the Hub, and what the frozen reference load issues (LOAD_CASES)tests/cpu/parallelism/test_load_collective_trace.py
balancing_collectivesWhat configuring MoE balancing and building the run's callbacks issue on rank 0 of a two-rank job, for every moe_balancing mode, loss kind and weight-sync flag over tiny dense and MoE stand-ins: the collectives in order, the log records between them, the raise, the callback roster and the router config fields lefttests/cpu/callbacks/test_balancing_collectives.py
weight_sync_wireThe vLLM weight-transfer bootstrap: the rendezvous TCPStore arguments, its one write (the pickled ncclUniqueId under broadcast_from/0/0), the id the communicator initializes with, and the class's module, qualname and fieldstests/cpu/weight_sync/test_wire_pins.py
weight_sync_transcriptWhat each engine's client sends through its public lifecycle in twelve scenarios per engine (dense and MoE syncs, formation and finish failures, aborts before and mid-stream, an unusable client at exit): every request with its route, body, timeout and retry policy, what went on the NCCL wire, and the at-exit hooks lefttests/cpu/weight_sync/test_request_transcript.py
registry_orderWhat each toolkit registry holds, in its own order: the EP layer classes and their claims, the CP, PP and head claims, the weight-sync clients, the Liger and MoE family specs, the sequence-classification heads, the named optimizers and the environment typestests/cpu/conventions/test_registry_order.py

The grid (tests/common/goldens/shape_grid.py) is every shape ParallelismConfig accepts with a pipeline stage of at most 32 ranks, at every NVLink domain dividing the stage, plus the same sweep at 64, 72, 128, 144 and 256 ranks on 8-GPU nodes and 72-GPU NVL72 racks; each grid golden (a GridGolden) holds a digest per shape plus the full record of a readable subset, so a failure reads as a diff. Notes per golden:

  • ep_group_order. Recorded on a gloo default group, where every group is a new_group call (an NCCL group bound to its device splits each call's groups at once: tests/cpu/parallelism/test_group_split.py). new_group is a collective over the default group, so a rank making other calls than rank 0 hangs a real job (the generator raises there), and two calls over the same members build two communicators, so a reordered call fails here even where rank_layout, which compares memberships, passes. tests/cpu/parallelism/test_parallelism_config_gate_order.py separately pins the order of ParallelismConfig's gates.
  • load_collective_trace. tests/common/collective_recorder.py runs a load as one simulated rank over torch's fake process group and an in-memory store, replacing each torch.distributed function under every name a module holds it by (a collective DTensor issues is recorded too); a collective it has no model of raises.
  • weight_sync_wire. The vLLM image unpickles the id as its own class through a shim at the same module path, so the test also checks that Dockerfile.vllm writes the shim for the module the golden names. Moving ncclUniqueId changes the pickle bytes: regenerate the golden in the commit that moves the class, and move the shim with it.
  • weight_sync_transcript. The clients run against FakeVLLMServer and FakeSGLangServer (tests/common/fake_engines.py) with NCCL and CUDA faked on the CPU.
  • registry_order. The check renders in a fresh interpreter, so the order is the one the code and the generator's imports produce, and classes are recorded by name, so a move that keeps each name and its order leaves the file unchanged.

Move kit

A structural change (a module moved or renamed, code carved between modules, a package renamed) proves it changed no code with make verify-move BASE=<sha> (tests/common/equivalence/move_kit.py). The kit reads both commits from git (HEAD_REF, default HEAD, against BASE), parses the wheel's packages and scripts/ without importing them, and writes evidence/move_kit-<head sha>.json (git-ignored) with one verdict per check:

CheckFails when
renamesa file rename the change declares is missing from git diff -M50%, or below --min-similarity (90 by default)
definitionsa module docstring, function, class header, method or module-level assignment hashes differently once each first-party name it reads is spelled in head terms, or head adds one the change does not declare
public_namesa public top-level name appears or disappears
string_literals, raise_templatesthe multiset of string literals (docstrings aside) or of raised expressions changes
side_effectsa module's ordered import-time statements change: bare calls, assignments that call first-party code or set an attribute or item, first-party decorators, control flow, and imports marked noqa: F401 or E402
goldensa golden file the change does not declare changes, a declared one does not, or a golden's generator fails its check against head's code

Spelling a name in head terms follows every rename git finds, each declared symbol rename, and each definition that moved to another module under its own name; a name a function binds hides the module-level one. A path in a string follows its file's rename, or its directory's when every Python file of that directory moved to one new directory. A library decorator (@dataclass, @lru_cache) is the definition's business, so a carve may move it. Each patch in head on a name a touched module only imports is listed under patch_targets for review, and a renamed method's callers through an instance read as changed.

Declare what the definitions do not show in a JSON file under the repo, passed by its repo-relative path as RENAMES=:

JSON

{
  "files": {"halo/<old path>.py": "halo/<new path>.py"},
  "symbols": {"halo.<module>:<old qualname>": "halo.<module>:<new qualname>"},
  "added": ["halo.<new module>:__doc__", "halo.<new module>:logger"],
  "goldens": ["weight_sync_wire.json"]
}

added names the definitions a carve adds, goldens the golden files the change regenerates on purpose; a declaration the trees contradict fails. MOVE_ARGS passes further flags: --require names the checks a carve's proof rests on (one that extracts new functions fails definitions by design), --min-similarity lowers the bar for a rename that carries edits, and --ignore-docstrings compares a prose-only change:

Bash

make verify-move BASE=<sha> RENAMES=<map.json> \
  MOVE_ARGS="--require renames string_literals raise_templates side_effects goldens --min-similarity 70"

make verify-move exits non-zero when a required check fails and runs as the caller, so the evidence is the caller's. tests/cpu/conventions/test_move_kit.py seeds each kind of change on a scratch repository.

Comparison harnesses

The checkpoint, EP, CP and efficiency harnesses share one contract: each runs its own module in BASE and in HEAD_REF with that tree's code (so BASE must already carry it) and writes evidence/<harness>-<head sha>.json (git-ignored) with every delta keyed by its path in the output. ACCEPT="<case> ..." names the cases whose deltas a flagged change declares, each a case path or a prefix of one (ACCEPT=cpu/qwen3_moe covers every EP record of that family); their deltas stay in the evidence under accepted without failing the verdict, and FAMILIES= limits a run to some tiny families. The checkpoint and EP harnesses run only the families both trees' FAMILIES rosters hold, so a change that adds, drops or renames a family still compares the rest against its base; each family left out is logged by the tree whose roster lacks it and listed under skipped (tests/cpu/conventions/test_harness_family_rosters.py). Every verdict also fails on an unexpected raise. The EP, CP and efficiency harnesses run a negative control: the head tree reruns one case with a deliberate perturbation, and the verdict fails when that record matches its base or raised.

  • Checkpoint artifacts (make verify-artifacts, tests/common/equivalence/checkpoint_artifacts.py) saves a tiny model of every family (the dense roster, every EP family and its multimodal wrappers) at fp32 and at bf16 through every saver mode one CPU process runs, as an export and as a training checkpoint: the FSDP2 gathered stream, CP over plain parameters and over FSDP2, the TP saver, EP with FSDP2-managed experts (ep_size == 1) and with FSDP-ignored EP layers, native expert LoRA as an adapter and merged, and attention LoRA through the PEFT saver; a full-weight training checkpoint adds the optimizer shard and its meta after one AdamW step. It compares file by file: the file set and permission bits, each safetensors header and tensor digest, each JSON by value, each torch file by key and tensor. Both producers run under one string hash seed, since a set's order reaches adapter_config.json. The per-rank EP save, the PP stage save and every multi-rank gather need more ranks, which the GPU checkpoint suites cover (tests/cpu/checkpoint/test_checkpoint_artifacts.py holds the seeded changes it catches).
  • EP bitwise (make verify-ep, tests/common/equivalence/ep_bitwise.py) compares every EP family's records bit for bit: per forward case the loss, logits, every parameter gradient, the routing each EP layer dispatched and the expert-load counts; per case the ordered parameters, state-dict keys, weight-decay groups and each parameter's gradient hooks, recorded by what they do to a gradient of ones (each all-reduce with its op, dtype and group ranks, and the scale left). TIER=cpu (the default) runs the per-expert loop on one process (defaults, an fp32 router on FSDP-ignored experts, native expert LoRA with dropout, an armed routing-replay mask, bias-update balancing) and builds every rank of a few layouts on stubbed groups; TIER=gpu runs the grouped-GEMM path in bf16 under full_determinism (DeepEP's deterministic dispatch included) on one rank, ep2 (2 GPUs) and ep2+etp2 (4 GPUs), each with HALO_EP_SHARED_OVERLAP off and on. Records outside KNOWN_RAISING must not raise; the negative control rotates every expert bank by one expert (tests/cpu/parallelism/test_ep_bitwise_harness.py). Both trees read one Triton cache, so it does not compare autotune picks across caches, and another run filling that cache beside it can read as deltas on the families that call fla's kernels (Docker).
  • CP equivalence (make verify-cp, tests/common/equivalence/cp_equivalence.py) runs a two-rank CP group in each tree on the CP roster, tiny models covering every Ulysses wrapper class (the Qwen3 wrapper on its dense, MoE and VL classes), at two levels: the first attention layer on fixed hidden states, and the model under UlyssesModel through its loss. A family the base tree's roster lacks has nothing to compare with: it is skipped and listed under skipped. The base tree runs twice and must repeat bit for bit; a record whose base does not repeat is compared forward bit for bit with each gradient inside its base-vs-base envelope (listed under enveloped). TIER=gpu runs bf16 under full_determinism with the flash kernel CP picks per family (FA4 on Blackwell where the family allows it, FA2 otherwise, FA3 on Hopper); the CPU tier runs fp32 on gloo with a plain torch attention under the flash contract. The negative control moves one attention weight by a few ulps (tests/cpu/parallelism/test_cp_equivalence_harness.py).
  • Efficiency (make verify-efficiency, tests/common/equivalence/efficiency_callback.py) drives each tree's EfficiencyCallback through the same scripted steps and compares every logs dict it fills, every record logged under the package, and the snapshot_efficiency row. CUDA answers as a scripted B300 and the clock, token counter and document source follow a fixed schedule, so every figure is a function of the code. The cases cover every family on one rank and under EP with FSDP2, three families on every parallel shape (pipeline stages included) built with the real wraps on torch's fake process group, and the edge scenarios (an unknown GPU, fp8 / fp32 / tf32, a frozen LoRA base, an estimate that raises, no layer list, no parameters); a case only one tree has is listed under new or dropped. The negative control moves one step's clock by a tick (tests/cpu/callbacks/test_efficiency_harness.py).

Docs

Update the owning doc page in the same PR when you change halo/ (skills/docs/docs-ownership.md is the map), following the style charter of the docs skill (skills/docs/style.md). make docs runs the link, anchor and code-reference check that blocks a merge (Continuous Integration → Hosted tier). Each figure under agent-docs/assets/ is rendered from the HTML page beside it, which holds its data and draws it in the visuals skill's style (skills/visuals/SKILL.md). Every page links the skill's shared kit (skills/visuals/kit/: the colour tokens, type scale and page skeleton, and the element, arrow and axis helpers), so a change there reaches every figure on the next render. Edit the page or the kit, run make diagrams on a host with Chrome, the skill's fonts and OptiPNG, and commit the page, the PNG, agent-docs/assets/figures.sha256 and any alt-text change together. Every render is optimized losslessly with OptiPNG. CI does not re-render the figures, since the fonts are licensed and stay off the runners; it fails a figure left stale against what it is rendered from (Continuous Integration → Hosted tier).

Proof of Value

Change typeWhat to show
No-op / refactorLoss is bitwise-identical to main at a fixed seed, and the original checkpoint still loads
Behavior changeAn end-to-end test demonstrating the new behavior or convergence
PerformanceBefore/after tokens/s/GPU and peak memory (utilization is a diagnostic, not the headline)

State the GPU, model and sequence/batch shape with any number. Report a measured net-slower result too, with the reason.

Submitting a PR

  1. Get approved first and work from a signed fork branch: the gate, the fork workflow, the signed-commit setup and the secrets rule are in CONTRIBUTING.md.
  2. Pass the gates. make lint, make format, make seed-hf-cache (again when tests/common/models.py or examples/ gain a repo), then make test-cpu (plus make test-gpu-core for GPU-affecting changes) and make docs. The test tiers run locally, so report their result in the PR.
  3. Fill the PR template: what and why, type of change, Proof-of-Value evidence, checklist.

Where to look next

Source on GitHubUpdated 18 November 2026