# Continuous Integration

Halo's CI is GitHub Actions (`.github/workflows/`), in two tiers by where the runner lives, plus the
contribution gate.

| Tier | Runner | Workflows | Triggers | Status |
|------|--------|-----------|----------|--------|
| Hosted | GitHub `ubuntu-latest` | `.github/workflows/lint.yml`, `.github/workflows/docs.yml`, `.github/workflows/sensitive-label.yml` | every PR + push `main` (`sensitive-label`: PR only) | active |
| Self-hosted | GPU box `[self-hosted, halo]` | `.github/workflows/cpu-tests.yml`, `.github/workflows/gpu-tests.yml` | `workflow_dispatch` | dispatch-only |

Hosted jobs need no image. The test tiers import torch and run inside the prebuilt image, built on the
self-hosted box and never pulled or rebuilt by CI; GPU tests also need Blackwell (SM100) or Hopper
(SM90) for FA3/FA4 and DeepEP, which no hosted runner provides. CodeQL (the default code-scanning
setup), GitGuardian and CodeRabbit report on every PR from outside `.github/workflows/`.

## Hosted tier

- **`lint.yml`.** `ruff format --check` plus the full pyproject rule set (`PLC0415`, the no-inline-imports
  rule, included), and `actionlint` over the workflows (self-hosted labels declared in
  `.github/actionlint.yaml`). Its `conventions` job runs the convention tests it lists, the ones that
  read the tree without importing it, on Python 3.12 with pytest alone;
  `tests/cpu/conventions/test_hosted_conventions_are_stdlib.py` fails, in the image too, when a listed
  file, a package `__init__.py` or conftest pytest runs for it, or a file one of them imports or loads
  reaches torch, `halo` or any other non-stdlib import. `.github/scripts/locked_version.py` reads each
  job's tool pin (ruff, pytest) out of `uv.lock`.
- **`docs.yml`.** `scripts/docs/check_links.sh` blocks on a broken relative link or heading anchor across
  `agent-docs/`, `human-docs/`, `skills/` and the root markdown (an anchor must be GitHub's slug, so a
  `{#id}` heading fails), and resolves their code: a repo-root path must exist, and a dotted first-party
  name must name a module and a name it binds, read off the source. `scripts/docs/figure_manifest.sh
  check` blocks on a stale figure: `make diagrams` records in `agent-docs/assets/figures.sha256` the
  SHA-256 of every figure PNG and of what it is rendered from (its page, the data folders the page names,
  `skills/visuals/kit/`, `render.sh` and `canvas-size.sh`), and any of them changed since, or a page
  never rendered, fails.
- **`sensitive-label.yml`.** A PR whose own commits change a path listed in `.github/sensitive-paths` (a
  moved file counts under its old path) fails until it carries the `sensitive` label, which a maintainer
  adds once the change is approved as `CLAUDE.md` requires; a label change re-runs it. It runs on
  `pull_request_target`, so the workflow, the check (`.github/scripts/check_sensitive_label.py`) and the
  list are the base branch's, and a PR that edits any of them is still judged by the base's copy.
  `tests/cpu/conventions/test_sensitive_label.py` also fails when a listed glob matches no file.

Blocking is the `main` ruleset's **required status checks** (`ruff`, `actionlint`, `build`); a red check
outside that set still lets a PR merge. `conventions` and `sensitive-label` block
once a repo admin adds them to the set and creates the `sensitive` label.

`.pre-commit-config.yaml` mirrors the ruff gates on the host at commit time, plus whitespace fixers and
a 1 MB file-size cap (optional: `pipx install pre-commit`, then `pre-commit install`).

## Self-hosted tier

`cpu-tests.yml` asserts the image is present and runs `make test-cpu` (the image without `--gpus`, so it
does not contend with GPU jobs on the runner) in one pytest process, uploading its JUnit XML. One process
takes about 3.5 to 4.8 hours (estimated from samples), inside the workflow's 360-minute cap.

`gpu-tests.yml` asserts the image (`halo:blackwell` by default) and runs
`make test-gpu-core ENV_FILE= AWS_DIR=`: creds-free, `-m "gpu and core"`, JUnit XML as an artifact. Once
its `pull_request` trigger is enabled a PR run needs **both** a non-draft PR and the `run-ci-gpu` label;
it has no `push` trigger, which would fire the tier on every merge with no label gate, so post-merge
runs go through `workflow_dispatch`. Its `timeout-minutes` (360) is the budget the `core` tier must fit
in; a new core entry that pushes the tier past it belongs in `full`
([tier composition](https://whitecircle.com/halo/docs/agent-docs/contributing.md#tests)).

The full GPU tier (`make test-gpu-full`) and the two server tiers (`make test-gpu-vllm` /
`make test-gpu-sglang`, each needing its server running on a GPU outside `TRAINER_CUDA_DEVICES`) have no
workflow: run them by hand. A per-family pass of a server tier serves the family's checkpoint and
points the async GRPO rows at it (`HALO_TEST_ENV_GRPO_MODEL` for vLLM, `HALO_TEST_ENV_GRPO_SGLANG_MODEL`
for SGLang); the rows move the served policy with an expert-only perturbation as well as a dense one.
A vLLM pass also serves the family's tool parser
([Tool parsers are per family](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#running-it)): under one that cannot read the family's
calls (the compose default `hermes` reads JSON `<tool_call>` calls only) the rows raise by their second round
([tool-call watch](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#running-it)).

### Hub seed

`make seed-hf-cache` (`tests/common/hub_seed.py`) fetches what the CPU tier reads from the Hub into
`HF_CACHE`: for every Hub id an `examples/` config trains, every checkpoint constant in
`tests/common/models.py` and the revisions its `PINNED_REVISIONS` pins, the configs, tokenizers, chat
templates and remote-code `*.py` files, never weights. It names any repo it cannot
fetch. Over a seeded cache the tier runs offline, and `HALO_TEST_REQUIRE_HUB_CACHE=1` turns a Hub miss
into a failure instead of a skip:

```bash
make seed-hf-cache
make test-cpu EXTRA_DOCKER_ENV="-e HF_HUB_OFFLINE=1 -e HALO_TEST_REQUIRE_HUB_CACHE=1" PYTEST_ARGS="-n 8"
```

Gated repos (`GATED_REPOS` in `tests/common/hub_seed.py`) cannot be fetched anonymously, so their cases
still skip, as do the Liger coverage checks for the remote-code Bailing V2/V3 norms (they need the
family's modeling module imported, which a config load never does). A test that loads a new repo names
it in `tests/common/models.py` so the seed carries it.

### Enabling the test tiers (repo admin)

1. Register a self-hosted runner on the GPU box (repo → Settings → Actions → Runners) with labels
   `self-hosted` and `halo`, version 2.327.1 or newer (the pinned actions run on Node 24), as a systemd
   service under a dedicated non-root user in the `docker` group.
2. Build the image on that box (`make build-blackwell`); CI reuses it, so refresh it when the
   `Dockerfile` or the dependencies change.
3. Repo → Settings → Actions → General: require approval of fork-PR runs for **all outside
   collaborators** (the *first-time contributors* setting does not gate returning ones). This must
   precede step 4: the CPU tier has no label gate, so per-run approval is its only opt-in.
4. Uncomment the `push` / `pull_request` triggers in `cpu-tests.yml` and the `pull_request` trigger in
   `gpu-tests.yml`.
5. Create the `run-ci-gpu` label.

## Release validation

The scope Halo 1.2.0 is validated on:

- **GPU:** 8× B300 machines: the core and pipeline tiers, both rollout-server tiers (vLLM 0.32.0,
  SGLang 0.5.22) and the EP and CP bit-for-bit equivalence gates (`make verify-ep` / `make verify-cp`).
- **Throughput band** (`make bench BAND=1`; [gates](https://whitecircle.com/halo/docs/agent-docs/contributing.md#dev-loop)): on a second
  B300 machine, 8 of 9 rows sat within band over three repeats, and no median moved more than 1.7% from the
  [benchmark table](https://whitecircle.com/halo/docs/agent-docs/optimization/throughput-benchmarks.md). FSDP2 read under on its mean, from a slow first
  repeat on a cold node; its second and third repeats are within band.
- **Multi-node:** two 8-GPU machines ran node-local `ep8` with DP through a per-node save, resume and
  evaluation, cross-node `ep16` through a gathered save and reload, cross-node weight sync on both
  engines, and the two PP recipes (training; the Qwen3-8B one also a per-node save, resume, merge
  and load). The other multi-node shapes run as a two-node simulation on one host
  ([Verified topologies](https://whitecircle.com/halo/docs/agent-docs/parallelism/multi-node.md#verified-topologies)).
- **Model size:** full-size checkpoints ran up to GLM-5.3-Flash (321B): `ep8` LoRA on one machine, `ep16` and
  `pp2 × ep8` full fine-tunes on two 8-GPU machines. Also Qwen3.6-35B-A3B (`ep8`), Nemotron 3.5 Super VL at `ep8`
  for 5 to 20 steps per recipe, a few full-depth EP8 LoRA steps of Qwen3.8-Flash-Next, and Step-3.7 Flash as a weight
  push at EP=2 and one expert-LoRA step at EP=4. gpt-oss-120b and Qwen3.5-122B-A10B at EP8 last ran on earlier code.
- **Upgrades:** checkpoints written by 1.1.0 resume with the optimizer restored (step counters continue, every
  second moment carries over, losses continue), and export and merge: FSDP2 full fine-tuning, LoRA adapters, EP
  gathered saves, EP sharded saves after `merge_ep_shards.py`, and async GRPO whose prefetched rounds hold plain
  prompts and answers.

## Contribution gate

Three workflows implement the issue-first gate [`CONTRIBUTING.md`](https://github.com/whitecircle/halo/blob/872a3e4284ef3a533c18b651f91b76adcbdcf306/CONTRIBUTING.md#the-gate-read-first)
states:

- `.github/workflows/pr-gate.yml` (`pull_request_target`, opened/reopened) comments on and closes a PR
  whose author is neither a bot, nor a write-access maintainer, nor listed in
  `.github/APPROVED_CONTRIBUTORS`. It reads the allowlist through the API and checks out nothing, so PR
  code never runs.
- `.github/workflows/approve-contributor.yml`: a maintainer's `/approve @username` comment (several lines
  approve several users) appends each user to the allowlist and assigns them to the issue (`stale.yml`
  exempts assigned issues; a user GitHub refuses to assign, one who never commented there, is reported
  with a `keep-open` hint). The commenter's write access is verified first, since `issue_comment` fires
  on PR comments too; a permission check that errors fails the run on the issue rather than approving
  nobody in silence. Bot comments are skipped.
- `.github/workflows/approve-merged-contributor.yml` adds a merged PR's author to the allowlist, so
  repeat contributors skip the gate (write-access authors pass on that access and are not listed).

The org's signed-commit ruleset covers every branch with no bypass actor, so a PR merges only once every
commit on its head, a fork's included, is signed.

The allowlist lives on the **`allowlist` branch**, outside `main`'s ruleset, because GitHub refuses the
Actions app as a ruleset bypass actor. The approval workflows commit to it through
`createOrUpdateFileContents`, whose commits arrive GitHub-signed; `main` keeps a pointer stub at the
same path. Failures (missing branch, permissions) are reported on the issue or PR, and concurrent
approvals refetch and retry once on a file-sha race. `pr-gate.yml` fails open on a transient API error
(a maintainer's PR must never be auto-closed by a 500) but treats a missing branch or file as an empty
list and gates every outsider, so protect the branch with a repository ruleset on
`refs/heads/allowlist` carrying the `deletion` and `non_fast_forward` rules. If it is ever deleted,
recreate it as a single signed commit whose tree holds `.github/APPROVED_CONTRIBUTORS` (one username per
line, `#` comments).

## Security

The self-hosted runner executes contributor code on your hardware, beside training and secrets:

- **The GPU tier is creds-free but mounts the scratch volume.** `make test-gpu-core ENV_FILE= AWS_DIR=`
  drops the `.env` and `~/.aws` mounts, but `MNT_MOUNT` still bind-mounts all of `HALO_SCRATCH`
  read-write: HF cache, dataset caches, checkpoints. Fork-PR approval is the boundary; the label only
  picks which approved PRs run. The mount and the in-container `HF_HOME` / `HF_DATASETS_CACHE` /
  `TMPDIR` / `HALO_DATA_ROOT` all derive from `HALO_SCRATCH`, so export it in the runner's environment:
  unset, it resolves to the local mount with the most free space, which can move the seeded HF cache
  between runs. Inject `HF_TOKEN` from a repo secret only when a gated model is needed.
- **The CPU tier mounts the host's HF cache.** `DOCKER_RUN_CPU` bind-mounts `HF_CACHE` (default
  `$(HALO_SCRATCH)/hf`) read-write, so PR code can write into the cache every later job reads. CPU tests
  that call `from_pretrained` directly hard-fail when the cache is missing and the Hub is unreachable;
  tests going through `tests/common/tokenizers.py` skip. `HF_CACHE=` runs cache-less: those tests skip,
  and the guards that refuse an all-skipped file fail.
- **No repository secrets to fork code.** The test triggers are `pull_request`, not
  `pull_request_target`, so repo secrets stay away from fork PR code (a same-repo branch is a
  maintainer's and gets them). `pr-gate.yml` uses `pull_request_target` and checks out nothing;
  `sensitive-label.yml` uses it so a PR cannot edit its own gate, checks out the base branch, and only
  fetches and diffs the PR's commits with a read-only token.
- **The label gate is not the boundary.** A `pull_request` run executes the PR's own copy of the workflow
  and the Makefile, and the label survives later pushes, so fork approval is what keeps unreviewed code
  off the box. Require it for **all outside collaborators** *before* enabling the triggers: `pr-gate`
  closing an unapproved PR does not stop workflows the same `opened` event already started.

## Repo hygiene

`.github/workflows/stale.yml` (daily, 01:30 UTC) marks issues idle for 30 days stale and closes them 7
days later. A `keep-open` or `help wanted` label, an assignee or a milestone exempts an issue, so an
accepted issue (`/approve` assigns its contributor) stays open while it is worked. PRs are left alone.

`.github/workflows/branch-cleanup.yml` (weekly, Monday 02:00 UTC) deletes remote branches older than 90
days with no open PR, never the default branch, `allowlist` or `release/*`; a branch a ruleset shields
fails its delete and is logged. Every deleted tip SHA is logged (recoverable by SHA), and
`workflow_dispatch` defaults to a dry run that prints the list.
