Docker Guide
The host has no usable Python: PyTorch, DeepEP and Flash Attention live only inside the images, so anything that executes runs inside one.
halo:hopper— H100/H200 (SM90): FA2 + FA3 + DeepEP.halo:blackwell— B200 (SM100) / B300 (SM103): FA2 + FA4 + DeepEP (no FA3).halo:vllm-0.32.0— the vLLM rollout server (separate container).halo:sglang-0.5.22— the SGLang rollout server (separate container).
All four are published to Amazon ECR Public under public.ecr.aws/whitecircle/halo with the same tags
(anonymous pulls, no AWS account; Registry), or build credential-free from source
(Building) under these local names. The images are x86_64-only: Grace-based GB200/GB300
hosts (aarch64) need an arm64 build the Dockerfile does not provide.
The host needs an NVIDIA driver R580 or newer, the CUDA 13 floor the images' CUDA 13.2 runs on (the GPU tiers run on 580.159.03), and the NVIDIA Container Toolkit.
Standard training launch
The Makefile wraps the docker run once; train runs halo launch with METHOD (default sft):
Bash
make train CONFIG=examples/sft/gptoss/gptoss-20b-multinode-ep.yaml NPROC=8 EXTRA="--expert_parallel_size=8 --use_grouped_gemm=true"The Makefile's DOCKER_RUN runs in the foreground and adds --network host (the server GPU tiers
reach the compose server at localhost:8000) plus -e PYTHONPATH=/workspace -e CUDA_DEVICE_MAX_CONNECTIONS=1. It mounts --env-file and ~/.aws only while ENV_FILE / AWS_DIR
are non-empty, so CI runs creds-free. It omits --cap-add=SYS_PTRACE, so py-spy attaches to a make
job only where kernel.yama.ptrace_scope is 0
(Debugging).
The detached equivalent:
Bash
# The local mount other than / (not tmpfs, overlay or a network FS) with the most free space over 20 GB.
D=$(findmnt -rbno TARGET,AVAIL,FSTYPE | awk '$1!="/" && $3!~/tmpfs|overlay|squashfs|nfs|fuse|autofs/ && $2+0>20e9{print $2,$1}' | sort -rn | head -1 | awk '{print $2}')
docker run -d --rm --name <job> --gpus all \
--ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --shm-size=128g \
--cap-add=SYS_PTRACE --env-file .env \
-e HF_HOME="$D/hf" -e HF_DATASETS_CACHE="$D/hf/datasets" \
-e TMPDIR="$D/tmp" -e HALO_DATA_ROOT="$D" \
-v $(pwd):/workspace -v "$D:$D" -v ~/.aws:/root/.aws -w /workspace \
halo:blackwell \
bash -lc "torchrun --nproc_per_node=8 scripts/training/sft.py <config> > \"$D/<job>.log\" 2>&1"Load-bearing flags:
--env-file .envsuppliesWANDB_API_KEY/HF_TOKEN/AWS_DEFAULT_REGION. Nothing auto-loads.env; start from the template (cp .env.example .env), which the GPUmaketargets and the dev container require.HF_HOME/HF_DATASETS_CACHE/TMPDIR/HALO_DATA_ROOTmove the caches, temp files and the toolkit scratch root (S3 dataset cache unders3_datasets/, profiler artifacts underprofiling/) off the small root FS. None is a code default: unset, they fall back to~/.cache/huggingface, the system/tmpand~/.cache/halo. KeepTMPDIRa short path: a multiprocessing socket under it caps at 108 bytes (Troubleshooting).--ipc=host,--shm-size=128gand the twoulimits are required by NCCL and the DataLoader.--cap-add=SYS_PTRACElets py-spy attach for hang triage (halo run py-spy-diag).
HALO_SCRATCH is the one knob for that volume: the Makefile derives the mount and the four variables
from it, the dev container requires it, and docker-compose.vllm.yml mounts it. Unset, the Makefile
resolves it with the recipe above and stops a target that needs it when no mount qualifies; export it
on a host with more than one large volume (the recipe picks by free space) and on a CI runner whose
seeded HF cache must stay put. The compose file has no fallback
(Rollout server images). Confirm the volume with findmnt / df -h before
any multi-GB write.
On a host whose docker default runtime rejects --gpus (sysbox-runc, for one), set
DOCKER_RUNTIME=nvidia on any make target.
A CPU-only job (CPU tests, a one-off script) drops --gpus all, the ulimits and the shm size, as the
Makefile's DOCKER_RUN_CPU does. Keep both mounts: without -v "$D:$D" the log dies with the --rm
container, and the CPU tests that call from_pretrained directly fail without an HF cache once the Hub
is unreachable.
Bash
docker run -d --rm --name <job> \
-e PYTHONPATH=/workspace -e HF_HOME="$D/hf" \
-v $(pwd):/workspace -v "$D:$D" -w /workspace \
halo:blackwell \
bash -lc "pytest -m cpu tests/cpu > \"$D/<job>.log\" 2>&1"Image matrix
TARGET_GPU writes the build's CUDA arch list to /etc/cuda_arch: 9.0 for Hopper, 10.0+PTX for
Blackwell (one image serves SM100 and SM103). Each source build exports it as TORCH_CUDA_ARCH_LIST;
it is not an image ENV, and only the login shell re-exports it (.bashrc), so a bare
docker run ... python sees it unset. DeepEP overrides it (DeepEP explains why that list
does not bound the kernels the toolkit runs). transformer_engine is uninstalled.
On Blackwell FA4 is the auto-selected attention and FA2 stays available (Flash Attention). Either training image is ~30–32 GB; a cold build takes tens of minutes, longest on Hopper, which builds FA2 and FA3 from source.
Building
The build needs no GPU (it cross-compiles via TORCH_CUDA_ARCH_LIST) and no credentials.
Bash
make build-hopper
make build-blackwell
make build-vllm # halo:vllm-0.32.0
make build-sglang # halo:sglang-0.5.22
# Raw equivalent; pass the stamps `make` adds yourself
docker build --build-arg SOURCE_REVISION=$(git rev-parse --short HEAD) \
--build-arg BUILD_DATE=$(date -u +%Y-%m-%d) --build-arg TARGET_GPU=blackwell -t halo:blackwell .A bare docker build targets Hopper (ARG TARGET_GPU=hopper). The server tags carry the engine
release read off Dockerfile.vllm / Dockerfile.sglang (ARG VLLM_VERSION, ARG SGLANG_VERSION).
- uv, not Poetry. Dependencies install into the system interpreter:
uv export --locked(the dev group plus thegigatokenandflash-optimizersextras) into a requirements file, thenuv pip install --system --no-deps -rit.--no-depsis load-bearing: without it the resolver reinstalls torch, Flash Attention and DeepEP over the source builds. The source-built pre-installs (causal-conv1d,flash-linear-attention) take it too, so their dependencies come from the lock.make check-lockreplays this install in a throwaway container and fails on a stale lock. SOURCE_REVISIONbusts the source-COPY cache:make build-*stamps the short git SHA, so a code-only change reruns only the COPY and the editable reinstall. All four images carry it asorg.opencontainers.image.revision, withVERSIONandBUILD_DATEas.versionand.created(a raw build without them labelsdevand an empty date), sodocker inspect --format '{{index .Config.Labels "org.opencontainers.image.revision"}}' halo:blackwellnames the commit an image carries.- Every upstream clone is pinned to a commit or tag: DeepEP
af9a040; Hopper's FA2v2.8.3.post1and FA3 at themaincommitc46b8144(no FA release supports CUDA 13);aws-ofi-ncclat the commit the EFA script pins, configured--disable-tests(its tests needmpi.h);gdrcopyatGDRCOPY_COMMIT. Blackwell's FA2 comes fromBASE_IMAGE(defaultnvcr.io/nvidia/pytorch:26.03-py3). - Hopper FA2 split-K stub. CUDA 13.2 ptxas hangs on the 24 sm_90 split-K kernels, so the build
swaps them for throw-stubs (
docker/training/flash_attn_split_stubs_hopper.cpp) and pins the dense and varlen forwards to one split (num_splits0 → 1 inflash_api.cpp, asserted in the build): their occupancy heuristic would otherwise pick split-K for a small unpadded batch and for decode under GQA. The backward never uses split-K; onlyflash_attn_with_kvcache, which nothing in Halo calls, still can reach a stub. - No
flash_attn.cuteon Hopper. FA2's source build also installs an old copy of FA4's CuTe DSL sources, which does not import against the pinned cutlass-dsl; the Hopper build removes it, so every FA4 probe reads FA4 as not installed there. - Known
pip checkgaps. The cutlass-dsl metapackage requires its CUDA 12 runtime flavor, which neither image installs (both run CUDA 13), and neither carries thenvidia-cuda-nvdisasmwheel it declares: the pinned set is installed without its dependencies, and cutlass-dsl runsnvdisasmonly to keep SASS (CUTE_DSL_KEEP=sass). - Build-time checks.
docker/training/verify_image.pyimports the stack, the target GPU's attention kernels, the cutlass-dsl the locked install re-pins and the extras; the build then runshalo --helpand checks the bubblewrap jail's subordinate id maps. DeepEP needslibcuda.so.1, so it is checked at runtime (Verifying). - License. The images are labeled
LicenseRef-Haloand carryLICENSE+APACHE-2.0.txt.
Rebuild after a pyproject.toml / uv.lock change (run uv lock first) or to move Flash Attention,
DeepEP or the base image.
Baked-in environment variables
FLASH_ATTENTION_CUTE_DSL_CACHE_DIR and TRITON_CACHE_DIR are not baked: the toolkit points each at
$HF_HOME/<subdir> (else the temp dir) when unset, so one mounted volume keeps every kernel cache
across --rm containers. An unwritable location falls back with a warning; set the variables to keep
the caches elsewhere. The Triton cache matters: fla's autotuners persist measured configs there
(FLA_CACHE_RESULTS defaults on), and some keys are shape-derived, so an ephemeral cache re-benchmarks
per sequence length every run. Fresh caches can pick differently, and a different pick changes bf16 rounding,
which routing and index selection can amplify: across separate cold caches tiny GLM-5 Next's logits moved up to
12.4%, while Qwen3.5-35B-A3B, Qwen3.6-35B-A3B and GLM-5.3-Flash cut to six layers stayed bit-identical. On one cold
cache shared by two runs, tiny Qwen3-Next's gradients moved up to 6.1e-5, Qwen3.5 MoE's 2.4e-7 and
Qwen3.8-Flash-Next's 6e-8, and the 13 other tiny families stayed bit-identical. Runs that must match bit for bit share
an already warm cache, and no other run fills it meanwhile.
The FA4 cache serializes each kernel file's writer with flock. Where flock fails (Lustre without
flock), the first use of each kernel raises after a 15 s lock timeout, mid-run included; where locks
are node-local (NFS nolock or local_lock=flock, Lustre localflock), one node can load a file
another is still writing. On either mount, set FLASH_ATTENTION_CUTE_DSL_CACHE_DIR to node-local
storage.
The runtime NCCL is the pip wheel nvidia-nccl-cu13 at the exact uv.lock pin; the NGC
NCCL_VERSION env is the base image's label and does not reflect it. All three Dockerfiles read the
pin through docker/nccl_pin.py, which also owns the DeepEP V2 floor, and its --verify fails the
build when a later step re-resolved the wheel or when the libnccl.so.2 a process resolves is not
that wheel's (the base's older system copy would otherwise reach the weight-sync communicator).
The runtime cuBLAS is the pip wheel nvidia-cublas, pinned by an override-dependencies entry in
pyproject.toml because torch's cuda-toolkit[cublas] extra locks 13.1.0.3, which drops the trailing
rows of some bf16 matmuls (pytorch/pytorch#203481).
13.8.0.4 passes the reproducer cases; 13.7.0.27 was reported to still fail. The wheel's nvidia-cuda-nvrtc
dependency is unpinned and stays at torch's lock. docker/training/verify_image.py fails the build when
the library torch loads is not that wheel. The pin reaches the two training images only: the vLLM and SGLang
rollout images keep the cuBLAS of their own bases and neither pin nor check it.
RDMA networking (InfiniBand and EFA)
Every image is EFA-ready as built. docker/efa/install_efa_userspace.sh installs the EFA installer
1.46.0's rdma-core (60) and AWS libfabric 2.3.1amzn4.0 (/opt/amazon/efa), replacing the NGC base's
MOFED rdma-core in the training images and the distro rdma-core of the server bases, and builds
aws-ofi-nccl at one pinned commit against the uv.lock NCCL. The plugin is GIN-capable and exposed
as libnccl-gin.so; GDRCopy's libgdrapi ships beside it in the training images. Both ends of a
weight-sync group must run this one build, rdma-core included
(Rollout Servers → Servers on other nodes).
InfiniBand and RoCE run on the baked defaults; EFA is selected per
job, so there is no separate -efa tag. The launch env per fabric:
Multi-Node → RDMA fabrics.
Rollout server images
halo:vllm-0.32.0 is upstream vllm/vllm-openai:v0.32.0 and halo:sglang-0.5.22 upstream
lmsysorg/sglang:v0.5.22, each with the uv.lock NCCL, the EFA userspace and the weight-sync patches
asserted at build. Each pins its own torch and transformers, so the trainer never imports either
engine. Their compose files require one host path each, with no fallback: HALO_SCRATCH for
docker-compose.vllm.yml (even to start the server alone) and SGLANG_MODEL_DIR for
docker-compose.sglang.yml; set them in the shell or the repo-root .env. Build gates, patches,
serving flags and the families each serves: Rollout Servers.
Registry
Prebuilt images live on Amazon ECR Public:
Bash
docker pull public.ecr.aws/whitecircle/halo:blackwell # or :hopper
docker pull public.ecr.aws/whitecircle/halo:vllm-0.32.0
docker pull public.ecr.aws/whitecircle/halo:sglang-0.5.22Each moving tag has an immutable release pin: blackwell-1.3.0rc1, hopper-1.3.0rc1, vllm-0.32.0-1.3.0rc1,
sglang-0.5.22-1.3.0rc1. There is deliberately no latest, which would let a Hopper host pull a
Blackwell image. Pair a trainer with the server images of the same release: the vLLM image unpickles
the trainer's ncclUniqueId through a shim at the trainer's module path. Pin a release by pulling its
tag and retagging it under the local name the make targets use:
Bash
docker pull public.ecr.aws/whitecircle/halo:hopper-1.3.0rc1
docker tag public.ecr.aws/whitecircle/halo:hopper-1.3.0rc1 halo:hopperThe compose files take the server images from VLLM_IMAGE / SGLANG_IMAGE (default: the local
halo:vllm-0.32.0 / halo:sglang-0.5.22) and build nothing, so a pulled server image needs no
retag: export VLLM_IMAGE=public.ecr.aws/whitecircle/halo:vllm-0.32.0.
Maintainers publish with make push-public-all (ecr-public-login plus the four push-public-*
targets). To host the images elsewhere, docker tag and docker push the local builds; nothing
sensitive rides along. tests/cpu/config/test_release_version_in_sync.py holds every statement of the
release to pyproject.toml's version: the uv.lock entry, the Makefile VERSION, each Dockerfile's
ARG VERSION, CITATION.cff, the README's newest release entry and every documented image pin.
Verifying and debugging
Bash
docker run --gpus all --rm halo:hopper bash -c "
python -c \"import torch; print(torch.__version__, torch.cuda.is_available())\" && \
python -c \"import transformers, trl, accelerate; print(transformers.__version__, trl.__version__)\" && \
python -c \"import flash_attn; print('FA2', flash_attn.__version__)\" && \
python -c \"from flash_attn_interface import flash_attn_func; print('FA3 OK')\" && \
python -c \"import deep_ep; print('DeepEP OK')\"
"
docker exec -it <container_id> bash # shell into a running container
docker logs <container_id> # view job logs
docker system prune -a # free disk on "no space left"The FA3 line is Hopper-only. DeepEP imports only with --gpus: a runtime DeepEP import error usually
means the container ran without --gpus all. Containers carry no AWS credentials; mount
-v ~/.aws:/root/.aws for S3 and re-run aws sso login on the host after an SSOTokenLoadError
(AWS Auth).
Claude Code installs only with --build-arg INSTALL_CLAUDE_CODE=1, at
/root/.local/bin/claude (best-effort, non-fatal). The skills (skills/, symlinked from
.claude/skills and .agents/skills) are copied into /workspace at build; a checkout mounted at
/workspace replaces them with its own.
Cloud and dev setup
- RunPod — override the CMD with
sleep infinity: the default CMD prints a banner and exits, which stops a pod (RunPod). - SkyPilot / Nomad — SkyPilot, Nomad.
- VS Code, dev container, host
.venv— Development Environment.
Source on GitHubUpdated 18 November 2026