Installation
Halo runs inside a Docker image that holds the whole Python stack: PyTorch 2.11
(CUDA 13), Transformers, TRL, Flash Attention and DeepEP. Don't pip install
Halo on the host; the compiled kernels exist only in the image.
The host needs an NVIDIA driver R580 or newer (Docker ↗), Docker with the NVIDIA Container Toolkit, and git. The images are x86_64 only, so Grace-based GB200/GB300 hosts (aarch64) need an arm64 build, which the Dockerfile does not provide.
1. Clone the repo
The configs, scripts and make targets live in the repo:
Bash
git clone https://github.com/whitecircle/halo
cd halo2. Get the image
Pick the image that matches your GPUs:
Pull a prebuilt image from Amazon ECR Public (no login or AWS account needed) and give it the local name:
Bash
# Blackwell
docker pull public.ecr.aws/whitecircle/halo:blackwell
docker tag public.ecr.aws/whitecircle/halo:blackwell halo:blackwell
# Hopper
docker pull public.ecr.aws/whitecircle/halo:hopper
docker tag public.ecr.aws/whitecircle/halo:hopper halo:hopperDon't skip the retag: this guide, the make targets and the compose files use
the local name. They default to halo:blackwell. On Hopper, pass
IMAGE=halo:hopper to make and set TRAIN_IMAGE=halo:hopper for compose.
:blackwelland:hoppertrack the latest release.:blackwell-1.3.0rc1and:hopper-1.3.0rc1pin it.- There is no
latesttag, so a Hopper host can't pull a Blackwell image by mistake. - The RL rollout servers are in the same repository, as
:vllm-0.32.0and:sglang-0.5.22, pinned as:vllm-0.32.0-1.3.0rc1and:sglang-0.5.22-1.3.0rc1.
To build from source instead (no token or registry login):
Bash
make build-blackwell # or: make build-hopperThe first build is slow: DeepEP, DeepGEMM (Blackwell) and Flash Attention (Hopper) compile from source.
3. Create a .env file
Put your secrets in .env at the repo root. .env.example is a template:
Bash
HF_TOKEN=hf_... # gated models and datasets on the HuggingFace Hub
WANDB_API_KEY=... # Weights & Biases logging
AWS_ACCESS_KEY_ID=... # only for s3:// datasets
AWS_SECRET_ACCESS_KEY=...
AWS_DEFAULT_REGION=...Leave out any key you don't need. A run on a public model with no tracking works with an empty file.
Halo never loads .env itself. A docker run you type needs --env-file .env.
make train and the training service in docker-compose.vllm.yml pass it for
you.
4. Point caches at a large disk
Model weights, dataset caches and checkpoints add up to hundreds of gigabytes,
and the root filesystem is usually small. Find your large volume and check its
size. A path named /mnt can still sit on the root disk.
Bash
df -h # find the volume with real space
findmnt -T /mnt -no TARGET,AVAIL # the filesystem that actually holds /mntThe container takes that volume through four variables: HF_HOME (model
cache), HF_DATASETS_CACHE (dataset cache), TMPDIR (temp files) and
HALO_DATA_ROOT (Halo's scratch: S3 dataset cache, profiler output).
5. Start a container
From the repo root:
Bash
D=/path/to/large-volume # the volume you checked in step 4
docker run --rm -it --gpus all \
--ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --shm-size=128g \
--cap-add=SYS_PTRACE --env-file .env \
-e HF_HOME="$D/hf" -e HF_DATASETS_CACHE="$D/hf/datasets" \
-e TMPDIR="$D/tmp" -e HALO_DATA_ROOT="$D" \
-v $(pwd):/workspace -v "$D:$D" -v ~/.aws:/root/.aws -w /workspace \
halo:blackwell bash--ipc=host,--shm-size=128gand the two ulimits are required. NCCL and the dataloaders fail without them.--cap-add=SYS_PTRACElets py-spy attach to a hung run. Without it, that works only on hosts wherekernel.yama.ptrace_scopeis 0.- For an RL run, add
--network hostso the trainer can reach a rollout server that compose started on the same host. - Drop
-v ~/.aws:/root/.awsunless you read S3 through your AWS profile.
make train runs the same command for you, with --network host and without
--cap-add=SYS_PTRACE. It mounts the volume named by HALO_SCRATCH, or, when
that is unset, the local mount other than / with the most free space:
Bash
make train CONFIG=examples/sft/qwen3/qwen3-4b-ultrachat.yaml NPROC=8If your Docker default runtime rejects --gpus (for example sysbox-runc), add
DOCKER_RUNTIME=nvidia to any make target.
6. Verify
Inside the container:
Bash
nvidia-smi # GPUs visible
python -c "import torch; print(torch.cuda.is_available())" # True
halo launch --list # CLI works, methods indexedIf a check fails, the usual causes are a missing --gpus all, the wrong image
for your GPU architecture, or a missing NVIDIA Container Toolkit. See
Troubleshooting.
Next steps
Quickstart launches your first run, and Writing a config shows what to change in it. Image internals, the multi-container RL setup and registry publishing are in Docker ↗. A build outside the image is unsupported; DeepEP ↗ and Flash Attention ↗ carry the pins it would need.
Source on GitHubUpdated 18 November 2026