Installation

Halo runs inside a Docker image that holds the whole Python stack: PyTorch 2.11 (CUDA 13), Transformers, TRL, Flash Attention and DeepEP. Don't pip install Halo on the host; the compiled kernels exist only in the image.

The host needs an NVIDIA driver R580 or newer (Docker ↗), Docker with the NVIDIA Container Toolkit, and git. The images are x86_64 only, so Grace-based GB200/GB300 hosts (aarch64) need an arm64 build, which the Dockerfile does not provide.

1. Clone the repo

The configs, scripts and make targets live in the repo:

Bash

git clone https://github.com/whitecircle/halo
cd halo

2. Get the image

Pick the image that matches your GPUs:

GPUImageAttention
B200 / B300halo:blackwellFlash Attention 4 + 2
H100 / H200halo:hopperFlash Attention 3 + 2
A100, RTX 3090 / 4090 (single GPU, LoRA/QLoRA)halo:blackwellFlash Attention 2 / SDPA; no DeepEP; not validated

Pull a prebuilt image from Amazon ECR Public (no login or AWS account needed) and give it the local name:

Bash

# Blackwell
docker pull public.ecr.aws/whitecircle/halo:blackwell
docker tag public.ecr.aws/whitecircle/halo:blackwell halo:blackwell

# Hopper
docker pull public.ecr.aws/whitecircle/halo:hopper
docker tag public.ecr.aws/whitecircle/halo:hopper halo:hopper

Don't skip the retag: this guide, the make targets and the compose files use the local name. They default to halo:blackwell. On Hopper, pass IMAGE=halo:hopper to make and set TRAIN_IMAGE=halo:hopper for compose.

  • :blackwell and :hopper track the latest release. :blackwell-1.3.0rc1 and :hopper-1.3.0rc1 pin it.
  • There is no latest tag, so a Hopper host can't pull a Blackwell image by mistake.
  • The RL rollout servers are in the same repository, as :vllm-0.32.0 and :sglang-0.5.22, pinned as :vllm-0.32.0-1.3.0rc1 and :sglang-0.5.22-1.3.0rc1.

To build from source instead (no token or registry login):

Bash

make build-blackwell     # or: make build-hopper

The first build is slow: DeepEP, DeepGEMM (Blackwell) and Flash Attention (Hopper) compile from source.

3. Create a .env file

Put your secrets in .env at the repo root. .env.example is a template:

Bash

HF_TOKEN=hf_...            # gated models and datasets on the HuggingFace Hub
WANDB_API_KEY=...          # Weights & Biases logging
AWS_ACCESS_KEY_ID=...      # only for s3:// datasets
AWS_SECRET_ACCESS_KEY=...
AWS_DEFAULT_REGION=...

Leave out any key you don't need. A run on a public model with no tracking works with an empty file.

Halo never loads .env itself. A docker run you type needs --env-file .env. make train and the training service in docker-compose.vllm.yml pass it for you.

4. Point caches at a large disk

Model weights, dataset caches and checkpoints add up to hundreds of gigabytes, and the root filesystem is usually small. Find your large volume and check its size. A path named /mnt can still sit on the root disk.

Bash

df -h                             # find the volume with real space
findmnt -T /mnt -no TARGET,AVAIL  # the filesystem that actually holds /mnt

The container takes that volume through four variables: HF_HOME (model cache), HF_DATASETS_CACHE (dataset cache), TMPDIR (temp files) and HALO_DATA_ROOT (Halo's scratch: S3 dataset cache, profiler output).

5. Start a container

From the repo root:

Bash

D=/path/to/large-volume   # the volume you checked in step 4
docker run --rm -it --gpus all \
    --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --shm-size=128g \
    --cap-add=SYS_PTRACE --env-file .env \
    -e HF_HOME="$D/hf" -e HF_DATASETS_CACHE="$D/hf/datasets" \
    -e TMPDIR="$D/tmp" -e HALO_DATA_ROOT="$D" \
    -v $(pwd):/workspace -v "$D:$D" -v ~/.aws:/root/.aws -w /workspace \
    halo:blackwell bash
  • --ipc=host, --shm-size=128g and the two ulimits are required. NCCL and the dataloaders fail without them.
  • --cap-add=SYS_PTRACE lets py-spy attach to a hung run. Without it, that works only on hosts where kernel.yama.ptrace_scope is 0.
  • For an RL run, add --network host so the trainer can reach a rollout server that compose started on the same host.
  • Drop -v ~/.aws:/root/.aws unless you read S3 through your AWS profile.

make train runs the same command for you, with --network host and without --cap-add=SYS_PTRACE. It mounts the volume named by HALO_SCRATCH, or, when that is unset, the local mount other than / with the most free space:

Bash

make train CONFIG=examples/sft/qwen3/qwen3-4b-ultrachat.yaml NPROC=8

If your Docker default runtime rejects --gpus (for example sysbox-runc), add DOCKER_RUNTIME=nvidia to any make target.

6. Verify

Inside the container:

Bash

nvidia-smi                                                   # GPUs visible
python -c "import torch; print(torch.cuda.is_available())"   # True
halo launch --list                                           # CLI works, methods indexed

If a check fails, the usual causes are a missing --gpus all, the wrong image for your GPU architecture, or a missing NVIDIA Container Toolkit. See Troubleshooting.

Next steps

Quickstart launches your first run, and Writing a config shows what to change in it. Image internals, the multi-container RL setup and registry publishing are in Docker ↗. A build outside the image is unsupported; DeepEP ↗ and Flash Attention ↗ carry the pins it would need.

Source on GitHubUpdated 18 November 2026