Model Cookbooks
Each cookbook takes one MoE family from an empty host to a trained, served checkpoint. The Supported Matrix is the authority: where a cookbook disagrees, the matrix wins.
Every cookbook has the same sections: Support, Recipes, Settings that matter, Limits, Export and serve,
and Reference. The GPU counts are the ones the recipes are written for, B300s unless a page says otherwise. On
H100 or H200, use the Hopper image. Those GPUs have far less memory than a B300, so a recipe may need
more GPUs or a shorter max_length there.
What ran on 1.2.0
Short runs on the full checkpoints covered these SFT recipes: Gemma 4, GLM-4.7-Flash, Ling 2.0, both ZAYA1 recipes, Qwen3.5-35B-A3B at 33,000 tokens, and on two 8-GPU machines gpt-oss-20b and the two pipeline recipes. They also covered gpt-oss-20b's classification, self-distillation, embedding, offline GRPO, DPO, SMPO and reward recipes and Gemma 4's DPO, SMPO and reward recipes. All four Nemotron 3.5 Super VL recipes ran on the text tower, and its image recipe on the composite. The Nemotron 3.5 Lightning recipe ran for 20 steps. Both Qwen3.6-35B-A3B MTP recipes ran, the joint one for 5 steps.
The Laguna S, LFM2.5-8B, Mistral Small 4, DeepSeek-V4 LoRA and Command A+ recipes have not run at their current shape on the full-size model. Their memory fit is computed (Command A+'s comes from an earlier shape) and their speed unmeasured, so try a short run first.
Start the training container
Every training, inference and conversion command runs inside the training container: start it as in
Installation → Start a container, with two changes. Mount your
large volume at /data (-v "$HALO_SCRATCH":/data, with HF_HOME=/data/hf,
HF_DATASETS_CACHE=/data/hf/datasets, TMPDIR=/data/tmp and HALO_DATA_ROOT=/data), and add
--network host so the trainer reaches the rollout servers. /mnt is not always large, so check with
df -h first. Load-bearing flags: Docker ↗.
Inside the container, /data is $HALO_SCRATCH. A run that writes /data/checkpoints/<run> leaves its
output in $HALO_SCRATCH/checkpoints/<run> on the host.
The recipes override config fields on the command line, so the shipped YAML files stay untouched. Write
each override as --field=value, and separate list items with commas
(--lora_target_modules=q_proj,v_proj). Fields such as dataset and rewards can't be set this way:
copy the YAML and edit them there. Edit the existing line rather than adding a second one, because a
repeated key fails to parse.
Every layout in these cookbooks passes Halo's startup checks at the GPU count shown. The rule people hit most often, EP on one eight-GPU node, is in Parallelism.
Smoke-test a checkpoint
A gathered save (save_sharded_ep: false, the default) is a standard Hugging Face checkpoint. Each
cookbook names the model class to load it with. The rest is the same for every family:
Python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
path = "/data/checkpoints/<run>"
tokenizer = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, dtype=torch.bfloat16, device_map="auto")
messages = [{"role": "user", "content": "Explain expert parallelism in three sentences."}]
inputs = tokenizer.apply_chat_template(
messages, add_generation_prompt=True, return_dict=True, return_tensors="pt"
).to(model.device)
output = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))A LoRA run saves an adapter, not a full checkpoint. Load it with
PeftModel.from_pretrained(base, path) or merge it first;
AutoModelForCausalLM.from_pretrained does not load an adapter directory under
PEFT 0.18.1. A run with merge_expert_lora_on_save: true saves the merged checkpoint instead.
Serve from the host
Start rollout and serving containers on the host from the repo root, not inside the training container, and give them GPUs the trainer does not use. Run this setup once in each host shell that starts a server:
Bash
cd halo
export HALO_SCRATCH=/path/to/large/volume # the volume the training container mounts
docker pull public.ecr.aws/whitecircle/halo:sglang-0.5.22
docker pull public.ecr.aws/whitecircle/halo:vllm-0.32.0
export SGLANG_IMAGE=public.ecr.aws/whitecircle/halo:sglang-0.5.22
export VLLM_IMAGE=public.ecr.aws/whitecircle/halo:vllm-0.32.0
export SGLANG_MODEL_DIR="$HALO_SCRATCH" # SGLang mounts it read-only at the same path
export HF_HOME="$HALO_SCRATCH/hf" # both servers share the training container's hub cacheBoth servers mount jinja-templates/ at /jinja-templates, the path a chat template takes. SGLang takes
host paths for checkpoints ($HALO_SCRATCH/checkpoints/<run>). The vLLM service mounts only the Hugging
Face cache for them. Add - ${HALO_SCRATCH}:/data:ro under the vllm-server service's volumes: in
docker-compose.vllm.yml, then give vLLM the same /data paths the trainer uses. vLLM answers a
request for any other model name with a 404, so the served path must match the trainer's model_name,
which defaults to its model_name_or_path.
The compose files already pass the MoE backend that weight sync needs (--moe-backend triton and
--moe-runner-backend triton), so leave it alone. A run with routing_replay: rollout (experimental) also needs
VLLM_ENABLE_R3=1 or SGLANG_ENABLE_R3=1 set where you run compose. Plain serving works on the upstream
SGLang 0.5.22 image, but weight sync needs this repo's server images. Parsers, ports and the other variables
are in Rollout Servers.
Before a GRPO run
GRPO in these cookbooks is async GRPO with environments. Ray actors run episodes against a rollout server, and the trainer pushes new weights into that server after each step.
- Split the GPUs. The server and the trainer can't share a GPU. Fence the trainer with
CUDA_VISIBLE_DEVICESand give the server the rest. - Match the NCCL transport. Both ends must use the same transport, or the weight sync hangs at its
first collective. On a host with no RDMA fabric, add
-e NCCL_IB_DISABLE=1 -e NCCL_NET=Socket -e NCCL_SOCKET_IFNAME=^docker,vethto the training container, which matches the compose defaults. On an EFA host, put both ends on the EFA overlay instead: a trainer left atSocketsends every collective over TCP and breaks DeepEP (Servers on other nodes ↗). - Code-contests recipes. Their
datasetis a placeholder underyour-org/. Build the problem pool first (Building a pool ↗). Theirauditjudge vetoes or prices hacks and tool misuse (Rewards). It calls OpenRouter withOPENROUTER_API_KEYfrom.env, and a judge outage trains nothing. Remove that reward term to train without a judge. - Sandbox. Code environments run programs the policy writes. The default
localsandbox lets them read the container's environment,.envsecrets included. Setsandbox_backendunderenvironment_kwargstoremote, which also needs a server URL insandbox_urlorHALO_SANDBOX_URL, or tobubblewrap, which needs extra container rights (Async GRPO with Environments).
Families without a shipped GRPO recipe start from
examples/grpo/environmental/environmental-grpo-template.yaml. Copy it, set dataset,
environment_type and rewards for your task, and pass the rest as overrides. Each cookbook gives the
exact server and trainer commands.
Source on GitHubUpdated 18 November 2026