Supported models
Any HuggingFace AutoModelForCausalLM trains on Halo's data-parallel path with
no extra work: point model_name_or_path at it and launch.
The families below also have expert, context, tensor or pipeline-parallel support (which, per family), a tuned recipe and GPU-validated coverage, except where a row says otherwise. Eighteen of them are MoE.
The last column names each family's main constraint. The modes each family supports are in the Supported matrix, which wins when a recipe disagrees.
Cookbooks
Eleven families have a cookbook: a worked path from
docker pull to a served checkpoint, with the exact halo launch lines and the
LoRA and RL variants the family supports. If your family has one, start there.
Multimodal checkpoints
Gemma 4, Mistral 4, Command A+, Inkling, GLM-5 Next, Step-3.7 Flash, Nemotron 3.5 Super VL, Qwen3.5/3.6, Qwen3.8-Flash-Next and Qwen3-VL are natively multimodal. Halo loads the full vision-and-text model and picks the data path from your dataset, not the checkpoint:
- Rows with images train through the VLM pipeline, with standard padding. Images can't be packed.
- A text-only dataset on the same checkpoint trains through the text pipeline, with packing available.
To load a multimodal checkpoint as text-only, set text_only_model: true, as the
Qwen3.8-Flash-Next and Nemotron 3.5 Super VL recipes do. Qwen3.5/3.6,
Qwen3.8-Flash-Next, Nemotron 3.5 Super VL and Llama 4 have a text-only variant;
the others refuse it.
Image data works for SFT, DPO, KTO, SMPO and both distillation trainers. PaliGemma takes it through SFT and distillation only, and Nemotron-H refuses it on DPO and KTO. Reward modeling takes images only on checkpoints with a multimodal sequence-classification head, such as Gemma 4 and Qwen3.5/3.6. Classification, the GRPO trainers and embedding are text-only.
Adding a family
A family that trains under plain FSDP2 needs no code. Context parallelism takes one wrapper class that registers itself; expert parallelism takes one EP layer file, the family's spec and a roster import. Model integration cost shows what an EP layer costs, and Adding a Model ↗ is the procedure.
Source on GitHubUpdated 18 November 2026