Supported models

Any HuggingFace AutoModelForCausalLM trains on Halo's data-parallel path with no extra work: point model_name_or_path at it and launch.

The families below also have expert, context, tensor or pipeline-parallel support (which, per family), a tuned recipe and GPU-validated coverage, except where a row says otherwise. Eighteen of them are MoE.

FamilyHub exampleKindWhat to know
Qwen3 MoEQwen/Qwen3-30B-A3B-Instruct-2507MoEthe reference MoE
Qwen3.5 / Qwen3.6 MoEQwen/Qwen3.5-35B-A3BMoEhybrid linear attention; Qwen/Qwen3.8-2.4T-A95B ↗ takes the same path, with no recipe and no GPU run
GPT-OSSunsloth/gpt-oss-20b-BF16MoEattention sinks; FA2 refused for on-policy RL
GLM-4 MoE Litezai-org/GLM-4.7-FlashMoEcompressed (MLA) attention
Gemma 4 MoEgoogle/gemma-4-26B-A4B-itMoEno flash kernel (SDPA, FlexAttention on sliding layers); no router balancing
Mistral 4 MoEmistralai/Mistral-Small-4-119B-2603MoEconvert the fp8 release to bf16 first
Laguna 2.1poolside/Laguna-S-2.1MoESDPA through trust_remote_code: true, and a flash pin is refused
LFM-2 MoELiquidAI/LFM2-24B-A2BMoEhybrid short-convolution layers
ZAYA1Zyphra/ZAYA1-8BMoEtrains without gradient checkpointing, always
Command A+CohereLabs/command-a-plus-05-2026-bf16MoE200B+; pinned revision
Nemotron 3.5 Super VLnvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16MoEMamba-2 layers; EP8 on B300; short runs on the release; no online RL; the text-only nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 ↗ has its own recipe, run for 5 steps at EP8
Bailing / Ling ↗inclusionAI/Ling-mini-2.0MoEneeds trust_remote_code
Inkling ↗thinkingmachines/Inkling-SmallMoEmultimodal; SDPA; the tokenizer ships no pad token
DeepSeek-V4 ↗deepseek-ai/DeepSeek-V4-FlashMoEeager attention only; one document per row (packing and padding_free refused); convert to bf16 first
GLM-5 Next ↗zai-org/GLM-5.3-FlashMoEconvert fp8 to bf16 first; SDPA only; LoRA fits one 8-GPU machine, a full fine-tune takes two
Step-3.7 Flash ↗stepfun-ai/Step-3.7-FlashMoEcomposite VLM; SDPA only
Qwen3-Next ↗Qwen/Qwen3-Next-80B-A3B-InstructMoEhybrid linear attention; tiny-model GPU tests only
Qwen3.8-Flash-Next ↗Qwen/Qwen3.8-Flash-NextMoEexperimental: text tower under EP, ETP and PP; 95 GiB n-gram table in host memory; online and async GRPO on vLLM, and on SGLang from the toolkit's image; offline GRPO CPU-checked only
Qwen3.8-27B ↗Qwen/Qwen3.8-27Bdensethe Qwen3.5 dense path; no recipe and no GPU run
Qwen3 dense ↗Qwen/Qwen3-4B-Instruct-2507densethe reference dense family
Any other HF causal LMLlama, Mistral, Phi, …denseFSDP2 by default; TP needs a tp_plan

The last column names each family's main constraint. The modes each family supports are in the Supported matrix, which wins when a recipe disagrees.

Cookbooks

Eleven families have a cookbook: a worked path from docker pull to a served checkpoint, with the exact halo launch lines and the LoRA and RL variants the family supports. If your family has one, start there.

Multimodal checkpoints

Gemma 4, Mistral 4, Command A+, Inkling, GLM-5 Next, Step-3.7 Flash, Nemotron 3.5 Super VL, Qwen3.5/3.6, Qwen3.8-Flash-Next and Qwen3-VL are natively multimodal. Halo loads the full vision-and-text model and picks the data path from your dataset, not the checkpoint:

  • Rows with images train through the VLM pipeline, with standard padding. Images can't be packed.
  • A text-only dataset on the same checkpoint trains through the text pipeline, with packing available.

To load a multimodal checkpoint as text-only, set text_only_model: true, as the Qwen3.8-Flash-Next and Nemotron 3.5 Super VL recipes do. Qwen3.5/3.6, Qwen3.8-Flash-Next, Nemotron 3.5 Super VL and Llama 4 have a text-only variant; the others refuse it.

Image data works for SFT, DPO, KTO, SMPO and both distillation trainers. PaliGemma takes it through SFT and distillation only, and Nemotron-H refuses it on DPO and KTO. Reward modeling takes images only on checkpoints with a multimodal sequence-classification head, such as Gemma 4 and Qwen3.5/3.6. Classification, the GRPO trainers and embedding are text-only.

Adding a family

A family that trains under plain FSDP2 needs no code. Context parallelism takes one wrapper class that registers itself; expert parallelism takes one EP layer file, the family's spec and a roster import. Model integration cost shows what an EP layer costs, and Adding a Model ↗ is the procedure.

Source on GitHubUpdated 18 November 2026