Model integration cost

Halo adds distributed behavior as layers and wrappers around the HuggingFace model, not as a fork of it, so a new family costs little code. For what is already supported, see Supported models.

Plain FSDP2 needs no code. Any AutoModelForCausalLM trains as is, multimodal checkpoints included (loaded through the full vision-and-text class). Code comes in only for parallelism:

Text

HuggingFace model load
  -> thin EP layer or CP wrapper (subclass a base class)
  -> registers by naming the HF class it claims
  -> existing FSDP / EP / ETP / checkpoint paths

What an EP layer costs

Each MoE family adds one file under halo/distributed/expert_parallel/families/ naming the HF MoE class it claims, one frozen MoEFamilySpec holding its config model_type spellings and capabilities (halo/models/moe/family_specs.py), and one import line in the family roster.

  • EP layers subclass a shared base. A family whose expert-weight layout matches an existing one subclasses that instead: Laguna reuses GLM-4's in 21 lines, Qwen3-Next reuses Qwen3.5-MoE's in 25, and Qwen3.8-Flash-Next reuses Qwen3-Next's in 11.
  • Every other EP layer file is under 90 lines.
  • GPT-OSS is the outlier at 168, because of its interleaved, biased gate/up weights and its own clamped SwiGLU compute.

Context parallelism works the same way: one UlyssesAttention subclass in halo/distributed/context_parallel/families.py that declares the attention classes it wraps. Both registries are built from the subclass tree, so a class that exists gets used.

Pipeline parallelism splits a family only once its backbone class is registered in halo/distributed/pipeline_parallel/families.py; any other is refused by name before a weight loads.

Tensor parallelism on a dense model uses HuggingFace's own base_model_tp_plan. On a MoE model it shards attention only, and the one list to edit is TP_SHARDABLE_ATTENTION_CLASSES.

Vendoring

A model that transformers doesn't ship yet can be vendored under halo/models/<name>/: its configuration_*.py and modeling_*.py, registered on import, with a CPU test that proves the registration. Remove the copy once upstream ships the model.

No model is vendored. The only model classes Halo adds are the sequence-classification heads transformers doesn't ship, for Gemma 4 and Qwen3.5/3.6 MoE (halo/models/upstream/seq_cls_heads.py).

The step-by-step procedure for each mode is in Adding a Model ↗.

Source on GitHubUpdated 18 November 2026