Supported Matrix
What runs, what does not, and what has been validated. The code is the source of truth; where a recipe disagrees with this page, this page wins.
Runtime
Pull the prebuilt images or build them from source: Installation.
Attention backends
Leave attn_implementation unset unless a recipe pins it. Halo picks FA4 on
Blackwell, FA3 on Hopper and FA2 as the fallback, except:
Padded-batch scripts use a different default on GPUs other than Blackwell (Writing a config). CP picks its own kernel and ignores the configured label: FA3 on Hopper, FA4 on Blackwell (FA2 for the families that fall back from FA4), FA2 otherwise. It rejects an explicit SDPA label, except for Bailing/Ling.
packing is refused where a packed row's documents would mix: GPT-OSS and Ling
2.0 / Ring off flash, and on every backend Ling 3.0, DeepSeek-V4 and flat rows of
GLM-5 Next (tree packing keeps GLM-5 Next's documents apart). Ring and Inkling mix
documents on every kernel, so packing them takes
allow_cross_document_linear_attention: true. padding_free needs a flash
kernel, and Qwen3.5/3.6 refuse it
(Document isolation ↗).
Per-family details:
Flash Attention ↗.
Training methods
No trainer refuses EP, TP or ETP. SMPO, KTO, teacher distillation and embedding are verified under EP and TP only, and self-distillation under EP only (Trainer Compatibility ↗). CP and PP are enabled per trainer; CP on any other loss would silently mis-pool across sequence shards. Under PP every trainer's checkpoint save, resume and export are verified.
Rollout engines
Online GRPO and online SDPG generate on vLLM only. Async GRPO with environments
uses vLLM by default; rollout_backend: sglang switches engines. The engines
differ in which families they can take an online weight update for. The scripts
refuse an unsupported pair at startup, naming the family and the reason.
The last column is what passed on this release: async GRPO, full fine-tuning on
a real checkpoint, on one 8× B300 node with a TP=1 server, unless it says
otherwise. ep1 is expert_parallel_size: 1 on the EP layers, tp2 is
tensor_parallel_size: 2, and replay is routing_replay: rollout.
A MoE with no family in this table, such as full GLM-4.5 or GLM-4.7 (glm4_moe), has
no validated expert layout on either engine and is refused too. Offline GRPO needs no
server, so these refusals do not apply to it. rollout_max_thinking_tokens and
carry_reasoning are vLLM-only. SGLang must be served from this repo's image. Setup,
ports, weight sync and routing_replay: Rollout Servers.
A push of unchanged weights leaves the served log-probs bit-identical on every
family that syncs, on each engine that takes it (MoE at ep1 and ep2). Trainer
and engine still disagree on identical weights. bf16 routing flips carry part of that
gap, and how much varies by family. On vLLM at ep1, before any update, one run per
arm, routing_replay: rollout lowers the mean gap about 2× on gpt-oss-20b, 2.5× on
Qwen3.6-35B-A3B and 1.2× on GLM-4.7-Flash, and recompute stays level with no replay.
Other families are not measured. Replay is experimental and does not make training exactly
on-policy:
numbers ↗.
A dense checkpoint under a spelling a MoE family claims is refused with that
family: LFM2-VL over a dense LFM-2 tower, Mistral 3 over a dense Mistral tower,
Gemma 4 with enable_moe_block: false, Command A Vision over dense Cohere2.
Ran means the code admits it and a row passed against a live server on this release, on at least one family; Yes means the code admits it. The Offline GRPO column states what the code admits. A family's own limits from Model families still apply.
Parallelism modes
Layout rules and the shapes rejected before a run starts: Parallelism.
Model families
Any HuggingFace AutoModelForCausalLM runs under FSDP2; this table covers
advanced parallelism. What each family is for:
Supported Models.
EP+CP runs where EP and CP both do, EP+TP where EP and TP both do. Online RL per family: Rollout engines. Why a family drops a mode: the model pages ↗.
PEFT, quantization and data
What each mode writes, and which resumes are exact: Checkpoints.
Validation
The GPU tests run on 8× B300 machines. Two 8-GPU machines ran node-local ep8
with DP through a per-node save, resume and evaluation, cross-node ep16 through
a gathered save and reload, cross-node weight sync on both engines, and the two
pipeline-parallel recipes, the Qwen3-8B one through a per-node save, resume and
merge. At four GPUs a machine, two machines also ran an MTP head under node-local EP
and plain FSDP2, and gpt-oss-20b at ep2 with ETP4. The other multi-node shapes run
as a two-node simulation on one host.
Checkpoints written by Halo 1.1.0 resume and export; a sharded EP save needs
merge-ep-shards first.
The full scope: Release validation ↗
and Verified topologies ↗.
Limits
- No hosted training UI and no hyperparameter search.
- No tested end-to-end Hub upload. Upload the final checkpoint yourself (Checkpoints).
- Advanced parallelism support is per family, not automatic.
- vLLM and SGLang run in their own containers, outside the training environment.
Source on GitHubUpdated 18 November 2026