Supported Matrix

What runs, what does not, and what has been validated. The code is the source of truth; where a recipe disagrees with this page, this page wins.

Runtime

PackageVersion in the image
Python3.12, installed with uv into the system interpreter
PyTorch2.11.0, cu130 wheel (CUDA 13.2 toolkit)
Transformers5.20.2
TRL1.14.2
Accelerate1.11.0
PEFT0.18.1
sentence-transformers6.1.0
Liger Kernel0.8.4
Flash Attention 44.0.0b37 (beta), Blackwell image only
flash-linear-attention0.4.2
vLLM0.32.0 on Model Runner V2, its own container: rollouts for both online RL methods
SGLang0.5.22, its own container: rollouts for async GRPO with environments
HardwareImageStatus
NVIDIA B200 / B300halo:blackwellsupported, primary target
NVIDIA H100 / H200halo:hoppersupported
NVIDIA GB200 / GB300 NVL72noneGrace hosts are aarch64; both images are x86_64-only
NVIDIA A100, RTX 3090 / 4090halo:blackwellsingle-GPU LoRA/QLoRA only (torch, FA2 and bitsandbytes carry sm_80–sm_89 kernels; DeepEP, FA3 and FA4 do not run); not validated
RTX 50-series (SM 12.x)halo:blackwellset attn_implementation: flash_attention_2 or sdpa, since the auto-selected FA4 does not run on SM 12.x; not validated

Pull the prebuilt images or build them from source: Installation.

Attention backends

Leave attn_implementation unset unless a recipe pins it. Halo picks FA4 on Blackwell, FA3 on Hopper and FA2 as the fallback, except:

FamilyBackend
Qwen3.5/3.6, Qwen3-NextFA4 on Blackwell, FA3 on Hopper; on Blackwell, packed SFT rows take FA2 unless you pin FA4, and full_determinism takes SDPA (FA2 on packed rows)
GLM-4 MoE LiteFA2 on Blackwell, FA3 on Hopper
Gemma 2 / Gemma 3FA2 on Blackwell at head_dim 256; Gemma 2 27B takes FA4 at one unpadded row per batch and FA2 otherwise; a softcapped model never runs SDPA
Gemma 4SDPA, with FlexAttention on sliding-window layers; no flash path
PaliGemma / PaliGemma 2SDPA / flex_attention; a pinned flash label falls back with a warning
GLM-5 Next, Step-3.7 Flash, InklingSDPA, chosen automatically
Qwen3.8-Flash-NextSDPA, with Halo's tiled kernel on the sparse layers
DeepSeek-V4eager only
Laguna with trust_remote_code: trueSDPA, and a flash pin is refused
Ling 3.0set attn_implementation: sdpa; the flash label fails the build
LFM2-VLSDPA, chosen automatically

Padded-batch scripts use a different default on GPUs other than Blackwell (Writing a config). CP picks its own kernel and ignores the configured label: FA3 on Hopper, FA4 on Blackwell (FA2 for the families that fall back from FA4), FA2 otherwise. It rejects an explicit SDPA label, except for Bailing/Ling.

packing is refused where a packed row's documents would mix: GPT-OSS and Ling 2.0 / Ring off flash, and on every backend Ling 3.0, DeepSeek-V4 and flat rows of GLM-5 Next (tree packing keeps GLM-5 Next's documents apart). Ring and Inkling mix documents on every kernel, so packing them takes allow_cross_document_linear_attention: true. padding_free needs a flash kernel, and Qwen3.5/3.6 refuse it (Document isolation ↗). Per-family details: Flash Attention ↗.

Training methods

No trainer refuses EP, TP or ETP. SMPO, KTO, teacher distillation and embedding are verified under EP and TP only, and self-distillation under EP only (Trainer Compatibility ↗). CP and PP are enabled per trainer; CP on any other loss would silently mis-pool across sequence shards. Under PP every trainer's checkpoint save, resume and export are verified.

MethodScriptCPPPNotes
SFTscripts/training/sft.pyYesYesalso VLM and continued pretraining
SMPOscripts/training/preference/smpo.pyYesYesreference-free preference
DPO / KTOscripts/training/preference/{dpo,kto}.pyNoYesreference log-prob sums block CP; PP needs precomputed reference log-probs, and KTO loss_type: apo_zero_unpaired
Reward modelingscripts/training/reward/reward_model.pyNoYesfull-sequence pooling blocks CP
Classificationscripts/training/classification.pyNoYesfull-sequence pooling blocks CP
Offline GRPOscripts/training/grpo/offline.pyYesYesCP is full fine-tuning only: PEFT, native expert LoRA and an explicit ref_model are rejected
Online GRPO (RLVR)scripts/training/grpo/online.pyNoNoneeds vLLM; --use_sdpg=true runs online SDPG
Async GRPO with environmentsscripts/training/grpo/environmental.pyNoNoneeds Ray and a vLLM or SGLang server
Distillationscripts/training/distillation/NoNoteacher and self distillation
Embeddingscripts/training/embedding.pyNoNoSentenceTransformer trainer

Rollout engines

Online GRPO and online SDPG generate on vLLM only. Async GRPO with environments uses vLLM by default; rollout_backend: sglang switches engines. The engines differ in which families they can take an online weight update for. The scripts refuse an unsupported pair at startup, naming the family and the reason.

The last column is what passed on this release: async GRPO, full fine-tuning on a real checkpoint, on one 8× B300 node with a TP=1 server, unless it says otherwise. ep1 is expert_parallel_size: 1 on the EP layers, tp2 is tensor_parallel_size: 2, and replay is routing_replay: rollout.

FamilyvLLM 0.32.0SGLang 0.5.22ReplayRan on this release
Dense familiesYesYesNo, MoE onlyvLLM: online GRPO, SDPG and async GRPO on Qwen3-0.6B. SGLang: a weight push on Qwen3-0.6B, no training loop
GPT-OSSYesYesYesunsloth/gpt-oss-20b-BF16. SGLang: every SGLang mode marked Ran below. vLLM: ep4, ep2 with tp2 or ETP2, tp2 and ep2 at DP2, HSDP, online GRPO at ep2
Qwen3 MoEYesYesYesQwen3-30B-A3B. vLLM: online GRPO, SDPG and async GRPO. SGLang: ep1, ep2, each also with replay
Qwen3.5 / Qwen3.6 MoEYesYesYesQwen3.6-35B-A3B. vLLM: ep2, tp2, online GRPO at ep2. SGLang: ep1, ep2. Both: LoRA at ep1, replay at ep2, also on Qwen3.5-35B-A3B
GLM-4 MoE LiteYesYesYesGLM-4.7-Flash, with and without replay. vLLM: ep1, ep2, tp2. SGLang: ep1, ep2
LFM-2 MoEYesYesYesLFM2-24B-A2B, with and without replay. vLLM: ep1, ep2, tp2. SGLang: ep1, ep2
Gemma 4 MoEYesYesNo, on either engineGemma 4 26B-A4B on both engines: ep1, ep2, LoRA at ep1. vLLM: online GRPO at ep2
Ling 2.0YesYesYesLing-mini-2.0. vLLM: ep1, ep2. Both: replay at ep2
LagunaYesNo (SGLang's loader asserts full coverage per call)vLLMLaguna-XS-2.1 on vLLM: ep1, ep2, replay at ep2
Step-3.7 FlashYesNo (SGLang's loader asserts full coverage per call)vLLMvLLM: Step-3.7-Flash on a TP=2 server, a weight push at ep2 and an expert-LoRA row at ep4; a full fine-tune at ep4 does not fit four GPUs
Mistral4, Zaya, DeepSeek-V4, Ling 3.0 and Ring's linear checkpointsNoNoNo
GLM-5 NextYesYesNot runA tiny GLM-5 Next checkpoint on both engines: serving, and an exact weight push at ep1 and ep2
Inkling, Qwen3-Next, Qwen3.8-Flash-Next, Cohere2 MoE, Nemotron-HNoNo (no online RL at all)No

A MoE with no family in this table, such as full GLM-4.5 or GLM-4.7 (glm4_moe), has no validated expert layout on either engine and is refused too. Offline GRPO needs no server, so these refusals do not apply to it. rollout_max_thinking_tokens and carry_reasoning are vLLM-only. SGLang must be served from this repo's image. Setup, ports, weight sync and routing_replay: Rollout Servers.

A push of unchanged weights leaves the served log-probs bit-identical on every family that syncs, on each engine that takes it (MoE at ep1 and ep2). Trainer and engine still disagree on identical weights. bf16 routing flips carry part of that gap, and how much varies by family. On vLLM at ep1, before any update, one run per arm, routing_replay: rollout lowers the mean gap about 2× on gpt-oss-20b, 2.5× on Qwen3.6-35B-A3B and 1.2× on GLM-4.7-Flash, and recompute stays level with no replay. Other families are not measured. Replay is experimental and does not make training exactly on-policy: numbers ↗.

A dense checkpoint under a spelling a MoE family claims is refused with that family: LFM2-VL over a dense LFM-2 tower, Mistral 3 over a dense Mistral tower, Gemma 4 with enable_moe_block: false, Command A Vision over dense Cohere2.

Ran means the code admits it and a row passed against a live server on this release, on at least one family; Yes means the code admits it. The Offline GRPO column states what the code admits. A family's own limits from Model families still apply.

ModeOnline GRPO, SDPG (vLLM)Async GRPO, vLLMAsync GRPO, SGLangOffline GRPO
Plain FSDP2 or one GPU, denseRanRanYesYes
MoE ep1, experts sharded (default)RanRanRanYes
MoE ep1, fsdp_shard_ep1_experts: falseRanRanYesYes
ZeRO-3 with no expert groupRan (dense)Ran (ep1)Ran (ep1)Yes
MoE with use_grouped_gemm: falseNoNoNoYes
EP, one group (expert_parallel_size = trainer GPUs)Ran (ep2)Ran (ep2, ep4)Ran (ep2, ep4)Yes
expert_parallel_size: 2, several groups in one domainRanRanYesYes
expert_parallel_size above 2, group narrower than the one domainNoNoNoNo
Node-local EP groups across domainsRan (two simulated domains)Ran (two simulated domains)YesYes
One EP group across domains (ep_scope: global)YesYesYesYes
TPRanRanRanYes
Pure ETPRan (attention LoRA)Ran (attention LoRA)YesYes
EP+TPRanRanRanYes
EP+ETPRanRanRanYes
HSDPRan (two simulated domains)Ran (two simulated domains)YesYes
CP, EP+CPNoNoNoFull fine-tuning
PP, PP+EP, PP+ETPNoNoNoFull fine-tuning
TP+CP, TP+ETP, ETP+CP, EP+TP+ETPNoNoNoNo
accelerate launch, plain data parallelismDense onlyDense onlyDense onlyYes
PEFT attention LoRA, off TP and EP+TPRanRanRanYes, off CP and PP too
Native expert LoRA, off ETP, TP and EP+TPRanRanRanYes, off CP and PP too
QLoRANoNoNoPlain data parallelism only
FeatureOnline GRPO, SDPG (vLLM)Async GRPO, vLLMAsync GRPO, SGLang
ResumeRanRanRan
Several serversNo, one serverRanRan
PrefetchNoRan (depth 1, 2)Ran (depth 1)
routing_replay: rolloutNoRanRan
routing_replay: recomputeNoRanRan
rollout_max_thinking_tokensNoRan (GPT-OSS, Qwen3.6)No
carry_reasoningNoYesNo
isr_engine_referenceNoYesYes
Server at TP above 1YesRan (tp2)Ran (tp2)
Server on another machineYesRanRan

Parallelism modes

ModeStatusBest for
FSDP2 / DPsupporteddense and small MoE runs (default)
HSDP (use_hsdp: true)supportedmulti-node DP; pure DP or CP only, no-op on one domain
EPsupportedMoE expert sharding
CPsupportedlong-context SFT, SMPO, offline GRPO
TPsupportedattention and weight sharding; family support varies
ETPexperimentalexpert FFN memory, experts replicated
EP+CP, EP+TPsupported for selected familiesMoE plus long context or attention sharding
EP+ETPexperimentalexpert memory pressure; each ETP group stays inside one NVLink domain, and cross-node EP+ETP forms one ETP group per domain
TP+CP, TP+ETP, ETP+CP, PP+TP, PP+CP, any three axesunsupportedrejected at config validation
PP, PP+EPsupported for registered familiesdepth across NVLink domains; each stage is a whole number of domains, so two nodes at least
PP+ETPexperimental (tiny models only)expert FFN memory inside each stage, experts replicated

Layout rules and the shapes rejected before a run starts: Parallelism.

Model families

Any HuggingFace AutoModelForCausalLM runs under FSDP2; this table covers advanced parallelism. What each family is for: Supported Models.

Model familyEPETPCPTPPPNotes
Qwen3 (dense)——YesYesYesreference dense family; PP refuses the tied 0.6B, 1.7B and 4B checkpoints
Qwen3 MoEYesYesYesYesYesbroadest coverage; PP on tiny models and one 20-step Qwen3-30B-A3B run at pp2 × ep4, with moe_balancing: none
Qwen3-VL (text)——YesNoNoCP on text rows only; SFT and SMPO refuse an image run under CP
Qwen3.5 / Qwen3.6 / Qwen3.8-27B (dense)——NoYesYesthe 9B-VL checkpoint trains on plain FSDP with sdpa; PP takes the text tower
Qwen3.5 / Qwen3.6 MoEYesYesNoYesYesTP up to 2; the MoE-VL checkpoints train with EP; PP takes the text tower, with moe_balancing: none
Qwen3-NextYesYesNoNoNotiny models only
Qwen3.8-Flash-NextYesYesNoNoYesexperimental, text rows: EP8 trains the release cut to four layers, PP ran the whole text tower for 13 steps with checkpointing on, the rest runs on tiny models; full fine-tuning or LoRA; PP takes the text tower, full fine-tuning with moe_balancing: none; the 95 GiB n-gram table stays in host memory; ep_size: 1 loads about 246 GiB on every GPU (details ↗)
GPT-OSSYesYesYesYesYestrainable attention sinks
GLM-4 MoE LiteYesYesYesYesYesPP on tiny models only
Laguna S / XS 2.1YesYesNoNoYesETP and PP on tiny models only; on SDPA, so use packing, not padding_free
Inkling-SmallYesYesNoNoYesmultimodal MoE; LoRA and PP on tiny models only; PP takes the text tower, no packing
Gemma 4 MoEYesYesNoNoNono router balancing; LoRA adapts the language model only
Bailing MoE / LingYesYesYesNoNoEP covers Ling 2.0, Ling 3.0 and Ring; CP on Ling 2.0 only, and EP+CP on a tiny-model LoRA test only
LFM-2 MoEYesYesNoYesNo
Mistral4 MoEYesYesYesYesNoEP+CP on a tiny-model LoRA test only
DeepSeek-V4YesYesNoNoYeseager only; packing and padding_free refused; ETP on tiny models only; LoRA skips the grouped o_a_proj
Nemotron-H (Nemotron 3.5 Super VL, Lightning)YesYesNoNoYesEP8 on B300; Super VL's recipes, image SFT included, ran 5–20 steps on that release, PP ran Super VL's text tower for 13 steps with checkpointing on, the other modes and image SMPO on tiny models only; Lightning ran 5 steps at EP8 and served an untrained export on both engines; no video; no online RL; PP takes the text tower
Zaya (ZAYA1)YesYesNoNoNoalways without gradient checkpointing
Command A+ (Cohere2 MoE)YesYesYesYesNoonly EP8 ran on the 200B+ checkpoint, on Halo 1.0.0 or earlier; the other modes and LoRA on tiny models
GLM-5 Next (GLM-5.3-Flash)YesYesNoNoYescomposite VLM, SDPA only; PP takes the text tower; the fp8 release needs halo run convert-glm5-bf16 first
Step-3.7 FlashYesYesNoNoYescomposite VLM, SDPA only; PP takes the text tower; sharded EP saves refused
Any other HF causal LM——Noif the model has a tp_planNoa dense model without a TP plan raises under TP

EP+CP runs where EP and CP both do, EP+TP where EP and TP both do. Online RL per family: Rollout engines. Why a family drops a mode: the model pages ↗.

PEFT, quantization and data

FeatureStatus
LoRAsupported except under TP, EP+TP and PP; expert-projection adapters also refused when expert_tensor_parallel_size > 1. LoRA ran on release checkpoints for GPT-OSS 20B, Qwen3-30B, Qwen3-0.6B, Qwen3.6-35B-A3B, Ring-mini-linear-2.0, Gemma 4 26B and Nemotron 3.5 Super VL and, for a few steps or one, Qwen3.8-Flash-Next and Step-3.7 Flash's experts. The other families' LoRA ran on tiny models.
QLoRADDP, FSDP and CP only, with the 4-bit base replicated; ran on dense Qwen3 and on gpt-oss-20b with stock experts; rejected by both online RL methods and by offline GRPO under CP
Adapter mergehalo run merge-peft-adapters writes a standalone checkpoint; native EP expert LoRA merges at save time instead (merge_expert_lora_on_save: true)
FP8 / FP4 MoE QAT and exportexperimental: simulated backend for QAT, quantize-to-lowp for export; --export_format compressed-tensors writes the layout vLLM 0.32.0 serves, per-expert MoE layouts only (Serving the export with vLLM ↗)
FP8 rollouts (rollout_weight_precision: fp8_block)online GRPO on vLLM and async GRPO on either engine, on Qwen3.5/3.6 MoE only; ran on Qwen3.5-35B-A3B; refused beside adapter sync and QLoRA (FP8 rollouts ↗)
Muon, FlashAdamWsupported optimizer options, except under PP
HuggingFace, local and s3:// datasetssupported; offline tokenize, pack and shard with halo run prepare-dataset
Tree packing (tree_packing)SFT on raw text datasets: GPT-OSS, Qwen3 and Qwen3.5/3.6 (dense and MoE) on FA4 or FA2, GLM-5 Next on SDPA; refused under CP, TP and PP; ETP and EP+ETP are admitted (Tree Packing ↗)
MTP head training (mtp_loss_weight)SFT on text rows: Qwen3.5/3.6 (dense and MoE), Qwen3.8-27B, Qwen3-Next, GLM-4 MoE Lite and Nemotron-H, each as its release ships it, under FSDP2, EP and ETP; refused under CP, TP, PP, HSDP, tree packing, adapters, quantized loads and lowp_precision, and on every other tail by name (MTP heads ↗)
VLM packing / padding-freeunsupported: images cannot be packed, so VLM inputs use standard padding
Streaming an infinite corpusunsupported: the loaders build a map-style Dataset; pre-tokenize and shard offline

What each mode writes, and which resumes are exact: Checkpoints.

Validation

The GPU tests run on 8× B300 machines. Two 8-GPU machines ran node-local ep8 with DP through a per-node save, resume and evaluation, cross-node ep16 through a gathered save and reload, cross-node weight sync on both engines, and the two pipeline-parallel recipes, the Qwen3-8B one through a per-node save, resume and merge. At four GPUs a machine, two machines also ran an MTP head under node-local EP and plain FSDP2, and gpt-oss-20b at ep2 with ETP4. The other multi-node shapes run as a two-node simulation on one host. Checkpoints written by Halo 1.1.0 resume and export; a sharded EP save needs merge-ep-shards first. The full scope: Release validation ↗ and Verified topologies ↗.

Limits

  • No hosted training UI and no hyperparameter search.
  • No tested end-to-end Hub upload. Upload the final checkpoint yourself (Checkpoints).
  • Advanced parallelism support is per family, not automatic.
  • vLLM and SGLang run in their own containers, outside the training environment.

Source on GitHubUpdated 18 November 2026