Optimization
Memory reduction, throughput, and hardware utilization during training. The GPU training theory page explains why each lever works; the pages below toggle them.
Foundations
- GPU training theory — The roofline, training-step bottlenecks, and why each kernel exists — every number measured on Blackwell (B300).
- Throughput benchmarks — 1.2.0 against 1.1.0 on twelve shapes, the comparison with TRL, NeMo AutoModel and Megatron Bridge, then tokens/s/GPU and achieved TFLOPS on B300 across parallelism modes, sequence lengths, and model sizes.
- Halo vs stock TRL — Halo 1.2.0 against
trl.SFTTrainer1.17.1 at matched model, rows and kernels: layouts, ratios, TRL's expert parallelism and gpt-oss long context.
Memory
- PEFT (LoRA) — Train a few percent of parameters at ~50% less memory, across EP and CP (attention-only under ETP; rejected under TP) — with merge-back to a standalone checkpoint.
- BF16 optimizer — AdamW with bf16 masters and stochastic rounding at 6 bytes/param. Auto-enabled with
bf16: trueon FSDP/EP/TP. - FlashAdamW — Quantized 8-bit states + 24-bit masters (~5 bytes/param) with AdamW-matching convergence. Set
optim: flash_adamw. - Low-precision compute (FP8/FP4) — FP8/FP4 matmul over bf16 masters and checkpoints — a QAT oracle plus a block-scaled weight export (
scripts/after_training/quantize_to_lowp.py). - Allocator —
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:Truelowers peak reserved memory on dense runs at unchanged throughput; the dense SFT, DPO and SMPO examples launch with it, an EP run sets it only to fit.

Throughput
- Padding-free collator — Concatenates sequences into Flash-Attention blocks instead of padding, with boundary masking against cross-sample leakage.
- MTP heads — Train the multi-token-prediction head a family's release ships, jointly, detached or over a frozen trunk, with a hard or a soft target. Set
mtp_loss_weight,mtp_train_scopeandmtp_loss_type. - Tree packing — Rows that share a prefix store it once and branch after it; each example trains as it does packed flat, at fewer tokens.
- Flash attention — Auto-detected per architecture — FA4 on Blackwell (B200/B300), FA3 on Hopper (H100/H200), FA2 where neither imports.
- Masked matmul attention — GLM-5 Next's sparse-attention layers on a tiled matmul with a closed-form backward, about 5× over a forward, its recompute and a backward, on by default on Blackwell. Set
HALO_DSA_MATMUL=0to leave them on stock SDPA. - Block-sparse QSA attention — Qwen3.8-Flash-Next's sparse-attention layers on a runtime-length Triton kernel over the selected blocks, one compile for every pack length.
HALO_QSA_KERNEL=autoby default;offkeeps the tiled path. - Hyper-connection kernels — GLM-5 Next's and DeepSeek-V4's hyper-connections and mixes on one launch per step, at about a sixth of the time. Set
HALO_HC_KERNELS=0to leave upstream's forwards. The same page holds Qwen3.8-Flash-Next's gated residual, on by default;HALO_GATED_RESIDUAL_KERNELS=0leaves upstream's forward. - Grouped GEMM — Batched MoE expert matmul via
torch.nn.functional.grouped_mm, default on SM90+. Home to the atomic-free permute and MoE throughput playbook. - Liger kernels — Triton kernels for fused cross-entropy, RMS normalization, and SwiGLU (Qwen3 MoE, GLM-4.7-Flash, GPT-OSS, Gemma 4 and Nemotron-H run their norms on torch's fused
F.rms_norm). On by default — setuse_liger_kernel: falseto disable. - torch.compile — Inductor compilation of the spans between MoE/attention graph breaks (
torch_compile: true) — at most 1% on EP MoE, and slower inreduce-overheadmode. - Checkpointing plan —
gradient_checkpointing_plan: automeasures one fully checkpointed step and checkpoints only the decoder layers the memory budget needs (opt-in, packed SFT rows). - Muon optimizer — Newton-Schulz orthogonalization for matrix params, with fused Triton momentum and update kernels. Set
optim: muon. - NUMA binding —
torchrun --numa-binding noderuns each rank on its GPU's NUMA node, which helps host-bound rows on a multi-socket host.torchrunlaunches only.
Levers that do not help at these shapes
Each of these ran on B300 and did not pay off; its page holds the shapes.
- Halo's fp8/fp4 paths — QAT, not a speedup: whole steps at ep8 ran at 0.10–0.54× bf16 on B300 with the eager activation round trip, and simulated mxfp4 with the fused one runs at 0.93–0.99× on H200: Why bf16 is the default.
- DeepGEMM adapter — whole steps at 0.37–0.54× bf16, since casts and row copies cost more than the kernel saves: DeepGEMM.
- torch.compile on EP MoE — at most 1%, and
reduce-overheadis slower: Benchmark results. - Sub-bf16 master weights — parameters, checkpoints and optimizer state never go below bf16; bf16 with stochastic rounding is the floor: BF16 optimizer.
- FA2 on Blackwell where FA4 runs the model — the slow outlier against FA4 and SDPA at head_dim 128: FA4 vs FA2 vs SDPA. The defaults land on FA2 on purpose for GLM-4.7-Flash, Gemma 2/3 at head_dim 256 and softcapped rows that reach FA4's varlen kernels; leave those on it. Packed head_dim-256 GatedDeltaNet rows (Qwen3.5/3.6, Qwen3-Next) default to FA2 as well, where FA4 gains under 5% (Model-specific handling).
- MoE throughput measured at batch 1 — noisy from run to run; measure at batch ≥ 4 (Liger benchmarks).
Source on GitHubUpdated 18 November 2026