Core Framework

Halo: Frontier-Lab
Training for Everyone

in collaboration with others at White Circle

Today we are open-sourcing Halo, the training framework we use to train every model at White Circle. Halo adds distributed training directly to HuggingFace models, and it is deliberately minimal. There is no porting step, no new checkpoint format, and significantly less code to write: adding a new model family takes ~100 lines instead of a full re-implementation. On the same hardware, it delivers 2.3–2.8x the throughput of stock TRL at lower peak memory.

If you fine-tune open models, you have probably hit the wall that made us build Halo: your models are too large for stock HuggingFace tooling like TRL, and not large enough for Megatron to make sense. Training models with gradients, optimizer states, and activations stacked on top of the weights eventually pushes the setup past what a single GPU can hold. And if you want to train a Mixture-of-Experts (MoE) model, the frameworks that most people go for like TRL, Axolotl, or LLaMA-Factory do not have the parallelism strategies MoE training requires.

The alternative so far has been to shift to a framework like Megatron, which comes at quite a significant cost. You leave the HuggingFace ecosystem behind, re-implement the model in a new codebase, have the checkpoints converted to a new format and write thousands of lines of porting and bridging code for every model family trained. Megatron was built for massive scale, 500+ GPU clusters, and only makes sense when operating at that scale.

Most teams find themselves somewhere between the two current solutions, and this is what Halo is built for. It brings the parallelism strategies of the big frameworks to the models HuggingFace already defines, along with the components that make that parallelism usable in practice: fused kernels, bf16 optimizer, packing, and an async RL loop. None of it requires new algorithms, a new checkpoint format, or a new mental model.

The result is the fastest training we measured among the frameworks we benchmarked, and it scales well past plain fine-tuning: agentic and async RL, on-policy distillation, embeddings learning, and more.

2.3-2.8x

Halo throughput compared to stock TRL, with the model kept in plain Hugging Face format.

Throughput against peak memory · B300

higher is betterEP8EP4EP2EP1Halo · bs2EP8EP4EP2Megatron-LM · bs2Halo EP1 · 1 GPU · bs1Halo EP2 · 2 GPU · bs1NeMo AutoModel EP2Axolotl · 1 GPUMegatron-LM · EP2 · bs1stock TRL · ZeRO-3ms-swift · 1 GPU010,00020,00030,00060120180240 GBpeak memory (GB)tokens/s/GPUless memory is better
B300, sequence 4,096, GC off. Solid points: batch 1, each framework at a representative configuration. Up and to the left is faster at less memory. Dashed curves add the batch-2 Expert-Parallel sweep (EP1→EP8) for Halo and Megatron-LM; Halo's stays up and to the left at every width, and Megatron OOMs at EP1.

So what is Halo?

Halo is a distributed-training layer for Hugging Face models. Currently, most distributed training frameworks work by owning the model. They define their own version of each architecture, tuned for their parallelism implementation, and expect you to write the conversion scripts that move weights in and out. Every new model family means a new port with model code, config mapping, a checkpoint bridge, conversion tooling.

Halo inverts this. It adds four ways of splitting work across multiple GPUs directly onto Hugging Face models, without requiring a separate model definition or checkpoint format. A raw Hugging Face model goes in, and a raw Hugging Face model comes back out, loadable through the standard from_pretrained.

The four splitting strategies target different bottlenecks:

  • Expert Parallelism (EP) splits an MoE model's experts across GPUs.
  • Context Parallelism (CP) splits the input sequence itself across GPUs rather than the model. CP chops long sequence into chunks distributed across GPUs and coordinates them to still compute attention correctly.
  • Tensor Parallelism (TP) slices a single matrix multiplication across GPUs, with each GPU computing its portion and the results combined afterward. This applies to the dense, non-expert parts of the model.
  • Expert-Tensor Parallelism (ETP) applies TP's slicing technique inside an individual expert, for cases where even one expert is too large for a single GPU.

All of Halo's parallelism degrees are set through config rather than requiring manual intervention or a kernel support matrix to navigate by hand. They can also be combined together: EP+CP and EP+TP run in production, including EP across nodes, while EP+ETP works but is still experimental.

The rest of Halo makes each step faster: custom Triton kernels alongside Liger (the open-source fused-kernel library), a fused bf16 AdamW (AdamWBF16) that halves optimizer memory by keeping the master weights and Adam moments in bf16 instead of the fp32 stock TRL holds, and an asynchronous RL loop.

Our design choices for Halo

Two things set Halo apart from how these frameworks usually get built: it works with the open-source ecosystem instead of replacing it, and it's built for the scale most teams operate at day to day. The following choices cascaded from that:

1

HuggingFace is the training representation

Every trainer subclasses a stock Hugging Face or TRL trainer, and checkpoints stay plain SafeTensors. New architectures and community methods come for free, and the methods we add ship back through one public repo, with none of the private checkpoint formats the Megatron route needs.

2

Distributed behavior is added as wrappers

Frameworks like Megatron require you to re-define the model inside the framework, which means a custom class rebuilt from the framework's own building blocks before any training can start. Halo skips that step: EP wraps the existing MoE block, CP wraps attention, and dense and expert-FFN weights are sharded in place. No need to rewrite anything.

3

PyTorch-native parallelism

Tensor and data parallelism ride PyTorch's own FSDP2, DTensor, and DeviceMesh; only Expert Parallelism needs hand-built process groups, since DeepEP's data-dependent all-to-all is more than DTensor can express. We also deliberately left out pipeline parallelism. Pipeline parallelism splits a model into sequential stages that live on different GPUs, and its known weakness is the "bubble": GPUs sitting idle while they wait for the stage before them to finish. On a modern rack-scale machine, where every GPU shares a fast NVLink connection, that trade-off no longer pays off. FSDP2 plus Expert Parallelism achieves the same memory savings with no idle time, and the two can be scaled independently of each other. The EP/CP/TP/ETP degrees are set in YAML and resolved at startup — no manual sharding, no kernel support matrix to navigate.

4

Every method shares one distributed core

Every method is one trainer subclass over the same distributed layer, so it inherits the parallelism, kernels, and bf16 optimizer for free. Wrap any TRL method (SFT, DPO, GRPO) or add one of our own (SMPO, Offline GRPO, online distillation, reward modeling) — either way it's a small, self-contained change, and switching methods never means re-architecting the parallelism.

5

Tuned for the newest GPUs

We train on B200/B300 (full-bf16 AdamWBF16, FlashAttention-4), but the stack steps down to FlashAttention-3 on Hopper, FA2 on anything else, and LoRA fine-tuning on a 24 GB consumer card. Whichever GPU you run on, the software underneath is the same and tracks the latest versions of the ecosystem Halo builds on: FSDP2, DeepEP V2, Transformers v5, CUDA 13.

6

Rollout generation stays outside the training image

Online and multi-turn RL need an inference engine generating rollouts while the trainer runs. In Halo that engine is SGLang and it lives in its own isolated serving environment, so the training stack's Transformers/PyTorch versions stay untouched.

7

Halo is modular

Each speedup is a separate unit that can be added or dropped independently. DeepEP, Liger kernels, and Ulysses-style attention wrappers all attach through the same mechanism, so a new kernel or attention variant is one more wrapper, and not a stack rework.

8

Built to be driven by agents

The Docker images ship a CLI and Claude Code skills covering Halo's configs and common failure modes, so a coding agent can set up and launch runs without a human reading the docs first. It's how we run Halo ourselves.

Running a job

In practice, a run is one Docker image and one YAML file. Images are built per GPU architecture with FlashAttention and DeepEP V2 already compiled in, so there is no host Python environment to assemble. Pull the image for your GPU type from our registry and clone the repo to see examples and cookbooks.

Bash
# Clone to see cookbooks and examples
git clone --recurse-submodules https://github.com/whitecircle/halo

# Pull the image for B200 / B300
docker pull public.ecr.aws/whitecircle/halo:blackwell

# Pull the image for H100 / H200
docker pull public.ecr.aws/whitecircle/halo:hopper

Everything about the run then exists within a single YAML: the model, the data, the training method, and the parallelism degrees (EP, CP, TP, ETP), all resolved at startup. There is no kernel-support matrix to navigate and no manual sharding to reason about. The standard TrainingArguments / SFTConfig fields work alongside Halo's own blocks:

YAML
# examples/sft/gptoss/oss-20b-multinode-ep.yaml (abridged)
model_name_or_path: openai/gpt-oss-20b
dataset:
- ...

expert_parallel_size: 16

bf16: true # AdamWBF16 engages automatically
learning_rate: 5.0e-6
per_device_train_batch_size: 2

Then launch it:

Bash
docker run --rm --gpus=all --ipc=host -it public.ecr.aws/whitecircle/halo:blackwell \
halo launch sft examples/sft/gptoss/oss-20b-multinode-ep.yaml \
--nproc=8 -- --expert_parallel_size=8

One thing to know about this example: the YAML above is written for two nodes. expert_parallel_size: 16 spreads the experts across 16 GPUs, with DeepEP's all-to-all running over AWS EFA or InfiniBand between them. The --expert_parallel_size=8 at the end of the launch command overrides that so the same config runs on a single 8-GPU node. Any YAML field can be overridden this way — whatever follows -- on the command line takes precedence over the file.

Switching training methods means changing the launch target and its config: halo launch smpo examples/preference/smpo-qwen3-4b-ultrafeedback.yaml runs SMPO (Smooth Margin Preference Optimization), our reference-free preference method.

One trainer architecture, one parallelism stack

Every training method runs through the same trainer stack, over the same distributed setup:

  • Supervised fine-tuning
  • Preference optimization: DPO, KTO, and our SMPO
  • The GRPO family: offline, online with verifiable rewards (RLVR), and environmental
  • Distillation: teacher, self, and online (Self-Distilled Policy Gradient, SDPG)
  • Reward modeling, classification, and embedding
1Pick training method

Start with the trainer you already use — stock Hugging Face or TRL.

Every trainer gets the sameHalo layer
2Halo makes it distributed

DistributedTrainerMixin: One abstract layer mixed into every trainer. Everything below it already works by definition — you don't wire any of it up.

OptimizersAdamWBF16, distributed-ready
Lossesevery method, untouched
KernelsFlashAttention · DeepEP V2
Data & checkpointsparallel-aware
Just one parallelism config in your YAML
3Choose parallelism

Pick one or more dimensions. Halo creates the process groups.

EPExperts
CPContext
TPTensors
ETPExpert Tensors

Whatever data-parallel dimension is left falls through to FSDP2 fully_shard — native PyTorch.

Each method's trainer is just its upstream HuggingFace or TRL trainer (SFTTrainer, GRPOTrainer, DPOTrainer) with one shared DistributedTrainerMixin mixed in ahead of it, so the mixin's overrides win and everything else falls through to the trainer you already know. The mixin is where the distributed work actually happens, and it's ours: we strip Accelerate's DDP wrap and synchronize gradients ourselves, keep gradient clipping correct when experts are sharded across GPUs, swap in the AdamWBF16 optimizer, and make data loading and checkpointing parallelism-aware.

Underneath, the parallelism you set in the YAML (EP, CP, TP, ETP) partitions GPU groups first, and FSDP2 shards whatever data-parallel dimension is left. Adding a method is mostly subclassing its HuggingFace base and declaring which splits it supports. Not all support every split: Context Parallelism, for instance, is currently limited to the SFT and SMPO trainers, since the others work over whole sequences or paired completions. Everything else rides the same EP/TP/FSDP2 engine.

Staying inside the Transformers ecosystem means new architectures and community methods come for free. And because the trainer stack is ours, we drop our own algorithms in right next to them.

LoRA runs under all of Halo's data-parallel modes and under EP and CP. Under Expert Parallelism, Halo applies LoRA directly to the fused MoE experts. Stock PEFT can't reach these at all because fused experts are stored as single 3-D weight tensors rather than the nn.Linear layers PEFT knows how to adapt. This is why we built a grouped LoRA that adapts them natively.

The payoff is that, with the experts frozen, they carry no optimizer state and skip the expert gradient all-to-all entirely, so LoRA trains about 1.25x faster than full fine-tuning — 9,600–9,900 vs 7,896 tok/s/GPU on gpt-oss-20b at EP2 (8x B300, 4k sequence, GC on) — at about a third of the memory. On dense models the effect reverses: attention-only LoRA stays close to full fine-tuning while all-linear costs more; those numbers are in the appendix, along with the full LoRA/QLoRA support matrix across parallelism modes.

We also added quantization-aware training and a real fp8/fp4 path on DeepGEMM (actual low-precision kernels, not simulated quantization that is common in most stacks), though both are still experimental and aimed at producing quantized models rather than faster training.

The numbers: faster at less memory across the board

Setup for the runs below, unless a caption says otherwise: gpt-oss-20b in bf16 on an 8-GPU NVIDIA B300 node (274 GB usable per GPU), synthetic exact-length sequences, GC on.

The TRL baseline is the strongest stock configuration we could build: it runs the same FlashAttention-4, Liger kernels (with fused-linear cross-entropy), and grouped-GEMM experts as Halo, so only the trainer, the sharding, and the optimizer differ. The harness behind the TRL comparisons ships in the repo, so those numbers are reproducible end to end.

The four figures test one claim — faster at less memory — from four angles: whether it fits at all, how it compares to the tools already used, whether it holds without MoE, and what are the results at extreme length.

Same convergence

Halo wraps the same HuggingFace modules TRL trains, so a bf16 run reproduces the stock TRL loss curve step for step. The throughput and memory wins below are free, with no quality tradeoff.

Convergence · Halo tracks stock TRL

stock TRLHalo dense · EP1Halo EP2Halo EP8
1010.10.01050100150200training step
gpt-oss-20b, same seed and data: training loss over 200 steps for stock TRL and Halo at EP1 (no expert sharding), EP2, and EP8.

Halo vs. stock TRL

This is the comparison that matters for most teams, as stock TRL's SFTTrainer is where most fine-tuning happens today.

Halo EP1 reaches roughly 3x the throughput of the stock TRL SFTTrainer at 4k, 16k, and 32k (top figure), while staying below it on peak memory the whole way. Peak measured throughput on this model: 24,456 tok/s/GPU (EP1, 4k, batch 4, GC off).

Throughput by sequence-and-batch shape

Seq · batch
stock TRL · ZeRO-3Halo EP1 · ZeRO-2Halo EP1 · ZeRO-3Halo EP2 · ZeRO-2Halo EP8 · ZeRO-2
4k·b1
022,000 tokens/s/GPU
Tok/s/GPU by sequence-and-batch shape: stock TRL ZeRO-3 against Halo EP1 (both ZeRO-2 and the matched ZeRO-3), EP2, and EP8, GC on.

Matched sharding · Halo EP1 vs stock TRL

Seq · batch
Halo EP1 · ZeRO-2Halo EP1 · ZeRO-3stock TRL · ZeRO-3

Throughput · multiplier vs stock TRL

4k·b1
022,000 tok/s/GPU

Peak memory · lower is better

4k·b1
0170 GB per GPU
The matched-sharding view: same EP1 (DP=8) layout and identical kernels on both sides, so the trainer and AdamWBF16 account for the whole gap.

Qwen3-30B-A3B has 128 experts to gpt-oss-20b's 32, and that narrows the gap. It runs from a near-tie at EP8, batch 2 up to 1.8x at EP2, batch 1.

Qwen3-30B-A3B throughput

stock TRLHalo EP8Halo EP2
4k·b1
4k·b2
012,000 tokens/s/GPU
Qwen3-30B-A3B, 8x B300, 4k, batch 1 and 2, GC on: stock TRL against Halo EP8 and EP2.

Fitting in less memory

The fastest configurations above spend memory to buy speed. Halo's default ZeRO-2 (FSDP2 shards the optimizer and gradients but replicates parameters) keeps more state resident than TRL's fully-sharded ZeRO-3.

When memory matters more than speed, the opposite configuration wins. Expert Parallelism shards the experts, which are most of a MoE's weights, across GPUs, and AdamWBF16 holds optimizer state at half the bytes of fp32-master AdamW. At a 4k, batch-1 shape, Halo at EP8 trains gpt-oss-20b in 26 GB per GPU against stock TRL ZeRO-3's 47.6 GB — under the 40 GB of an older-generation card, though all measurements here are on B300. Matched ZeRO-3 lands at 28.7 GB and stays faster.

Peak memory by sequence-and-batch shape

Seq · batch
stock TRL · ZeRO-3Halo EP1 · ZeRO-2Halo EP1 · ZeRO-3Halo EP2 · ZeRO-2Halo EP8 · ZeRO-2
4k·b1
0130 GB per GPU
Peak GB per GPU at the same shapes as the throughput figure; lower is better. EP8 at 4k, batch 1 is the 26 GB configuration.

Still faster, even without MoE

Everything so far involved Expert Parallelism. We therefore decided to run tests on Qwen3.5-4B, a dense VLM with no experts to distribute. We conducted the run in the same single-GPU FSDP2 route as every other framework, and Halo still leads at 16k. This means the other optimizations behind Halo, like kernel stack and the bf16 optimizer state, win on their own.

Training at 256k tokens

Finally, we compared gpt-oss-20b at up to 256k tokens on 8x B300. The gains are more modest here but still present; at these lengths, moving data is as important as computing on it. Halo runs 2.1x stock TRL's throughput at 64k, and then to 1.3x at 256k. We find that splitting the sequence across GPUs with Context Parallelism roughly halves the footprint per device, which is what makes 256k trainable at all.

Long context · up to 256k tokens

Ctx size
stock TRL · ZeRO-3Halo EP1 · ZeRO-3Halo EP8+CP8Halo CP-only · ZeRO-3

Throughput

64k
015,000 tok/s/GPU

Peak memory · lower is better

64k
0160 GB per GPU
gpt-oss-20b, 8x B300, batch 1, GC on, fused-linear cross-entropy on both sides. Left: throughput — Halo EP1 (sequence kept whole) leads TRL at every length. Right: peak memory — EP8+CP8 and CP-only (no EP) land at roughly half TRL's footprint.

End-to-end example: GLM-4.7-Flash-Coder

We used Halo to train GLM-4.7-Flash-Coder, a 30B MoE model, with multi-node SFT on a 177M-token agentic dataset. It lifted the base model's resolved rate on SWE-rebench-V2 from 33.15% to 41.65%, a 26% relative gain on a live agentic coding benchmark. The config, evaluation, and agentic setup are documented on the model card.

SWE-rebench-V2 resolved rate

before Halo SFT
after Halo SFT
0%50% resolved
SWE-rebench-V2 resolved rate before and after the Halo SFT run.

Agentic and multi-turn RL, natively

Online and multi-turn RL require a separate inference engine for rollout generation. The model talks to an environment — calling tools, running code, searching — before any reward exists, while a separate serving fleet generates those conversations. Halo folds this into the same trainer stack as SFT, still on top of TRL's GRPO trainer, with a runnable code-contests example on gpt-oss-20b in the repo.

SGLang is the primary rollout engine. Halo uses SGLang as the primary backend: SGLang runs in an isolated serving environment, returns sampled token IDs, log probabilities, and optional MoE routing decisions, while Halo synchronizes updated model parameters over NCCL. With more than one server, the next batch generates while the current one trains, and a truncated importance-sampling ratio corrects the small off-policy lag.

Halo supports selective full-parameter fine-tuning, where only a subset of transformer blocks is optimized. Instead of synchronizing the full policy after every optimizer step, Halo performs an element-wise comparison against the parameter state currently loaded by SGLang and transmits only modified BF16 elements as (flat_index, value) pairs.

In our LFM R3 run, transformer blocks 7, 14, and 21 were trainable, totaling 1.1B parameters. The median synchronization contained 19.46M modified BF16 elements, or ~116 MB of indexed parameter updates. This is lossless and includes all modified parameter types, not only MoE experts.

Compared with Halo's standard 16.94 GB full-policy dense synchronization on LFM, this reduces the median transfer volume by 145x. MoE routing replay is independent of parameter synchronization and adds only ~44 KB of metadata per rollout request.

Environments are built in: code contests graded by their tests, SWE tasks, tool use over MCP, and search QA, with a registry for your own and SandboxFusion for sandboxed execution. We handle the multi-turn objective exactly, too. A model's own tokens and a tool's injected outputs land in one flat sequence with no marker between them, so training naively would put loss on the tool's text; instead we recover each assistant turn's exact token span by diffing tokenized chat-template prefixes, and train on nothing else.

Model-family integration is cheap

Adding a model family to Halo takes a few hundred lines at most, orders of magnitude less than Megatron. The integration is only a wrapper around the model's MoE or attention block.

Once registered, the family reuses Halo's existing trainer, optimizer, checkpoint, and launcher code.

Our assessment of LOC needed across Halo and Megatron-Bridges counts the model, config, bridge, and conversion files needed to connect the same family to a Megatron-Core stack.

Lines of code per integration

Halo · EP wrapper
Megatron-Bridge · Qwen3.5
Megatron-Bridge · Bailing/Ling
02,000 LOC
Halo: one wrapper file around the Hugging Face block, counted from the repo. Megatron-Bridge: a full upstreamed integration for a recent decoder family — bridge, config, recipe, and conversion tests, ~900–1,800 lines; the weight-mapping file alone is smaller.

Novel model integration example

Poolside · Laguna-S-2.1

Laguna-S-2.1 is a non-common architecture model released by Poolside. Its official implementation is not yet merged into the Transformers repository. Nevertheless, Halo provides support for Laguna in just 44 LoC.

Laguna S 2.1 118B-A8B · throughput and peak memory · 4x B300

Throughput

Halo · EP4
Axolotl
012,000 tok/s

Peak memory · lower is better

Halo · EP4
Axolotl
0250 GiB
Full-parameter BF16 SFT, sequence 2,048, batch 1 per GPU, mean of 20 post-warmup steps. Halo runs EP4 with grouped GEMM and fused SwiGLU; Axolotl runs FSDP2 with grouped MM. Lower peak memory is better. Laguna support is 44 lines of Halo code.

LiquidAI LFM support

Liquid AI · LFM2.5-8B-A1B

Liquid's LFM2.5-8B-A1B is a hybrid MoE with 8.3B total and 1.5B active parameters, 18 double-gated LIV convolutional blocks, six GQA blocks, and a 128k context window.

Halo trains it in native Hugging Face format. Roughly 120 lines wrap the original MoE block with EP, ETP, and combined EP+ETP, after which every Halo trainer and capability works unchanged.

That includes dedicated embedding-model fine-tuning. There is no model reimplementation or checkpoint conversion.

Among the frameworks we benchmarked, Halo is the only one with model-aware LFM2.5 support for both EP and ETP, including combined EP+ETP configurations.

On one B300, Halo was 10–20% faster than Axolotl across the two working single-GPU comparison runs.

SequenceHalo tok/s/GPUAxolotl tok/s/GPU
8,19231,26228,330
16,38435,51729,520

Supported families

Parallelism support is per family. Families not listed fall back to the standard FSDP2 route, and any causal LM or VLM that Transformers loads trains there. Cond. () marks support with a condition attached: Zaya's EP and ETP require gradient checkpointing off. Context Parallelism currently applies to SFT and SMPO. The full matrix, including the combined modes, is in the appendix.

FamilyEPCPTPETP
GLM-4 MoE LiteYesYesYesYes
GPT-OSSYesYesYesYes
Mistral4 MoEYesYesYesYes
Qwen3 MoEYesYesYesYes
LFM2.5-8B-A1BYesNoYesYes
Qwen3.5/Qwen3.6 MoEYesNoYesYes
InclusionAI LingYesNoNoYes
Gemma 4 MoEYesNoNoYes
Kimi LinearYesNoNoYes
Poolside LagunaYesNoNoYes
Zyphra ZayaNoNo
Cohere Command A+YesNoNoNo
Thinking Machines InklingYesNoNoNo

The table tracks parallelism only; the full method-by-family matrix is in the repo's agent-docs/models/index.md.

Some of the hard problems along the way

Making an unmodified Hugging Face MoE train fast and reliably at frontier scale surfaced a run of non-obvious problems.

The step is almost all communication. In an MoE layer, each token is routed to a few chosen experts, computed there, and gathered back. Even with DeepEP moving the tokens as fast as the hardware allows, that round trip takes up 88–93% of a training step. The expert matmul itself is only 6.6%.

This shaped what we optimized. Rewriting the routing's backward pass to be atomic-free (PyTorch's default serializes on atomic adds) improved end-to-end throughput by up to 65% on a 256-expert model. Quantizing the experts to fp8 or fp4, on the other hand, changed nothing: the matmul already runs as fast as memory can feed it, so making it cheaper to compute doesn't help.

A crash three files from its cause. For weeks, EP8 training died once sequences reached 64k tokens, with a CUDA fault deep in DeepEP's network transport. The actual cause was elsewhere. A batch holds many sequences, and MoE routing is uneven — in the failing runs, the router had concentrated 745k tokens onto the experts of a single GPU. At that size, a fused SwiGLU kernel in a different part of the codebase overflowed its 32-bit index arithmetic, and the wrapped index became an out-of-bounds memory access. Because the fault was asynchronous, it surfaced on a different GPU, disguised as a networking error. The fix was a single cast to int64 — finding it was quite a bit of work.

A race between two collectives. At certain Expert Parallelism widths, half of all runs hung. Two collective operations were sharing one NVLink fabric (DeepEP's expert-combine step and FSDP2's reduce-scatter) and kept racing each other for it. A lock can't fix a race between collectives, so we made the bad configuration impossible instead: Halo rejects it at startup, and every backward-pass collective is kept inside the expert group so none of them can overlap the combine.

Full BF16 master weights. AdamWBF16, our fused-Triton optimizer (with Muon and FlashAdamW as alternatives), keeps weights, gradients, and optimizer state in bf16 and halves the optimizer memory at no cost to quality of full FP32 master weights training, but only after we ran down three quiet ways bf16 breaks. The two Adam moments have to round by opposite rules: nearest rounding biases the variance estimate upward, so only it gets stochastic rounding. A bf16 all-reduce accumulates more error the more GPUs join it, so that one collective upcasts to fp32. And a naively cast bf16 master weight simply stops moving once an update is smaller than bf16 can represent at that weight's magnitude — the update rounds away and the weight freezes.

None of this leaks into the model. During training, experts are fused into their sharded layout across GPUs; at save time, Halo reassembles them into standard HuggingFace weights, loadable with from_pretrained as usual.

What Halo does not do

We left pipeline parallelism out: on large NVLink domains, FSDP2 plus Expert Parallelism avoids the pipeline bubble and keeps EP orthogonal to data parallelism. We'll likely add pipeline parallelism down the line, but it stays a lower priority while most post-training runs fit comfortably on FSDP2 plus EP inside an NVLink domain.

There's no elastic orchestration either — resuming a run requires the same number of GPUs it started with — and no hosted UI, experiment tracking, hyperparameter search, or no-code workflows. Halo is ultimately a training framework, not a platform.

From-scratch pretraining works on the standard FSDP2 route, but not yet with EP/TP/CP/ETP. And because images are built per GPU architecture, some kernels and features depend on the card and image you're running.

Our ambition with Halo

We are excited to be finally releasing Halo. It started out as internal infrastructure, carried our own training runs for months, and releasing it is the natural next step. We are now closing the gap between the lightweight HuggingFace tooling that most teams start with, and the heavyweight systems built for significant scale.

We will keep building in the open. The next priority is making Halo easier to extend with new research methods, through self-contained, versioned components, so that trying a new training method never puts the methods you already run at risk.

We'd love for anyone who has read this far to build on Halo with us. Issues, PRs, and new wrapped model families are all welcome!

github.com/whitecircle/halo · Apache 2.0

Appendix

Supporting measurements: optimizer-state memory, EP scaling against Megatron-LM, and per-kernel deltas.

Gradient checkpointing on vs. off

The headline numbers keep GC on, the setting most real runs use. Switching it off removes the recompute pass on both sides; Halo's lead over stock TRL grows to about 3x.

Gradient checkpointing on vs. off

Seq · batch
GC onGC off
stock TRL
EP1 z2
EP1 z3
EP2
EP8
013,000 tokens/s/GPU
gpt-oss-20b, throughput at 4k, 16k, and 32k (batch 1) with GC on and off, per configuration.

Optimizer state in bf16

AdamWBF16 holds bf16 master weights and bf16 Adam moments at 6 bytes per parameter — half the 12 bytes of an fp32-master AdamW, and part of why a 20B model fits on one GPU.

A naive down-cast would drift, so we wrote it as a fused Triton kernel that applies stochastic rounding on every update and keeps the second moments in bf16; the training behavior then tracks fp32 closely — the same bf16-optimizer-state choice DeepSeek V3 made at 671B scale. Muon and FlashAdamW are supported as alternatives, FlashAdamW building on Databricks' implementation.

Optimizer state for a 20B model

fp32-master AdamW
AdamWBF16
0
Bytes per parameter times 20.7B parameters. AdamWBF16 holds bf16 weights and moments with stochastic rounding; fp32-master AdamW holds fp32 weights and moments.

EP scaling vs Megatron-LM

gpt-oss-20b at 4k, batch 2, GC off, on 8x B300, sweeping Expert-Parallel width. Halo is the only framework that fits at EP1, where Megatron-LM runs out of memory, and it leads through EP4. At EP8 the run is communication-bound and Megatron-LM edges ahead.

Throughput vs Expert-Parallel width · 8x B300 · batch 2 · GC off

HaloMegatron-LM
27,70721,64217,51911,85613,93215,87614,734OOM010,00020,00030,000EP1EP2EP4EP8
gpt-oss-20b, sequence 4k, batch 2, GC off. Tok/s/GPU for Halo and Megatron-LM at each Expert-Parallel width; Megatron-LM OOMs at EP1 and passes Halo at EP8.

Peak memory vs Expert-Parallel width · 8x B300 · batch 2 · GC off

HaloMegatron-LM
158 GB104 GB74 GB58 GB246 GB151 GB104 GBOOM0 GB70 GB140 GB210 GB280 GBEP1EP2EP4EP8
Same sweep, peak allocated GB per GPU.

LoRA and QLoRA

Qwen3-8B, 1x B300, sequence 16k, GC on, r=64, full stack (AdamWBF16, Liger, FA4). This is a separate run from the kernel table's Liger A/B, which isolates Liger alone — hence the slightly different full fine-tuning figure.

ConfigTrainabletok/s/GPUPeak mem
Full fine-tuning100%17,34264.6 GB
LoRA r=64, attention0.7%16,82434.6 GB
LoRA r=64, all-linear2.1%11,71735.9 GB
QLoRA r=64, all-linear3.6%15,80225.3 GB

QLoRA's higher trainable share is the same 175M adapter parameters over a 4-bit-packed base; its speed edge over bf16 all-linear LoRA comes from the 4-bit base cutting weight bandwidth on the MLP matmuls.

Fused kernels, measured one at a time

Each kernel is measured on its own model and setup, so the delta comes from the kernel, not the config around it.

KernelModelSetupMeasurement
Grouped GEMM (expert FFN)Qwen3-30B-A3B128 experts, seq 4k17.24 → 1.45 ms fwd+bwd (11.9x)
Grouped GEMM (end-to-end)Qwen3-30B-A3BEP, batch 47,211 → 15,307 tok/s/GPU
Liger fused kernelsQwen3-8Bseq 16k, GC on11,943 → 16,396 tok/s/GPU; 78.6 → 64.6 GB
Expert-bias gathergpt-oss-20bEP backward1,256 → 6,310 tok/s/GPU (serialized reduction → one GEMM)

Grouped GEMM against the per-expert loop

per-expert loop · opt-ingrouped GEMM · default
stock TRL
Halo EP1
Halo EP2
Halo EP8
011,000 tokens/s/GPU
gpt-oss-20b, 4k, batch 1, GC on. Both frameworks default to grouped GEMM; the ladder shows what the opt-in per-expert loop would cost at each configuration.

A related atomic-free permute kernel in the EP backward is worth another 18% (4k) to 65% (16k) end-to-end on Qwen3.6-35B at EP8. And for GPT-OSS specifically, a Triton kernel fuses six memory-bound SwiGLU operations into one. Liger does not provide this, since its SwiGLU is a no-op for this family.

A few more levers

Packing. With variable-length data, padding every sample to the longest one wastes compute. Halo packs multiple sequences into each row and runs variable-length FlashAttention with per-sequence boundaries, so no sample attends across another. On a dataset that is roughly 75% padding, packing is worth about 9.2x the throughput of naive padding; a padding-free collator that keeps samples separate but drops the waste gets about 2.3x.

torch.compile. Inductor composes with the kernel stack instead of competing with it. It fuses the non-expert spans between DeepEP dispatches to about Liger's gains, and stacks with Liger to within a couple of percent, since both target the same regions.

One environment variable. CUDA_DEVICE_MAX_CONNECTIONS=1 is neutral on dense and EP2 runs but worth about 9.7% at EP8, where a single intra-node expert group benefits from serialized launch queues.

Full parallelism matrix

FamilyEPCPTPETPEP+CPEP+TP
GLM-4 MoE LiteYesYesYesYesYesYes
GPT-OSSYesYesYesYesYesYes
Mistral4 MoEYesYesYesYesYes
Qwen3 MoEYesYesYesYesYesYes
LFM2.5-8B-A1BYesNoYesYesNoYes
Qwen3.5/Qwen3.6 MoEYesNoYesYesNoYes
InclusionAI LingYesNoNoYesNoNo
Gemma 4 MoEYesNoNoYesNoNo
Kimi LinearYesNoNoYesNoNo
Poolside LagunaYesNoNoYesNoNo
Zyphra ZayaNoNoNoNo
Cohere Command A+YesNoNoNoNoNo
Thinking Machines InklingYesNoNoNoNoNo

Cond.: Zaya's EP and ETP require gradient checkpointing off; Mistral4's EP+CP needs a multi-node or NVLink-domain topology.