Glossary

Parallelism, kernel, optimizer, and method terms. Sizes and flags anchor to the config field in ParallelismConfig (halo/distributed/parallelism_config.py) or the YAML key that sets it. Legal combinations: parallelism mode matrix. Per-trainer support: Trainer Compatibility.

Parallelism

TermDefinition
Data Parallel (DP)Each rank holds a model replica and processes a distinct batch; gradients all-reduce across ranks. data_parallel_size is the number of distinct batches (Stacking the dimensions). PP, TP, CP, and ETP reduce it — EP does not. See Data Parallelism.
FSDP2The DP implementation for all torchrun modes: fully_shard with reshard_after_forward=False (SHARD_GRAD_OP behavior — params stay unsharded between forward and backward, faster, higher peak memory). EP modules are excluded via ignored_params when ep_group_size > 1; at 1 the default fsdp_shard_ep1_experts lets FSDP2 shard them. Set fsdp_reshard_after_forward: true (rejected under any ep_group_size > 1 — EP and pure ETP alike — under TP when data_parallel_size > 1, and under PP) for FULL_SHARD (lower peak memory). See Data Parallelism.
Expert Parallel (EP)Shards a MoE model's experts across ep_size GPUs and routes each token to its experts with an all-to-all (DeepEP dispatch → expert compute → combine). EP is orthogonal to DP — every rank still gets a different batch, so EP alone does not reduce data_parallel_size. Requires DeepEP (no NCCL fallback). See Expert Parallelism.
Context Parallel (CP)Splits one long sequence across cp_size GPUs and reconstructs attention with all-to-all (Ulysses). Each CP group processes one batch, so CP reduces data_parallel_size. CP must be NVLink-local. Incompatible with TP and ETP. Runs only under trainers that declare SUPPORTS_CP: SFT, SMPO and offline GRPO (full fine-tuning). See Context Parallelism.
Tensor Parallel (TP)Shards large weight matrices across tp_size GPUs using DTensor. Reduces data_parallel_size by tp_size. Dense models use their native base_model_tp_plan (tp_plan="auto", shards attention and MLP); MoE models use the Halo selective-TP path (attention sharded, experts left to EP, embeddings and lm_head replicated). TP must be NVLink-local. See Tensor Parallelism.
Expert Tensor Parallel (ETP)Shards each expert's FFN weights across expert_tp_size GPUs — independent of tp_size, MoE-only, NVLink-local. Pure ETP (ep_size=1) replicates experts and shards their FFN; EP+ETP (ep_size>1 and expert_tp_size>1) distributes experts and shards each one. The expert-TP reduce runs in token space, outside the DeepEP dispatch→combine span. Mutually exclusive with attention TP and with CP. See Expert Tensor Parallelism.
ep_sizeExpert distribution size — how many distinct expert subsets exist. The all-to-all routes tokens across ep_size ranks. Set with --expert_parallel_size.
expert_tp_sizeETP size — how many ranks each expert's FFN is sharded over. Set with --expert_tensor_parallel_size.
ZeRO-2 / ZeRO-3The two FSDP2 sharding depths: ZeRO-2 (fsdp_reshard_after_forward: false, the default) keeps parameters unsharded between forward and backward — faster, higher peak memory; ZeRO-3 (true) reshards after the forward for lower peak memory. See Data Parallelism.
ep_group_sizeThe full EP process-group size: ep_size × expert_tp_size. Inside it, the DeepEP dispatch/combine runs over ep_size-rank dispatch groups and the expert-FFN reduce over expert_tp_size-rank groups (layout). The whole group is what must fit the NVLink domain (node scope) or world (global scope). Computed; not set directly.
ep_scopeWhere EP groups live: node (within one NVLink domain, all communication over NVLink) or global (spanning domains, all-to-all over RDMA). auto resolves to node when ep_group_size ≤ nvlink_domain_size, else global. EP+CP requires node scope with ep_group_size == nvlink_domain_size; an EP group larger than one NVLink domain needs global, while node-local groups replicated across nodes stay node.
nvlink_domain_sizeThe locality unit for "node-local" EP/CP/TP grouping — the number of GPUs reachable over NVLink. Defaults to gpus_per_node (one OS node on a standard cluster). On GB200/GB300 NVL72 the NVLink fabric spans the rack (up to 72 GPUs across ~18 OS nodes), so set NVLINK_DOMAIN_SIZE to use NVLink-wide parallelism. See Multi-Node.
Pipeline Parallel (PP)Splits the layer stack into pp_size sequential stages, each a torchrun rank group; only P2P boundary activations cross a stage. Outermost axis — it carves the world first (stage_world_size = world_size / pp_size). See Pipeline Parallelism.

Which axis combinations may run is an allowlist (SUPPORTED_AXIS_SETS in parallelism_rules.py), checked before any rank math — anything unlisted is rejected by default. Tabulated in the parallelism matrix.

Kernels and performance

TermDefinition
Flash Attention 2 (FA2)The fused attention kernel present on every image (flash-attn 2.x). The portable fallback: auto-detection picks it where FA4/FA3 are unavailable, and per-family rules route to it what FA4 cannot train. See Flash Attention.
Flash Attention 3 (FA3)Hopper-only (SM90) attention kernel, built from source (flash-attention/hopper/). On Hopper the image keeps FA2 + FA3. See Flash Attention.
Flash Attention 4 (FA4)Blackwell CuTe-DSL kernel (flash-attn-4[cu13], the flash_attn.cute submodule), co-installed with FA2. Faster than FA2 on the attention kernel in isolation; the end-to-end MoE training-step delta is much smaller (attention is a minority of MoE step FLOPs). Auto-selected on SM100+, so it is the production default on Blackwell. See Flash Attention.
Grouped GEMMtorch.nn.functional.grouped_mm batches the per-expert matmuls of a MoE layer into one kernel launch. The default on SM90+ (use_grouped_gemm: true); the end-to-end win scales with local experts per rank (num_experts/ep_size) and is smallest at high EP (few local experts), where the per-expert loop is still slower at every measured shape. Works EP-distributed or dense-style (ep_size=1). See Grouped GEMM.
Liger kernelsFused Triton kernels for cross-entropy, SwiGLU, RMSNorm and RoPE (use_liger_kernel: true, a toolkit default). Qwen3 MoE, GLM-4.7-Flash, GPT-OSS and Gemma 4 fill the RMSNorm role with torch's fused F.rms_norm instead, under the same rms_norm flag. FLCE (fused linear cross-entropy) fuses the lm_head matmul with the loss, removing the large logits tensor — the biggest memory win for high-vocab models. See Liger Kernels.
DeepEPThe expert-parallel communication library — the all-to-all dispatch/combine engine behind EP. The toolkit uses DeepEP V2 (deep_ep.ElasticBuffer — the NCCL Gin (RDMA) backend inter-node, a non-Gin NVLink path intra-node). Required for EP (no NCCL fallback). See DeepEP.
tokens/s/GPUThe headline throughput metric, hardware- and sparsity-independent: the cluster's unique tokens per second over the world size, so ranks sharing a batch (TP, expert-TP, CP, a pipeline chain) count it once. Use it to compare runs because a utilization percentage is computed against a specific GPU's peak FLOPS (the same achieved TFLOPS reads ~2.3× lower against a B300's 2250 TFLOPS peak than against an H200's 989.5). See Throughput Benchmarks.
S-MFUSparsity-aware Model FLOPS Utilization: the roofline fraction a step achieves, scaling the expert FLOP term by (top_k / num_experts) × ep_size so it does not credit experts that never fired (a plain MFU over-counts for MoE). A diagnostic, not the headline — it is GPU-peak-relative, so compare tok/s/GPU or achieved TFLOPS across hardware. Full formula: Throughput Benchmarks; roofline background: GPU Training Theory.
Gradient checkpointing (GC)Recompute activations in the backward pass instead of storing them — trades ~20–30% wall-clock for memory. Turn it off when the batch fits unchecked (a free speedup). Zaya rejects GC on every path (a toolkit patch clears its supports_gradient_checkpointing). See GPU Training Theory.
Padding-free / packingTwo ways to remove padding waste. Packing concatenates short examples into full-length sequences; the padding-free collator passes a flat position_ids + cu_seqlens batch with no pad tokens, so attention never attends across example boundaries. See Padding-Free Collator.

Optimizers and precision

TermDefinition
BF16 mixed precisionbf16: true (a toolkit default): params and compute in bf16. Long-sequence forward keeps outputs in bf16 (the fp32 output conversion is disabled) to avoid OOM on the logits. See BF16 Optimizer.
AdamWBF16BF16 optimizer auto-enabled under bf16 on FSDP/EP/TP (6 bytes/param). Stochastic rounding on both the weight write and the exp_avg_sq store (nearest rounding biases the always-positive second moment by tens of percent, moving the effective LR); exp_avg stays nearest. Not auto-enabled under replicated DDP, which sits outside the validated SR matrix — bf16_optimizer=True opts in. See BF16 Optimizer.
Stochastic rounding (SR)Round up or down with probability set by the distance to each bf16 grid point, so small updates below the bf16 ULP land in expectation instead of being dropped. Used in AdamWBF16 and in Muon's bf16 weight write. See BF16 Optimizer.
fp32_grad_reduceReduce gradients across ranks in fp32 while keeping bf16 storage (still 6 B/param). Covers every reduction the mixin owns: FSDP2's reduce_dtype, the EP router/expert grad sync, the TP replicated-grad sync and the QLoRA adapter all-reduce; default off. Distinct from fp32_non_ep_params (12 B/param fp32 storage), which implies it for the FSDP2 reduce only. See BF16 Optimizer.
MuonMomentum optimizer with a Newton–Schulz orthogonalization of the update; the bf16 weight write uses SR (updates near lr are otherwise below the bf16 ULP). See Muon Optimizer.
FlashAdamWAdamW variant with quantized optimizer states — lower optimizer-state memory at a per-step state quant/dequant cost. Optional uv extra (flash-optimizers). See FlashAdamW.
Low precision (fp8/fp4/mxfp4)Mixed-precision compute: bf16/fp32 master weights and checkpoint unchanged, low-precision GEMM operands. lowp_precision: bf16 | fp8 | fp4 | mxfp4 (bf16 = off; fp4 = nvfp4, most accurate). The default backend is simulated (block-scale fake-quant — correct numerics, the QAT oracle, not faster); native DeepGEMM is opt-in (HALO_DEEPGEMM_NATIVE=1). bf16 is the production default: no fp8/fp4 path beats it (simulated mxfp4 comes closest, 0.93–0.99× bf16 on 8× H200). See Low-Precision MoE Kernels.
DeepGEMMThe native fp8 and FP8×MXFP4 grouped-GEMM kernels with on-device m_indices, behind HALO_DEEPGEMM_NATIVE=1. Never auto-routed; Halo's adapter runs whole steps at 0.37–0.54× bf16 at ep8. See Low-Precision MoE Kernels.

Methods and infrastructure

TermDefinition
SFTSupervised fine-tuning on {"prompt": [...]} conversations (DistributedSFTTrainer). See SFT.
DPODirect Preference Optimization on {prompt, chosen, rejected} triples with a reference model (DistributedDPOTrainer). See DPO.
SMPOSmooth Margin PO — reference-free preference optimization with a target margin (SMPOTrainer). See SMPO.
Offline GRPOGroup-relative PO with pre-computed completions and rewards ({prompt, completions, rewards}); no online sampling (OfflineGRPOTrainer). See Offline GRPO.
Online GRPOGRPO with live vLLM generation and NCCL weight sync (DistributedOnlineGRPOTrainer). See Online GRPO.
Async GRPO with EnvironmentsMulti-turn environment RL — async Ray-actor rollouts, rollout-server HTTP generation, NCCL weight sync (DistributedAsyncEnvironmentalGRPOTrainer). See Async GRPO.
OPDOn-policy distillation, named after the SDPG paper (arXiv:2606.04036): the term beta(k)·L_OPD both SDPG arms add to their objective, pulling the student's next-token distribution toward the same model's no-grad forward on a hint revealing the gold answer. Online SDPG runs it on the student's rollouts; self-distillation runs the same term on a dataset's fixed responses. Knobs: SDPGArguments.
RolloutOne sampled episode: the policy generates against an environment (single- or multi-turn) and the trajectory comes back with its reward. Collected by Ray actors in async GRPO, by direct server calls in online GRPO.
Masked rowAn episode the rollout infrastructure errored on, or one the environment marked episode_invalid (its reward carries no signal: a grading or sandbox outage, an inconclusive code grade, a failed scorer, a null answer). It still runs the forward so every rank's collectives match, but it is dropped from the group baseline and its tokens are masked out of the loss.
R3 (routing replay)MoE rollout↔training routing alignment: the rollout engine records which experts each sampled token was routed to (--enable-return-routed-experts), and training replays those choices in the no-grad pass and the update (routing_replay: rollout, experimental), which narrows the trainer-versus-engine gap without closing it. See Routing replay.
Forwarding rankThe one training rank that holds the rollout-server clients and sends during weight sync (global main; TP-rank 0 under TP). Every rank still participates in the weight gathers — only the sending is single-rank.
MoEMixture-of-Experts: a router sends each token to top_k of num_experts experts. The sparse side EP distributes; the dense (attention) side runs under FSDP/TP/CP. See Models — MoE.
MTP headMulti-token prediction: the tail a release ships for speculative decoding, drafting the token after next. mtp_loss_weight trains it under SFT, with mtp_loss_type and mtp_train_scope. See MTP heads.
Router balancingKeeps expert usage even, via moe_balancing: aux_loss (the model's native auxiliary loss), bias_update (DeepSeek-V3 auxiliary-loss-free bias landing in the family's own checkpoint slot, step size router_balancing_rate), or bias_update_transient (the same update held trainer-only, on families with no exportable slot — exports serve without it). auto picks bias_update where the bias lands in exported state, none + warning where only a transient bias would be possible, aux_loss where the forward takes output_router_logits, none + warning on a MoE with no route, and none for dense. Per-family resolution: MoE balancing modes.
NVL72 / MNNVLGB200/GB300 NVL72 packs 72 GPUs into one rack-wide NVLink (MNNVL) domain across ~18 OS nodes. The NVLink boundary is the rack, not the OS node, so nvlink_domain_size (not gpus_per_node) is the locality unit for node-local EP/CP/TP. Needs Fabric Manager, IMEX, and NCCL ≥ 2.25.2. See Multi-Node.

Source on GitHubUpdated 18 November 2026