# Glossary

Parallelism, kernel, optimizer, and method terms. Sizes and flags anchor to the
config field in `ParallelismConfig` (`halo/distributed/parallelism_config.py`) or
the YAML key that sets it. Legal combinations:
[parallelism mode matrix](https://whitecircle.com/halo/docs/agent-docs/parallelism.md). Per-trainer support:
[Trainer Compatibility](https://whitecircle.com/halo/docs/agent-docs/reference/trainer-architecture.md#trainer-compatibility).

## Parallelism

| Term | Definition |
|---|---|
| **Data Parallel (DP)** | Each rank holds a model replica and processes a distinct batch; gradients all-reduce across ranks. `data_parallel_size` is the number of distinct batches ([Stacking the dimensions](https://whitecircle.com/halo/docs/agent-docs/parallelism.md#stacking-the-dimensions)). PP, TP, CP, and ETP reduce it — EP does not. See [Data Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/data-parallelism.md). |
| **FSDP2** | The DP implementation for all `torchrun` modes: `fully_shard` with `reshard_after_forward=False` (SHARD_GRAD_OP behavior — params stay unsharded between forward and backward, faster, higher peak memory). EP modules are excluded via `ignored_params` when `ep_group_size > 1`; at `1` the default `fsdp_shard_ep1_experts` lets FSDP2 shard them. Set `fsdp_reshard_after_forward: true` (rejected under any `ep_group_size > 1` — EP and pure ETP alike — under TP when `data_parallel_size > 1`, and under PP) for FULL_SHARD (lower peak memory). See [Data Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/data-parallelism.md). |
| **Expert Parallel (EP)** | Shards a MoE model's experts across `ep_size` GPUs and routes each token to its experts with an all-to-all (DeepEP dispatch → expert compute → combine). EP is orthogonal to DP — every rank still gets a different batch, so EP alone does not reduce `data_parallel_size`. Requires DeepEP (no NCCL fallback). See [Expert Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/expert-parallelism.md). |
| **Context Parallel (CP)** | Splits one long sequence across `cp_size` GPUs and reconstructs attention with all-to-all (Ulysses). Each CP group processes one batch, so CP reduces `data_parallel_size`. CP must be NVLink-local. Incompatible with TP and ETP. Runs only under trainers that declare `SUPPORTS_CP`: SFT, SMPO and offline GRPO (full fine-tuning). See [Context Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/context-parallelism.md). |
| **Tensor Parallel (TP)** | Shards large weight matrices across `tp_size` GPUs using DTensor. Reduces `data_parallel_size` by `tp_size`. Dense models use their native `base_model_tp_plan` (`tp_plan="auto"`, shards attention and MLP); MoE models use the Halo *selective-TP* path (attention sharded, experts left to EP, embeddings and `lm_head` replicated). TP must be NVLink-local. See [Tensor Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/tensor-parallelism.md). |
| **Expert Tensor Parallel (ETP)** | Shards each expert's FFN weights across `expert_tp_size` GPUs — independent of `tp_size`, MoE-only, NVLink-local. Pure ETP (`ep_size=1`) replicates experts and shards their FFN; EP+ETP (`ep_size>1` and `expert_tp_size>1`) distributes experts *and* shards each one. The expert-TP reduce runs in token space, outside the DeepEP dispatch→combine span. Mutually exclusive with attention TP and with CP. See [Expert Tensor Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/expert-tensor-parallelism.md). |
| **`ep_size`** | Expert distribution size — how many distinct expert subsets exist. The all-to-all routes tokens across `ep_size` ranks. Set with `--expert_parallel_size`. |
| **`expert_tp_size`** | ETP size — how many ranks each expert's FFN is sharded over. Set with `--expert_tensor_parallel_size`. |
| **ZeRO-2 / ZeRO-3** | The two FSDP2 sharding depths: ZeRO-2 (`fsdp_reshard_after_forward: false`, the default) keeps parameters unsharded between forward and backward — faster, higher peak memory; ZeRO-3 (`true`) reshards after the forward for lower peak memory. See [Data Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/data-parallelism.md#zero-2-vs-zero-3-reshard_after_forward). |
| **`ep_group_size`** | The full EP process-group size: `ep_size × expert_tp_size`. Inside it, the DeepEP dispatch/combine runs over `ep_size`-rank dispatch groups and the expert-FFN reduce over `expert_tp_size`-rank groups ([layout](https://whitecircle.com/halo/docs/agent-docs/parallelism/expert-tensor-parallelism.md#process-groups-epetp-combo)). The whole group is what must fit the NVLink domain (node scope) or world (global scope). Computed; not set directly. |
| **`ep_scope`** | Where EP groups live: `node` (within one NVLink domain, all communication over NVLink) or `global` (spanning domains, all-to-all over RDMA). `auto` resolves to `node` when `ep_group_size ≤ nvlink_domain_size`, else `global`. EP+CP requires node scope with `ep_group_size == nvlink_domain_size`; an EP group larger than one NVLink domain needs `global`, while node-local groups replicated across nodes stay `node`. |
| **`nvlink_domain_size`** | The locality unit for "node-local" EP/CP/TP grouping — the number of GPUs reachable over NVLink. Defaults to `gpus_per_node` (one OS node on a standard cluster). On GB200/GB300 NVL72 the NVLink fabric spans the rack (up to 72 GPUs across ~18 OS nodes), so set `NVLINK_DOMAIN_SIZE` to use NVLink-wide parallelism. See [Multi-Node](https://whitecircle.com/halo/docs/agent-docs/parallelism/multi-node.md). |
| **Pipeline Parallel (PP)** | Splits the layer stack into `pp_size` sequential stages, each a `torchrun` rank group; only P2P boundary activations cross a stage. Outermost axis — it carves the world first (`stage_world_size = world_size / pp_size`). See [Pipeline Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md). |

Which axis combinations may run is an **allowlist** (`SUPPORTED_AXIS_SETS` in
`parallelism_rules.py`), checked before any rank math — anything unlisted is
rejected by default. Tabulated in the
[parallelism matrix](https://whitecircle.com/halo/docs/agent-docs/parallelism.md).

## Kernels and performance

| Term | Definition |
|---|---|
| **Flash Attention 2 (FA2)** | The fused attention kernel present on every image (`flash-attn` 2.x). The portable fallback: auto-detection picks it where FA4/FA3 are unavailable, and per-family rules route to it what FA4 cannot train. See [Flash Attention](https://whitecircle.com/halo/docs/agent-docs/optimization/flash-attention.md). |
| **Flash Attention 3 (FA3)** | Hopper-only (SM90) attention kernel, built from source (`flash-attention/hopper/`). On Hopper the image keeps FA2 + FA3. See [Flash Attention](https://whitecircle.com/halo/docs/agent-docs/optimization/flash-attention.md). |
| **Flash Attention 4 (FA4)** | Blackwell CuTe-DSL kernel (`flash-attn-4[cu13]`, the `flash_attn.cute` submodule), co-installed with FA2. Faster than FA2 on the attention kernel in isolation; the end-to-end MoE training-step delta is much smaller (attention is a minority of MoE step FLOPs). Auto-selected on SM100+, so it is the production default on Blackwell. See [Flash Attention](https://whitecircle.com/halo/docs/agent-docs/optimization/flash-attention.md). |
| **Grouped GEMM** | `torch.nn.functional.grouped_mm` batches the per-expert matmuls of a MoE layer into one kernel launch. The default on SM90+ (`use_grouped_gemm: true`); the end-to-end win scales with local experts per rank (`num_experts/ep_size`) and is smallest at high EP (few local experts), where the per-expert loop is still slower at every measured shape. Works EP-distributed or dense-style (`ep_size=1`). See [Grouped GEMM](https://whitecircle.com/halo/docs/agent-docs/optimization/grouped-gemm.md). |
| **Liger kernels** | Fused Triton kernels for cross-entropy, SwiGLU, RMSNorm and RoPE (`use_liger_kernel: true`, a toolkit default). Qwen3 MoE, GLM-4.7-Flash, GPT-OSS and Gemma 4 fill the RMSNorm role with torch's fused `F.rms_norm` instead, under the same `rms_norm` flag. FLCE (fused linear cross-entropy) fuses the `lm_head` matmul with the loss, removing the large logits tensor — the biggest memory win for high-vocab models. See [Liger Kernels](https://whitecircle.com/halo/docs/agent-docs/optimization/liger-kernels.md). |
| **DeepEP** | The expert-parallel communication library — the all-to-all dispatch/combine engine behind EP. The toolkit uses **DeepEP V2** (`deep_ep.ElasticBuffer` — the NCCL **Gin** (RDMA) backend inter-node, a non-Gin NVLink path intra-node). Required for EP (no NCCL fallback). See [DeepEP](https://whitecircle.com/halo/docs/agent-docs/infrastructure/deepep.md). |
| **tokens/s/GPU** | The headline throughput metric, hardware- and sparsity-independent: the cluster's unique tokens per second over the world size, so ranks sharing a batch (TP, expert-TP, CP, a pipeline chain) count it once. Use it to compare runs because a utilization percentage is computed against a specific GPU's peak FLOPS (the same achieved TFLOPS reads ~2.3× lower against a B300's 2250 TFLOPS peak than against an H200's 989.5). See [Throughput Benchmarks](https://whitecircle.com/halo/docs/agent-docs/optimization/throughput-benchmarks.md). |
| **S-MFU** | Sparsity-aware Model FLOPS Utilization: the roofline fraction a step achieves, scaling the *expert* FLOP term by `(top_k / num_experts) × ep_size` so it does not credit experts that never fired (a plain MFU over-counts for MoE). A diagnostic, not the headline — it is GPU-peak-relative, so compare tok/s/GPU or achieved TFLOPS across hardware. Full formula: [Throughput Benchmarks](https://whitecircle.com/halo/docs/agent-docs/optimization/throughput-benchmarks.md); roofline background: [GPU Training Theory](https://whitecircle.com/halo/docs/agent-docs/reference/gpu-training-theory.md#mfu-and-why-moe-complicates-it). |
| **Gradient checkpointing (GC)** | Recompute activations in the backward pass instead of storing them — trades ~20–30% wall-clock for memory. Turn it off when the batch fits unchecked (a free speedup). Zaya rejects GC on every path (a toolkit patch clears its `supports_gradient_checkpointing`). See [GPU Training Theory](https://whitecircle.com/halo/docs/agent-docs/reference/gpu-training-theory.md). |
| **Padding-free / packing** | Two ways to remove padding waste. Packing concatenates short examples into full-length sequences; the padding-free collator passes a flat `position_ids` + `cu_seqlens` batch with no pad tokens, so attention never attends across example boundaries. See [Padding-Free Collator](https://whitecircle.com/halo/docs/agent-docs/optimization/padding-free-collator.md). |

## Optimizers and precision

| Term | Definition |
|---|---|
| **BF16 mixed precision** | `bf16: true` (a toolkit default): params and compute in bf16. Long-sequence forward keeps outputs in bf16 (the fp32 output conversion is disabled) to avoid OOM on the logits. See [BF16 Optimizer](https://whitecircle.com/halo/docs/agent-docs/optimization/bf16-optimizer.md). |
| **AdamWBF16** | BF16 optimizer auto-enabled under bf16 on FSDP/EP/TP (6 bytes/param). Stochastic rounding on both the weight write and the `exp_avg_sq` store (nearest rounding biases the always-positive second moment by tens of percent, moving the effective LR); `exp_avg` stays nearest. Not auto-enabled under replicated DDP, which sits outside the validated SR matrix — `bf16_optimizer=True` opts in. See [BF16 Optimizer](https://whitecircle.com/halo/docs/agent-docs/optimization/bf16-optimizer.md). |
| **Stochastic rounding (SR)** | Round up or down with probability set by the distance to each bf16 grid point, so small updates below the bf16 ULP land in expectation instead of being dropped. Used in AdamWBF16 and in Muon's bf16 weight write. See [BF16 Optimizer](https://whitecircle.com/halo/docs/agent-docs/optimization/bf16-optimizer.md). |
| **`fp32_grad_reduce`** | Reduce gradients across ranks in fp32 while keeping bf16 storage (still 6 B/param). Covers every reduction the mixin owns: FSDP2's `reduce_dtype`, the EP router/expert grad sync, the TP replicated-grad sync and the QLoRA adapter all-reduce; default off. Distinct from `fp32_non_ep_params` (12 B/param fp32 storage), which implies it for the FSDP2 reduce only. See [BF16 Optimizer](https://whitecircle.com/halo/docs/agent-docs/optimization/bf16-optimizer.md). |
| **Muon** | Momentum optimizer with a Newton–Schulz orthogonalization of the update; the bf16 weight write uses SR (updates near `lr` are otherwise below the bf16 ULP). See [Muon Optimizer](https://whitecircle.com/halo/docs/agent-docs/optimization/muon-optimizer.md). |
| **FlashAdamW** | AdamW variant with quantized optimizer states — lower optimizer-state memory at a per-step state quant/dequant cost. Optional uv extra (`flash-optimizers`). See [FlashAdamW](https://whitecircle.com/halo/docs/agent-docs/optimization/flash-adamw.md). |
| **Low precision (fp8/fp4/mxfp4)** | Mixed-precision *compute*: bf16/fp32 master weights and checkpoint unchanged, low-precision GEMM operands. `lowp_precision: bf16 \| fp8 \| fp4 \| mxfp4` (`bf16` = off; `fp4` = nvfp4, most accurate). The default backend is *simulated* (block-scale fake-quant — correct numerics, the QAT oracle, not faster); native DeepGEMM is opt-in (`HALO_DEEPGEMM_NATIVE=1`). **bf16 is the production default**: no fp8/fp4 path beats it (simulated mxfp4 comes closest, 0.93–0.99× bf16 on 8× H200). See [Low-Precision MoE Kernels](https://whitecircle.com/halo/docs/agent-docs/optimization/low-precision-moe-kernels.md). |
| **DeepGEMM** | The native fp8 and FP8×MXFP4 grouped-GEMM kernels with on-device `m_indices`, behind `HALO_DEEPGEMM_NATIVE=1`. Never auto-routed; Halo's adapter runs whole steps at 0.37–0.54× bf16 at ep8. See [Low-Precision MoE Kernels](https://whitecircle.com/halo/docs/agent-docs/optimization/low-precision-moe-kernels.md). |

## Methods and infrastructure

| Term | Definition |
|---|---|
| **SFT** | Supervised fine-tuning on `{"prompt": [...]}` conversations (`DistributedSFTTrainer`). See [SFT](https://whitecircle.com/halo/docs/agent-docs/training-methods/sft.md). |
| **DPO** | Direct Preference Optimization on `{prompt, chosen, rejected}` triples with a reference model (`DistributedDPOTrainer`). See [DPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/preference/dpo.md). |
| **SMPO** | Smooth Margin PO — reference-free preference optimization with a target margin (`SMPOTrainer`). See [SMPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/preference/smpo.md). |
| **Offline GRPO** | Group-relative PO with pre-computed completions and rewards (`{prompt, completions, rewards}`); no online sampling (`OfflineGRPOTrainer`). See [Offline GRPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/offline.md). |
| **Online GRPO** | GRPO with live vLLM generation and NCCL weight sync (`DistributedOnlineGRPOTrainer`). See [Online GRPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/online.md). |
| **Async GRPO with Environments** | Multi-turn environment RL — async Ray-actor rollouts, rollout-server HTTP generation, NCCL weight sync (`DistributedAsyncEnvironmentalGRPOTrainer`). See [Async GRPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo.md). |
| **OPD** | On-policy distillation, named after the SDPG paper ([arXiv:2606.04036](https://arxiv.org/abs/2606.04036)): the term `beta(k)·L_OPD` both SDPG arms add to their objective, pulling the student's next-token distribution toward the same model's no-grad forward on a hint revealing the gold answer. [Online SDPG](https://whitecircle.com/halo/docs/agent-docs/training-methods/distillation/online-sdpg.md) runs it on the student's rollouts; [self-distillation](https://whitecircle.com/halo/docs/agent-docs/training-methods/distillation/self-distillation.md) runs the same term on a dataset's fixed responses. Knobs: [SDPGArguments](https://whitecircle.com/halo/docs/agent-docs/reference/configuration-reference.md#sdpgarguments). |
| **Rollout** | One sampled episode: the policy generates against an environment (single- or multi-turn) and the trajectory comes back with its reward. Collected by Ray actors in async GRPO, by direct server calls in online GRPO. |
| **Masked row** | An episode the rollout infrastructure errored on, or one the environment marked `episode_invalid` (its reward carries no signal: a grading or sandbox outage, an inconclusive code grade, a failed scorer, a null `answer`). It still runs the forward so every rank's collectives match, but it is dropped from the group baseline and its tokens are masked out of the loss. |
| **R3 (routing replay)** | MoE rollout↔training routing alignment: the rollout engine records which experts each sampled token was routed to (`--enable-return-routed-experts`), and training replays those choices in the no-grad pass and the update (`routing_replay: rollout`, experimental), which narrows the trainer-versus-engine gap without closing it. See [Routing replay](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/objective.md#routing-replay). |
| **Forwarding rank** | The one training rank that holds the rollout-server clients and sends during weight sync (global main; TP-rank 0 under TP). Every rank still participates in the weight gathers — only the sending is single-rank. |
| **MoE** | Mixture-of-Experts: a router sends each token to `top_k` of `num_experts` experts. The sparse side EP distributes; the dense (attention) side runs under FSDP/TP/CP. See [Models — MoE](https://whitecircle.com/halo/docs/agent-docs/models.md). |
| **MTP head** | Multi-token prediction: the tail a release ships for speculative decoding, drafting the token after next. `mtp_loss_weight` trains it under SFT, with `mtp_loss_type` and `mtp_train_scope`. See [MTP heads](https://whitecircle.com/halo/docs/agent-docs/optimization/mtp-heads.md). |
| **Router balancing** | Keeps expert usage even, via `moe_balancing`: `aux_loss` (the model's native auxiliary loss), `bias_update` (DeepSeek-V3 auxiliary-loss-free bias landing in the family's own checkpoint slot, step size `router_balancing_rate`), or `bias_update_transient` (the same update held trainer-only, on families with no exportable slot — exports serve without it). `auto` picks `bias_update` where the bias lands in exported state, `none` + warning where only a transient bias would be possible, `aux_loss` where the forward takes `output_router_logits`, `none` + warning on a MoE with no route, and `none` for dense. Per-family resolution: [MoE balancing modes](https://whitecircle.com/halo/docs/agent-docs/training-methods/callbacks.md#moe-balancing-modes). |
| **NVL72 / MNNVL** | GB200/GB300 NVL72 packs 72 GPUs into one rack-wide NVLink (MNNVL) domain across ~18 OS nodes. The NVLink boundary is the rack, not the OS node, so `nvlink_domain_size` (not `gpus_per_node`) is the locality unit for node-local EP/CP/TP. Needs Fabric Manager, IMEX, and NCCL ≥ 2.25.2. See [Multi-Node](https://whitecircle.com/halo/docs/agent-docs/parallelism/multi-node.md). |
