# Troubleshooting

Symptom→fix lookup for OOM, NCCL hangs, DeepEP faults, parallelism-config rejections, NaN loss, and
attention-backend mismatches. Run `/debug` to route a live symptom to the exact helper.
A step that is slow rather than broken is a bottleneck question: read power, not util %
([GPU Training Theory §11](https://whitecircle.com/halo/docs/agent-docs/reference/gpu-training-theory.md#watch-power-not-utilization)).

The trainer-side weight-sync errors are rows below; server-side symptoms are on
[Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#troubleshooting), and Ray-actor, rollout and
environment symptoms on [Async GRPO → Troubleshooting](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/troubleshooting.md).

## First things to check

Most "it won't start" reports are environment, not code:

| Check | Why it bites |
|---|---|
| **Right image for the GPU.** `halo:blackwell` on B200 (SM100) / B300 (SM103), `halo:hopper` on H100/H200 (SM90). | The host has **no usable Python** — PyTorch, DeepEP, and Flash Attention live only inside the image. FA3 is Hopper-only, FA4 Blackwell-only; the wrong image gives import errors or kernel hangs. |
| **`--env-file .env` passed to `docker run`.** | The code does not auto-load `.env`. Without it, `WANDB_API_KEY` / `HF_TOKEN` / `AWS_DEFAULT_REGION` are unset, so a run that reports to W&B, pulls a gated repo, or reads an `s3://` dataset fails at that step — each needs your own account. Start from `.env.example`. |
| **No `poetry run` / `uv run` prefix.** | The image installs deps into the system interpreter, so `python` / `torchrun` / `accelerate` / `pytest` run directly. |
| **The scratch mount is actually large.** | The root filesystem is often small and shares the Docker image pool; a path named `/mnt` is not guaranteed to be a separate volume. Check `findmnt` / `df -h`, then point `HF_DATASETS_CACHE`, `TMPDIR`, `HALO_DATA_ROOT`, and logs at whichever mount has capacity. |

## Symptom → cause → fix

| Symptom | Cause | Fix | Deep page |
|---|---|---|---|
| `torch.OutOfMemoryError: CUDA out of memory` | Activations + weights + optimizer states exceed VRAM. | Gradient checkpointing on, lower `per_device_train_batch_size` / `max_length`, or move to a sharded mode. See [OOM](#oom). | [Debugging §3](https://whitecircle.com/halo/docs/agent-docs/reference/debugging.md#3-gpu-memory-profiling) |
| `CUDA out of memory` after step 2 under `gradient_checkpointing_plan: auto` | The measured step did not hold the run's largest footprint: a pack fuller than any the step measured, EP routing drift past the 75% budget, or fragmentation growing past the measured ratio. | Relaunch into the same `output_dir` (resume or not): the unfinished plan's record makes the next plan checkpoint at least one step more. A timestamped `output_dir` gets a new directory, so no floor: pin it. `gradient_checkpointing_plan: off` keeps full checkpointing. | [GradientCheckpointingPlanCallback](https://whitecircle.com/halo/docs/agent-docs/training-methods/callbacks.md#gradientcheckpointingplancallback) |
| **`gradient_checkpointing_plan: auto is refused for this run: …`** at startup | The plan relaxes per-layer checkpointing from one measured step of packed rows; the message lists every reason this run cannot take one: pipeline parallelism, `torch_compile`, `auto_find_batch_size`, a trainer that generates rollouts, unpacked or multimodal rows, activation offload, an explicit `every_n_layers`, model-level checkpointing, or checkpointing off. | Remove what the message names, or set `gradient_checkpointing_plan: off`. | [GradientCheckpointingPlanCallback](https://whitecircle.com/halo/docs/agent-docs/training-methods/callbacks.md#gradientcheckpointingplancallback) |
| **`gradient_checkpointing_plan: auto measures CUDA allocator readings and needs a GPU`**, or **`The checkpointing plan record names every_n_layers=…`** at startup | The plan reads CUDA allocator stats, so a CPU run has nothing to measure. The record in `output_dir` names a policy the plan does not choose from at this layer count: another run's file, or an edited one. | Run on a GPU, or set `gradient_checkpointing_plan: off`. Remove a stale `_halo_run_plan.json` from `output_dir`. | [GradientCheckpointingPlanCallback](https://whitecircle.com/halo/docs/agent-docs/training-methods/callbacks.md#gradientcheckpointingplancallback) |
| Job **hangs at step 0** or first cross-rank sync, no error, then a watchdog timeout | Collective mismatch — one rank issued a collective another never reached (divergent control flow, a straggler still in the dataloader). | Dump every rank's stack; the rank *not* in a collective is the culprit. See [NCCL hang](#nccl-timeout--hang). | [Debugging §4](https://whitecircle.com/halo/docs/agent-docs/reference/debugging.md#4-diagnosing-multi-node-hangs) |
| Every GPU at **100% utilization but idle power**, all ranks spinning a CPU core, no NCCL error | A per-rank backward graph: a masked or empty row left disconnected from the loss let autograd prune that row's backward on one rank, so its FSDP2/EP collective never fired. | Keep every row connected to the loss (a value masked downstream is fine). Confirm with the NCCL flight recorder — mismatched collectives at one `collective_seq_id`. | [Debugging §4](https://whitecircle.com/halo/docs/agent-docs/reference/debugging.md#4-diagnosing-multi-node-hangs) |
| **`trains or takes a gradient under reentrant gradient checkpointing, and takes no floating tensor`** at a training forward | A checkpointed layer holding a trainable parameter, or taking a gradient in another argument, gets no float positionally and no tensor as its first argument, so a reentrant checkpoint could give its outputs no gradient. Every family on the roster passes one. | Pass the hidden state positionally in the layer's caller, or train without gradient checkpointing. | [Data Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/data-parallelism.md#tower-blocks-under-reentrant-checkpointing) |
| `None of the inputs have requires_grad=True. Gradients will be None` (torch `UserWarning`) on an image batch | A frozen tower's checkpointed block under reentrant checkpointing, a whole frozen tower or the blocks before the first trained one, with nothing to train. | None needed. A block holding an adapter or a trained parameter does not warn. | [Data Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/data-parallelism.md#tower-blocks-under-reentrant-checkpointing) |
| **`This rank's batch holds no image while the vision tower trains under data parallelism`** at a training forward | Under `accelerate launch`, or with a frozen input embedding beside a trainable tower, gradient checkpointing off, and a trained decoder layer or ZeRO-3 (`unfreeze_layers_patterns` matching the tower but not the embedding, or an adapter reaching the tower, which the message names), a text rank's reduces cannot pair with an image rank's, and the step would hang to the collective timeout. | Launch with `torchrun`; or train the embedding, or turn checkpointing on; or drop the text-only rows. A fine-tune can also set `freeze_vision_tower: true` or train adapters; an adapter run, which refuses that knob, keeps `lora_target_modules` off the tower. | [Data Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/data-parallelism.md#shard-groups) |
| `ValueError` at startup naming `expert_parallel_size=N on a single M-GPU NVLink domain forms K concurrent >2-rank DeepEP dispatch groups` (after the `parallelism config failed on … First (rank 0):` prefix under torchrun) | Racy single-domain multi-group EP: one NVLink domain, `ep_size > 2`, and `ep_group_size` smaller than the domain (`ep_size=4` on 8 GPUs). The combine barriers race FSDP2's DP-wide collectives, so the shape is refused at config time. | `ep_size=2`, `ep_size` = the domain, or raise `ep_size × expert_tp_size` to the domain (`ep4 + etp2` on 8). Attention TP leaves `ep_group_size` unchanged, so `ep4 + tp2` is refused too. | [Expert Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/expert-parallelism.md#single-domain-multi-group-ep-races-and-hangs) |
| **`EP capacity dedup: a MoE layer dispatched N tokens/rank, over the capacity C cached at forward generation G`** on one rank, its peers stuck in the dispatch | The forward reuses the first MoE layer's all-reduced capacity for every later layer, and a layer outgrew it: either the model's later MoE layers dispatch more tokens than its first, or this forward entered the backbone directly and opened no capacity scope of its own. The raising rank leaves its peers in the dispatch collective until `HALO_DEEPEP_GPU_TIMEOUT_SECONDS`, so it reads as a hang plus one traceback. | A caller that peels the backbone off the wrapper calls `bump_forward_generation()` once per forward (the chunked log-prob path does). Otherwise `HALO_EP_CAPACITY_DEDUP=0` sizes every layer with its own all-reduce — at one arena per layer. | [DeepEP](https://whitecircle.com/halo/docs/agent-docs/infrastructure/deepep.md) |
| **`An expert id outside the local experts at ep_size 1, where the routing hands over only local ids`** in a `device-side assert triggered` (`ep_size: 1`, grouped GEMM) | At `ep_size` 1 nothing is dispatched, so every expert id the routing hands the expert compute must lie in `[0, num_experts)`: the routing sort counts the ids in that range and asserts they cover every slot, since an ID outside would misalign the expert groups. A `-1` is not padding here; only DeepEP pads with it. The ID came from the family's routing (a custom router, a changed top-k) or a [routing-replay](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/objective.md#routing-replay) mask row; engine masks are range-checked when they are assembled. | Not a DeepEP fault and not a config problem: report it with the model, the parallelism shape and whether routing replay was on. | [Grouped GEMM](https://whitecircle.com/halo/docs/agent-docs/optimization/grouped-gemm.md) |
| `ImportError` / `NVSHMEM` / undefined symbol on `import deep_ep` | Wrong NVSHMEM package, or DeepEP used without a GPU / the right image. | Run inside the image ([DeepEP faults](#deepep-build--runtime-faults)). | [DeepEP](https://whitecircle.com/halo/docs/agent-docs/infrastructure/deepep.md) |
| **`FlashAttention-4's head_dim-256 kernel is not validated with a deterministic backward`** at load — a `full_determinism` run of Qwen3.5 / Qwen3.6 / Qwen3-Next or any other head_dim-256 model FA4 otherwise trains | `full_determinism` sets `FLASH_ATTENTION_DETERMINISTIC=1`, which transformers forwards to the dedicated head_dim-256 backward, whose deterministic mode is not validated here. | Auto-handled: `resolve_attn_implementation` demotes FA4 to SDPA (to FA2 under Context Parallelism, the kernel Ulysses runs then) and says so; SFT `packing` and `padding_free` rows on a GatedDeltaNet decoder load FA2 instead, an explicit `attn_implementation: flash_attention_4` included; elsewhere an explicit FA4 is demoted to SDPA too. Drop `full_determinism` to train on FA4. | [Flash Attention](https://whitecircle.com/halo/docs/agent-docs/optimization/flash-attention.md#model-specific-handling) |
| **`FlashAttention-4 trains head_dim-256 attention on a dedicated backward kernel: it takes no attention logit softcapping`** (or `no sliding-window attention`, or `multi-head latent attention at this width stays on FA2 as a precaution`) at load — Gemma 2 / Gemma 3 at head_dim 256, GLM-4.7-Flash | FA4 trains a 256-wide query/key head only through its dedicated backward kernel, which asserts on a softcap and on a window; GLM-4.7-Flash's latent attention is kept on FA2 as a precaution. | Auto-handled: the resolver demotes them to FA2, which takes all three; SDPA is never chosen for a softcapped model, since it drops the softcap. Nothing to do; a host with no FA2 build refuses instead, naming `flex_attention` and `eager`. | [Flash Attention](https://whitecircle.com/halo/docs/agent-docs/optimization/flash-attention.md#model-specific-handling) |
| **`FlashAttention-4's varlen kernels are not validated with attention logit softcapping`** at load (Gemma 2), or **`attn_implementation='flash_attention_4' cannot train this model on packed, padding-free or padded rows`** (Gemma 2 27B; the resolver moves a head_dim-256 pin to FA2) | FA4's varlen kernels are not validated with a softcap; packed, padding-free and padded rows (more than one per batch), training or evaluation, all reach them. | Unset: auto-handled, the run takes FA2. Pinned FA4: unset it or set `flash_attention_2`, or train unpacked at `per_device_train_batch_size: 1` (and `per_device_eval_batch_size: 1` where the run evaluates). | [Flash Attention](https://whitecircle.com/halo/docs/agent-docs/optimization/flash-attention.md#model-specific-handling) |
| **`A softcapped attention call reached FlashAttention-4's varlen kernels`** in the first forward (Gemma 2 on a pinned `flash_attention_4`, a padded script such as DPO, KTO, SMPO, reward, classification or GRPO) | The batch unpads to FA4's varlen entry point, which refuses a softcap (not validated there). The scripts outside SFT, self-distillation, teacher distillation and embedding read no row shape before the load, so the entry point refuses the call itself. | Leave `attn_implementation` unset (the padded scripts then take FA2) or set `flash_attention_2`. | [Flash Attention](https://whitecircle.com/halo/docs/agent-docs/optimization/flash-attention.md#model-specific-handling) |
| **`full_determinism: true cannot run with the sdpa_flex_sliding attention`** at trainer construction (Gemma 4 or Step-3.7 on SDPA, world size > 1) | HF's `Trainer` turns on deterministic-algorithms mode after the model loaded, and the masked layers' FlexAttention graphs were compiled at load without it. Dynamo guards on that mode, so each rank would recompile at its own first long masked call while its peers wait in the next collective (a DeepEP dispatch times out). | `HALO_FLEX_SLIDING=0` (plain SDPA), or drop `full_determinism`. | [Flash Attention](https://whitecircle.com/halo/docs/agent-docs/optimization/flash-attention.md#sliding-window-and-wide-head-layers-on-sdpa) |
| **`reset_sinks: true would neutralize the attention sinks of the GptOss teacher too`** at teacher distillation startup, before any weight loads, or, from a teacher built outside the script, **`The GptOss teacher was loaded with its attention sinks neutralized`** at trainer construction | `reset_sinks` applies to both models. The student trains away from neutralized sinks, but the frozen teacher never trains back: gpt-oss-20b without its sinks predicts ~6 nats/token (KL) from itself, so the student would fit a degenerate target. | Set `reset_sinks: false`: both models keep the sinks their checkpoints hold, frozen, on a sink-carrying backend (`flash_attention_4` on Blackwell; `flex_attention` or `eager`). A teacher built outside the script loads with `reset_sinks=False` through `load_frozen_auxiliary_model`, on the same backends. | [Teacher Distillation](https://whitecircle.com/halo/docs/agent-docs/training-methods/distillation/teacher-distillation.md#configuration) |
| **`This GptOss model runs attn_implementation='…', which drops its attention sinks, while the sinks are live`** at model load, in any loader (the reward-model scorer and a sentence-transformers backbone included) | The built model's backend takes no sink column (FA2, SDPA, an FA3 build without `s_aux`) while its sinks are live, so every logprob would shift by nats against the served model. The check reads the built model, so it also catches a backend no resolver chose. | Load on a sink-carrying backend (`flash_attention_4` on Blackwell; `flex_attention` or `eager`), or neutralize the sinks (`reset_sinks: true` in a training run). | [GPT-OSS → Attention sinks](https://whitecircle.com/halo/docs/agent-docs/models/gpt-oss.md#attention-sinks) |
| **`full_determinism cannot run over an EP group spanning NVLink domains`** at startup, before the load | `full_determinism` on an EP group spanning NVLink domains. DeepEP's cross-domain (hybrid RDMA) dispatch has no deterministic mode: it places tokens in atomic claim order, so expert weight gradients change from step to step. | `ep_scope: node` with DP across domains (on a multi-node NVLink fabric, `NVLINK_DOMAIN_SIZE` set to its size), or drop `full_determinism`. | [Expert Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/expert-parallelism.md#determinism) |
| **`full_determinism is set but FLASH_ATTENTION_DETERMINISTIC is not, so this row plan would pick …`** from a custom script, before any weight loads | The script built its config without `parse_script_args`, which turns transformers' deterministic flash switch on as it parses a `full_determinism` config. `plan_row_attention` reads that switch to pick the attention, as the loader does, so planning before it is set would pick a non-deterministic run's backend for kernels the trainer then runs deterministic. | Parse the config through `parse_script_args`, or call `enable_flash_attention_deterministic()` (`halo/models/attention/selection.py`) before planning. | [Configuration](https://whitecircle.com/halo/docs/agent-docs/reference/configuration-reference.md#parsing) |
| **Async GRPO with Environments: `episode/length_cutoff_in_call_turns` climbs** (turns cut while writing a tool call) | A turn reaches its token cap inside a tool call: either the call outgrew the turn's own cap (`rollout_max_tokens`), or the engine closed the turn's reasoning at its cap and the model carried the thought on inside the call's argument, as comment lines of the program (`episode/thinking_cap_turns` shows it starting; the run's `judge` term handles reasoning carried into programs: [Code Contests](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/environments/tasks/code-contests/tools.md)). Both rollout drivers read the cap off `usage.completion_tokens`, so the turn is a length cut whatever finish the server reports: the salvaged call is never run, the turn takes the nudge-and-retry path, and its fragment trains only on a negative advantage ([Untrainable turns](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/objective.md#untrainable-turns)). | Under `rollout_max_answer_tokens` an in-call cut after a forced close is the bound at work: the call carried the reasoning on in its comments; raise the bound only when honest calls outgrow it. Otherwise raise the cap that binds: `rollout_max_tokens` (on vLLM, the answer room: `rollout_max_tokens` less the reasoning cap, or `rollout_max_answer_tokens` where smaller) when the call itself outgrows the turn, the reasoning cap (`rollout_max_thinking_tokens`, else `rollout_max_tokens` less `rollout_max_answer_tokens`) when it cuts the reasoning short. | [Native tool-use](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/environments/protocols/native-tool-use.md#reward) |
| **Async GRPO with Environments: `episode/empty_turns` climbs**, episodes loop to `max_turns` while `episode/length_cutoff_turns` stays near zero | The model closes its reasoning and stops with neither a tool call nor an answer, below its budget. Each such turn is nudged and retried like an engine cut, and at `max_length_cutoff_recoveries: null` an episode can spend every turn this way. | Cap the recoveries (`max_length_cutoff_recoveries`, one cap for cuts and empty turns) and price them (`length_cutoff_penalty`); the turn itself trains only on a negative advantage, and the price charges the episodes that recover. | [Native tool-use](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/environments/protocols/native-tool-use.md#reward) |
| **Async GRPO with Environments: completions fill with `Error: <tool>: unknown argument '<name>'; its arguments are …`** | The call named an argument the tool does not declare: a misspelled name (`<parameter=…>` in the Qwen XML format), or arguments the gpt-oss parser could not read, which it passes on as `'_raw'`. The call is refused before it runs, its error naming the declared arguments; no tool budget is spent, though the turn is and `tool_error_penalty` applies. | A few are the model correcting itself on the next turn. Many mark a policy drifting off its call format: check `entropy` and the trainer-vs-sampler gap (`sampling/logratio_mean`). | [Native tool-use](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/environments/protocols/native-tool-use.md#reward) |
| **Async GRPO with Environments: `a vLLM reasoning cap can be enforced but the importance-sampling correction is off`** at trainer construction | A vLLM reasoning cap can bind (`rollout_max_thinking_tokens`, or `rollout_max_tokens` less `rollout_max_answer_tokens`) and the marker resolves, but the IS correction is off; the reasoning closes the engine forces are kept out of the loss only through its ratio. | Turn `train_on_sampled_tokens` and `vllm_importance_sampling_correction` back on (both default on). | [Objective](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/objective.md#importance-sampling-correction) |
| **`multiple-choice answer '…' is a digit string`** from `exam_qa` | A digit string names different choices as a 0-based index and as a 1-based label (ARC's `answerKey`), so the grader refuses to guess. | Convert the answer to a choice letter or an int index when preparing the dataset. | [QA Benchmarks](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/environments/tasks/qa-benchmarks.md) |
| **NaN loss**, other models, mid-run | LR too high for bf16, or a precision/grad-reduce mismatch. | AdamWBF16 with stochastic rounding is auto-on under `bf16: true` (FSDP/EP/TP). Lower the LR; set `fp32_grad_reduce: true` for a tighter grad reduce. | [BF16 Optimizer](https://whitecircle.com/halo/docs/agent-docs/optimization/bf16-optimizer.md) |
| **`Optimizer step N: the global gradient norm is nan; the update is skipped`** (warning; the logs carry `nonfinite_steps_skipped`, and a logging window with no applied step logs `loss: nan`) | One step's gradients went non-finite. The step was not applied: weights and optimizer moments are those of step N-1, and training continues. | Nothing, if it stays rare. A recurring one points at a data or numerics fault in the forward or backward of those batches. | [Trainer Architecture](https://whitecircle.com/halo/docs/agent-docs/reference/trainer-architecture.md#non-finite-steps) |
| **`Optimizer steps A-B: K consecutive steps had a non-finite global gradient norm`**, on every rank | More than `max_consecutive_nonfinite_steps` non-finite steps in a row. None was applied. | Find the cause (the batches of those steps, or the model's numerics), then resume from the last checkpoint. | [Trainer Architecture](https://whitecircle.com/halo/docs/agent-docs/reference/trainer-architecture.md#non-finite-steps) |
| **`The finite-weights check at optimizer step N failed on … rank(s)`** naming parameters that `already hold NaN/inf before the update` | The weights went non-finite before this step's gradients did: corrupted outside the optimizer update, which a skip cannot repair. | Resume from the last checkpoint. If it recurs at the same step, the batches of that step trigger it. | [Trainer Architecture](https://whitecircle.com/halo/docs/agent-docs/reference/trainer-architecture.md#non-finite-steps) |
| **`FSDP2 post-backward left undone after complete_fsdp2_post_backward`** (`RuntimeError`) after a backward | A shard group still holds an unreduced gradient, or stays gathered, after the trainer completed the post-backward that reentrant checkpointing skips for a layer recomputed more than once per backward. Either a backward ran outside the trainer's `training_step`, or the FSDP2 internals the completion drives changed with the installed torch. | Run every backward through `training_step`. After a torch upgrade, run `tests/cpu/parallelism/test_fsdp2_post_backward_internals.py`, which pins every member the completion touches. | [Data Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/data-parallelism.md#layers-recomputed-more-than-once-per-backward) |
| **Per-token loss degrades sharply past ~2048 tokens**, or logprobs differ between the image and a host Python install | The NGC image defaults fp32 matmuls to TF32; TF32's 10-bit mantissa collapses adjacent RoPE token positions past 2048, corrupting the positional encoding wherever RoPE is a matmul: remote code such as Ling and Ring. transformers' own RoPE multiplies in the pinned release. | Auto-handled: `configure_float32_matmul_precision` pins fp32 matmuls to `highest` at model load and the image sets `TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=0`. Opt into TF32 with `HALO_FP32_MATMUL_PRECISION=high`. | [Configuration](https://whitecircle.com/halo/docs/agent-docs/reference/configuration-reference.md) |
| **`LIGER_KERNEL_IMPL_ROPE=<value> selects a Liger rope implementation the toolkit does not run`** (or the same for `LIGER_KERNEL_BACKEND_ROPE`) at startup | The toolkit pins Liger's RoPE to its Triton kernel: Liger's CuTe DSL RoPE returns wrong results under `CUTE_DSL_ENABLE_TVM_FFI`, which the images set for FA4. | Unset the variable; only `nvidia-triton` (or `triton`) is accepted. | [Liger Kernels](https://whitecircle.com/halo/docs/agent-docs/optimization/liger-kernels.md#kernel-implementation) |
| **`The Liger applier for <model_type> rebound torch.nn.LayerNorm to <class> for the whole process`** (`RuntimeError`) at startup, before the model builds | An upstream Liger applier left `torch.nn.LayerNorm` rebound after the dispatch forced its `layer_norm` flag off, so every LayerNorm built afterwards (a teacher, a reference, a vision tower) would run Liger's class. | Set the flag that installs it to `false` in `liger_kernel_config`, or set `use_liger_kernel: false`. | [Liger Kernels](https://whitecircle.com/halo/docs/agent-docs/optimization/liger-kernels.md#supported-models) |
| **`mean_token_accuracy` reads 0 and `entropy` ln(vocab)** in a training log, beside a sane `loss` | Liger's cross-entropy kernel wrote into fp32 logits the forward returned, before the metrics read them. | Read eval metrics that mask label -100; the loss and gradients are right. | [Liger Kernels](https://whitecircle.com/halo/docs/agent-docs/optimization/liger-kernels.md#known-issue-metrics-on-fp32-logits) |
| **OOM on Gemma 4 at long sequence**, or FA2 rejects the model | Gemma 4's global layers have `head_dim=512`, above every flash kernel's limit; cuDNN SDPA rejects it too and the math kernel OOMs on the full score matrix. | Automatic: the run resolves to SDPA and is built as `sdpa_flex_sliding`. Sliding layers run FlexAttention; global layers run matmul attention within a 2 GiB per-layer score budget (about 4,700 tokens at batch 1) and, past it, mem-efficient attention per document on a packed row and mem-efficient SDPA on any other. Without gradient checkpointing the matmul global layers hold about 7 GiB more than SDPA at 4,096 tokens; `HALO_FLEX_SLIDING=0` builds plain SDPA. Set `attn_implementation: sdpa` to skip the redirect warning. | [Gemma 4](https://whitecircle.com/halo/docs/agent-docs/models/gemma4.md) |
| **Hang inside `ptxas`** at build/first kernel, Hopper, building FA2 | FA2's split-K forward kernels hang `ptxas` under CUDA 13.2 on SM90. | The Hopper image strips every `flash_fwd_split_*` source, links 24 throw-stubs in their place and pins the dense and varlen forwards to one split, so no forward Halo runs reaches a stub. Use the prebuilt image; building FA2 for Hopper outside Docker needs the same two changes or CUDA 13.1. | [Flash Attention](https://whitecircle.com/halo/docs/agent-docs/optimization/flash-attention.md) |
| `CheckpointError` (recomputed tensor shape ≠ forward) under EP or any MoE with gradient checkpointing, often in GRPO | `use_reentrant=False` validates recompute shapes, and a recomputed router can flip a near-tie pick (the extra no-grad GRPO forwards make it likelier), changing the routing tensors' shapes. | Let the override stand: the trainer forces `use_reentrant=True` under EP, CP or any MoE (`resolve_use_reentrant`, `halo/trainers/parallelism/ep_layers.py`) with a warning at init, even over a configured `false`. Pipeline parallelism is the exception: it runs non-reentrant and refuses `use_reentrant: true`. | [Expert Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/expert-parallelism.md#gradient-checkpoint-dispatch-replay) |
| `gradient_checkpointing_enable` raises on Zaya | cuDNN Conv1d backward-recompute fault under CUDA 13.2; the toolkit clears the model's GC support flag so it fails at enable time rather than deep in the first backward. | Run Zaya as plain FSDP2 **without GC**, EP without GC, or EP+ETP without GC — the supported configs. TP and CP are unsupported for Zaya. | [Zaya](https://whitecircle.com/halo/docs/agent-docs/models/zaya.md) |
| `fp32_non_ep_params=True cannot combine with fsdp_shard_ep1_experts=True` at model load (MoE, `ep_size: 1`) | The upcast skips every EP-layer parameter, so FSDP-managed replicated experts would stay bf16 inside an fp32 shard group and FSDP2 asserts a uniform original parameter dtype at the first forward. Rejected off `config.json` instead, before the process groups and the meta shell. | Set `fsdp_shard_ep1_experts: false`, or drop `fp32_non_ep_params`. Under TP/CP the first remedy is itself refused, so dropping `fp32_non_ep_params` is the only one; the message names whichever applies. | [Bailing/Ling](https://whitecircle.com/halo/docs/agent-docs/models/bailing.md#ling-30) |
| `fp32_non_ep_params=True is not supported for gemma4/gemma4_text under Expert Parallelism` at model load (`ep_size > 1` only) | Gemma 4's norms re-emit activations at weight dtype, so the upcast feeds fp32 tokens into DeepEP's 2-byte transport. The gate turns that into a named refusal instead of a raw C++ assert at the first dispatch, after the whole multi-GPU load. | Train Gemma 4 experts-distributed in plain bf16; `fp32_experts` still applies. | [Gemma 4](https://whitecircle.com/halo/docs/agent-docs/models/gemma4.md) |
| `ValueError: ... must divide ...` / `Node-local EP group size (N) cannot exceed the NVLink domain (M)` at startup | `ParallelismConfig` validation rejected the layout. | Fix the degrees so they divide `world_size`; keep TP and node-local EP within one NVLink domain. See [Config rejections](#parallelism-config-rejections). | [Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism.md) |
| **`text_only_model=True cannot load model_type=…`** at load | The family has no text-only `*ForCausalLM` sibling: transformers maps none, or the class it maps is the conditional-generation one (Gemma 4). | Set `text_only_model: false` and load the multimodal class. | [Supported Models](https://whitecircle.com/halo/docs/agent-docs/models.md) |
| **`flash-attn is required for Context Parallelism`** at load | Ulysses attention calls a flash kernel directly, and neither the run's backend nor `flash_attention_2` imports. | Run in the Blackwell or Hopper image. | [Context Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/context-parallelism.md#requirements) |
| **`lowp_precision=… cannot run …: it contracts over …`** at load | The fake quant blocks both GEMM operands along the contraction axis, and this module's axis does not split into the format's blocks. | Lower the parallel size that shards the axis where the message names one, pick a `lowp_precision` whose block divides it, or set the opt-out flag the message names. | [Low-Precision Kernels](https://whitecircle.com/halo/docs/agent-docs/optimization/low-precision-moe-kernels.md#usage) |
| **`…: TP style … keeps a …-wide activation full width, which tensor_parallel_size=… does not divide`** at load, or **`… ran with its parameters FSDP2-sharded over …`** in a forward | A rowwise or gathering TP linear splits that width across the group, which needs an exact divisor. The second is a projection called around its FSDP unit's forward, with the `(dp, tp)` shard still registered, which would compute part of its output. | Pick a `tensor_parallel_size` that divides the width. Call the projection through the FSDP-wrapped module. | [Tensor Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/tensor-parallelism.md#local-shard-linears) |
| **`… holds an external table …`**, **`The checkpoint in … holds no keys that load into …`**, **`The … parts in … hold … rows of widths …`** or **`… has no table attached`** on a Qwen3.8-Flash-Next load or save | The n-gram table attaches from the local directory the weights were read from. A model loaded outside the toolkit's loaders whose config records a Hub id has no revision to pair the table with, a config object names no checkpoint, and a directory without the table's keys or sizes does not match the config. | Load through a Halo loader, or from a local directory that holds the table, with the `config.json` it was saved with. | [Qwen3.8-Flash-Next](https://whitecircle.com/halo/docs/agent-docs/models/qwen4_exp.md#n-gram-table) |
| **`A qwen4_exp row whose position_ids restart at 0 (a packed row) arrived without seq_idx`**, **`seq_idx marks different documents than the position_ids restarts of the same row`** or **`Tiled QSA …`** in a Qwen3.8-Flash-Next forward | The per-document forwards read documents from `seq_idx` and padding from a 2-D mask, so a row without them, or with dropout or interleaved padding, would read across documents. | Pack with the toolkit's collators, which emit `seq_idx` for the family. Set `attention_dropout` to 0, and pad on one side. | [Qwen3.8-Flash-Next](https://whitecircle.com/halo/docs/agent-docs/models/qwen4_exp.md#attention-and-rows) |
| **`The document-aware PLE conv reads the weights of …`**, **`Zaya's document-aware CCA reads … weights directly`** or **`Nemotron-H's Mamba mixer reads conv1d.weight directly`** at the first forward | These forwards read the conv's weights themselves, so an adapter on that conv would be bypassed. | Leave the conv (`conv1d`, or `conv_qk_depthwise` / `conv_qk_grouped`) out of `lora_target_modules`. | [Qwen3.8-Flash-Next](https://whitecircle.com/halo/docs/agent-docs/models/qwen4_exp.md#attention-and-rows), [Zaya](https://whitecircle.com/halo/docs/agent-docs/models/zaya.md), [Nemotron-H](https://whitecircle.com/halo/docs/agent-docs/models/nemotron-h.md#lora) |
| **`trust_remote_code loaded … through the release's own modeling file, which cannot train here`** at load | The remote composite (Nemotron-H's) drops what this toolkit's rows read on the way to its text tower. | `trust_remote_code: false`, which reads the same checkpoint through transformers' classes, or `text_only_model: true`. | [Nemotron-H](https://whitecircle.com/halo/docs/agent-docs/models/nemotron-h.md#checkpoints-and-serving) |
| **`A remote modeling file replaced transformers' conversion mapping for …`** at load | A hub modeling file (Nemotron-H's) registers its conversions with `overwrite=True` for the whole process, so a model built from transformers' own classes would load under names it does not have. | Load the family one way per process: every model with `trust_remote_code`, or none. | [Correct the conversion mapping](https://whitecircle.com/halo/docs/agent-docs/models/adding-a-model.md#correct-the-conversion-mapping) |
| **`Expert Parallelism (…) needs routed experts`** or **`… needs a family EP layer`** at model load, before the process groups | An EP or expert-TP group on a dense model, or on a MoE with no family spec (Mixtral). No EP layer would wrap it, so the experts would stay replicated at a DP size that assumes them sharded. | Set `expert_parallel_size` and `expert_tensor_parallel_size` to 1, or [add the family](https://whitecircle.com/halo/docs/agent-docs/models/adding-a-model.md). | [Adding a Model](https://whitecircle.com/halo/docs/agent-docs/models/adding-a-model.md) |
| **`Expert Parallelism (…) on 'bailing_hybrid' needs the release's own modeling file`** at model load | `trust_remote_code: false` built transformers' native Ling 3.0 class, whose `BailingHybridMoE` no EP layer claims. | Set `trust_remote_code: true` and pin `model_revision`. | [Bailing/Ling](https://whitecircle.com/halo/docs/agent-docs/models/bailing.md#ling-30) |
| **Loss near `ln(vocab_size)` from step 1**, or a new release trains far worse than its siblings | The weights did not land where the class reads them. | Read rank 0's `LOAD REPORT` from `from_pretrained`: its `UNEXPECTED`, `MISMATCH` and `CONVERSION` lines name the keys. | [Adding a Model](https://whitecircle.com/halo/docs/agent-docs/models/adding-a-model.md#new-release-of-a-supported-family) |
| **QLoRA rejected** at model load under EP / TP / grouped-GEMM MoE | The EP lazy loader, the grouped-GEMM MoE loader and TP DTensor sharding all materialize plain de-quantized weights, so bitsandbytes `Params4bit` are lost and PEFT's 4-bit dispatch fails. An `ep_size=1` MoE run hits this through the grouped-GEMM EP layers alone. | QLoRA on plain DDP/FSDP (either launcher; a MoE also needs `use_grouped_gemm: false`), or CP on a dense model; plain LoRA (no quantization) for EP. TP and EP+TP reject every adapter shape, native expert LoRA included. | [Scale & Limits](https://whitecircle.com/halo/docs/agent-docs/reference/scale-and-limitations.md) |
| **`use_hsdp` / `fsdp_reshard_after_forward` / `fsdp_reshard_after_backward` / `fsdp_defer_grad_sync` rejected** at trainer construction under QLoRA | `fully_shard` cannot wrap bitsandbytes `Params4bit`, so QLoRA skips FSDP2 and syncs gradients with a post-backward all-reduce instead. The knobs shape a wrap that never happens — allowed through, a multi-node run asking for `use_hsdp` would silently train on flat replicated gradients. | Remove the flag(s), or drop the 4-bit quantization (plain LoRA or full fine-tuning) to get the sharding the config names. | [PEFT](https://whitecircle.com/halo/docs/agent-docs/optimization/peft.md) |
| **`Dataset <path>: the ranks … that must hold the same rows loaded different ones`** or **`The shard index of <path> differs across ranks`**, at dataset load on every rank | Ranks read their own copies of the source and the copies differ: a node serving an old per-node S3 cache while S3 is unreachable from it, a pre-staged directory synced at another time, another cached hub revision, or a re-prepare racing the launch. The row message names the ranks and nodes holding each version; the shard-index one lists each split's distinct digests and rank 0's. | Re-sync the stale copy, or delete it (`$HALO_DATA_ROOT/s3_datasets/<md5>` for an S3 source) so the node re-downloads, and relaunch. | [Data Loading](https://whitecircle.com/halo/docs/agent-docs/parallelism/data-loading.md) |
| **`Dataset-map cache key cannot hash the content of tokenizer ... and keys on its name_or_path ... instead`** at the dataset map | Reading the tokenizer's content raised — a custom Python component the fast backend cannot serialize, or a remote-code `get_vocab` — so the map cache keys on the tokenizer's path. A resume that loads it from the run's checkpoint then re-runs every map, and a tokenizer changed in place at the same path reuses the previous run's rows. | Nothing to do for an unchanged tokenizer beyond the re-map on resume. After editing one in place, save it to a new path or clear `HF_DATASETS_CACHE`. | [Data Loading](https://whitecircle.com/halo/docs/agent-docs/parallelism/data-loading.md) |
| **Embedding index / OOB crash** right after the dataset map, wrong-model token IDs | A map whose output depends on a closure value the cache fingerprint cannot hash. Tokenizer identity and chat-template hash are already in the key, so a plain model swap cannot collide. | Look for the `Dataset-map cache fingerprint ... skips a value of type ... it cannot fingerprint` warning and thread that value through `cache_key_extras`; clear `HF_DATASETS_CACHE` only as a blunt fallback. | [Data Loading](https://whitecircle.com/halo/docs/agent-docs/parallelism/data-loading.md) |
| **`Dataset path '…' is spelled as a HuggingFace Hub id, but a local directory of that name exists here`**, or **`Dataset … carries a ':config' or '@split' suffix`** from a dataset tool | Either reading of the first would be a guess, and the wrong one fails far from the cause. A tool with `--config` and `--split` reads a bare repo id. | Write `./org/name` or an absolute path for the directory, or run where no such directory exists for the Hub dataset. Pass the bare `org/name` with `--config` / `--split`. | [Dataset path formats](https://whitecircle.com/halo/docs/agent-docs/data/s3-utilities.md#dataset-path-formats) |
| **`… names its bucket and key itself; drop --bucket/--subfolder, or pass the bare key with them`** from `s3.py` | A full `s3://bucket/key` URI is taken as given, so a second bucket or subfolder contradicts it. | Pass one or the other. | [S3 CLI](https://whitecircle.com/halo/docs/agent-docs/data/s3-utilities.md#cli) |
| **`The preprocessed rows were rendered with …, but this run ends with …`** at startup | The checkpoint exports the run's template, so the model would train on one template and be served with another. | Set `chat_template` to the rows' template with `force_chat_template: true`, or re-preprocess with the template the run exports. | [Pre-Processing](https://whitecircle.com/halo/docs/agent-docs/data/dataset-preparation.md) |
| **`chat_template differs from the template … ships`** at startup | Without `force_chat_template: true` the run would keep the shipped template, train on it and export it. | Set `force_chat_template: true`, or remove `chat_template`. | [Chat templates](https://whitecircle.com/halo/docs/agent-docs/data/dataset-formats.md#chat-templates) |
| **`does not support these config fields (they would be silently ignored): ['chat_template_path']`** from SFT or self-distillation | The rows are rendered before the trainer applies `chat_template_path`, so the checkpoint would export a template they never saw. | Set `chat_template`, with `force_chat_template: true` over a template the checkpoint ships and `added_special_tokens` for any tokens it needs. | [SFT](https://whitecircle.com/halo/docs/agent-docs/training-methods/sft.md#configuration) |
| **`assistant_message_template … does not occur in this tokenizer's rendered chat template`** where a collator is built | Completion-only masking would mask every label and train zero tokens at a loss near 0. | Set the marker to the exact substring the template emits before assistant content, or set `train_on_completions_only: false`. | [Collators → Constraints](https://whitecircle.com/halo/docs/agent-docs/data/collators.md#constraints) |
| **`tree_packing needs: …`**, **`tree packing is not available for model_type=…`**, **`tree packing is verified on …`** or **`tree_full_attention='split' …`** before the load; **`the row is not a depth-first tree layout`** or **`a tree node starts on a trained token`** at collation | A requirement is unmet, or the family or backend is not admitted. The `split` form needs a gradient for the kernel's log-sum-exp, which only FA4 returns and its head_dim-256 kernel does not. The collation errors mean a row was cut below a tree. | Meet what the message lists; set `tree_full_attention: gather` for the `split` refusals. Keep `packing_strategy: bfd`, so rows reach the collator whole. | [Tree Packing → Limits](https://whitecircle.com/halo/docs/agent-docs/optimization/tree-packing.md#limits) |
| **`tree packing got an empty … split on at least one data-parallel rank`**, or **`Cannot pack the … split: it has no rows`** at the dataset map | The split has nothing left to pack after the dataset's own filters, or on a rank's share of it. | Check the split's size and `test_size` against the data-parallel degree. | [Tree Packing](https://whitecircle.com/halo/docs/agent-docs/optimization/tree-packing.md) |
| **`dataset_ratio=… cannot apply to the pre-processed dataset at …`** at the dataset load | A pre-processed dataset fixed its rows at preparation, so the run would train all of them. | Unset `dataset_ratio`, or subset the source before preparing it. | [Dataset Formats](https://whitecircle.com/halo/docs/agent-docs/data/dataset-formats.md) |
| **`packing / padding_free are not supported for self-distillation`** at startup | The self-distillation collators pad one row per conversation in each branch, and OPD pairs those rows one to one. | Remove both from the YAML. | [Self-Distillation](https://whitecircle.com/halo/docs/agent-docs/training-methods/distillation/self-distillation.md#configuration) |
| **`Dataset '…' can be neither used nor tokenized`** at SMPO construction | The dataset holds neither the prepared columns nor the raw columns the tokenize map reads. | Pass the raw `prompt`, `chosen`, `rejected` rows (and `images` for a VLM run). | [SMPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/preference/smpo.md#configuration) |
| **`glm5-chat.jinja renders system, user and assistant text turns only`** at the dataset map | `chat_template: jinja-templates/glm/glm5-chat.jinja`, the GLM-5 training template, renders plain text turns only, so it refuses a tool definition, an assistant turn with `tool_calls` or a `tool` turn instead of dropping it from the row. | Render tool data with the hub template (leave `chat_template` unset), or drop the tool turns from the dataset. | [GLM-5 Next](https://whitecircle.com/halo/docs/agent-docs/models/glm5-next.md#tree-packing) |
| **`packing=True on attn_implementation='sdpa' is refused for model_type=…`** (or `eager`, `flex_attention`) at startup — GPT-OSS, Bailing / Ling, GLM-5 Next | These families' forwards never pass `position_ids` into their mask construction, so on a dense backend a packed row runs as one causal sequence and its documents attend across each other. | Use a flash backend (GPT-OSS, Ling 2.0, Ring), or turn `packing` off. Ling 3.0 and GLM-5 Next have no flash backend: train on padded rows, or GLM-5 Next on `tree_packing`. | [Collators](https://whitecircle.com/halo/docs/agent-docs/data/collators.md#document-isolation-under-packing) |
| **`packing=True is refused for model_type=…: its linear-attention layers receive no document boundary`** (or `its depthwise causal short convolutions … receive no document boundary`, or `padding_free=True …`) at startup — Ring on a flash backend, Inkling, or any hybrid whose linear-attention layers read no segment marker | The state of those layers (Ring's Lightning Attention-2, Inkling's short convolutions) carries from each document of a row into the next on every attention backend, and nothing in the loss shows it. | Turn `packing` / `padding_free` off, or set `allow_cross_document_linear_attention: true` where that mixing is acceptable. | [Collators](https://whitecircle.com/halo/docs/agent-docs/data/collators.md#document-isolation-under-packing) |
| **`DPO on image data cannot train '<model>' (model_type='paligemma'): its decoder is a prefix LM`** (or `KTO`, `SMPO`) at startup, before any weight loads | PaliGemma's decoder needs `token_type_ids` marking each completion as the causal suffix, and no vision preference collator writes them: TRL's (DPO, KTO) calls the processor on the completions without images, which PaliGemma's raises on, and SMPO's marks the whole row as prefix, so each completion would score with its own next tokens in view. | Train PaliGemma on images with SFT or distillation, or run the preference method on text-only pairs. | [Modality support](https://whitecircle.com/halo/docs/agent-docs/reference/trainer-architecture.md#modality-support) |
| **`VLM … holds a row without an image on a prefix-LM (PaliGemma) model`** in the collator, at the first batch holding such a row | PaliGemma's processor splices `<bos>` after each row's last `<image>`; in a row without one it lands inside the text, and a batch with no image raises inside the processor. | Drop the text-only rows from the image dataset, or train them in a separate text-only run. | [SFT → Vision-language models](https://whitecircle.com/halo/docs/agent-docs/training-methods/sft.md#vision-language-models) |
| **`The chat template renders <bos>, which this prefix-LM (PaliGemma) processor already puts after each row's last image`** where the VLM collator is built | PaliGemma's processor puts `<bos>` after each row's last image itself, so a chat template that renders one too (a Gemma template opens with `{{ bos_token }}`) would leave two in every row, and in every prompt it renders for serving. | Remove the `{{ bos_token }}` from the template and point `chat_template` at the edited copy, with `force_chat_template: true` when the checkpoint ships a template (without it a differing one raises). | [SFT → Vision-language models](https://whitecircle.com/halo/docs/agent-docs/training-methods/sft.md#vision-language-models) |
| **`<collator>: no assistant turn of the chat template ends in a turn terminator [...]`** where a VLM collator is built, under `train_on_completions_only` | A completion span ends at a turn terminator, the tokenizer's eos or the config's `eos_token_id`. A template that closes assistant turns with another token (a Gemma template on PaliGemma closes them with `<end_of_turn>`, its eos is `<eos>`) leaves every span open, so completion-only masking would train no token. | Set `eos_token` to the token the template closes an assistant turn with (`eos_token: <end_of_turn>` for a Gemma template), or set `train_on_completions_only: false`. | [Collators → Constraints](https://whitecircle.com/halo/docs/agent-docs/data/collators.md#constraints) |
| **`packing under pipeline parallelism is refused for model_type=…`** at startup | The model's conv / linear-attention layers read no document boundary at all (GLM-5 Next's KDA, Lightning Attention-2, Inkling's short convolutions), so their state carries across the documents of a packed row. GatedDeltaNet without its segment-aware kernels is refused as `packing=True is refused for the linear-attention family`. | Turn `packing` off (padded rows hold one document each), or train without PP. | [Pipeline Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md#packing) |
| **`OSError: AF_UNIX path too long`** in a `SyncManager` process, then `EOFError` out of `Map (num_proc=N)` | `datasets.map` with `num_proc > 1` starts a manager that listens on an AF_UNIX socket under `$TMPDIR` (`pymp-*/listener-*`, 32 characters deeper). Linux caps a socket path at 108 bytes including the terminating NUL, so a `TMPDIR` longer than 75 characters fails. | Keep `TMPDIR` short: a top-level directory on the large volume (`$D/tmp` in the standard launch), or a short symlink to a deep one. | [Docker](https://whitecircle.com/halo/docs/agent-docs/infrastructure/docker.md#standard-training-launch) |
| **`NVRM: Xid ... 145, RLW_RXPIPE Nonfatal` flooding `dmesg`**, thousands per hour | A physically marginal NVLink lane. Forward error correction absorbs the errors, so the link keeps full bandwidth and no data is at risk — the Xid line is informational. | Run `python scripts/profiling/nvlink_health.py`: it exits 1 only on errors FEC did **not** absorb or a down link, and reports correctable churn as `marginal`. Escalate for a physical reseat only if a link turns unhealthy or codewords reach the deep FEC bins. | [Debugging](https://whitecircle.com/halo/docs/agent-docs/reference/debugging.md) |
| A serving engine rejects or silently drops a MoE checkpoint's experts (missing per-expert keys) | The engine's loader reads only per-expert `experts.{i}.*` keys and the checkpoint holds the fused pair: an EP-gathered DeepSeek-V4 save on vLLM, or a Cohere2 MoE or DeepSeek-V4 one on SGLang. GLM-5 Next's gathered saves are per-expert, so only a checkpoint an earlier release saved with the fused pair hits it. GLM-4 MoE Lite (either engine) and LFM-2 (vLLM) hit it only with a checkpoint fused before Halo loaded it: the EP-gathered save writes both per expert, and the other writers keep the layout the load read. | Rewrite it with `scripts/after_training/unfuse_moe_experts.py`. | [Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#serving-on-vllm--sglang) |
| **`N model PARAMETER(s) reached device placement still on the meta device`** | The checkpoint load lost those tensors — a key the loader's disk→model mapping did not resolve, or state the checkpoint does not carry that nothing initialized. They hold no weights, and filling them would train uninitialized memory that differs on every rank. | Check the named keys against the checkpoint's index; load with `ep_lazy_loading=False` to route through `from_pretrained`. | [Expert Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/expert-parallelism.md#model-loading) |
| **`N model BUFFER(s) reached device placement still on the meta device`** | Same gate, buffer side: a load path that skipped `finalize_loaded_model`, or a buffer none of its fixers recomputes. The lazy shell builds under `init_empty_weights(include_buffers=False)`, so buffers computed in `__init__` (a config-less rotary such as Qwen VL vision) arrive intact; reaching the gate is a loader or fixer gap, not an unsupported model or EP config. | Route the load through `finalize_loaded_model`, or add the family's buffer to the `_ROTARY_FIXERS` / `_NON_PERSISTENT_FIXERS` chain in `halo/model_loading/buffer_repair.py`. A zero `inv_freq` degenerates RoPE to NoPE at a plausible loss, which is why the gate is fatal. | [Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md) |
| **`Lazy load: tensor from checkpoint key(s) ... has shape ... but model tensor ... expects ...`**, or **`global expert index(es) ... are absent`** at an EP lazy load | The checkpoint and the config disagree: a `patch_vocab`-grown checkpoint paired with the base `config.json`, a changed `intermediate_size` or expert count, a truncated per-expert upload, or a stale hub rename. `from_pretrained` catches these itself; the lazy path re-implements the same gates. | Fix the pairing — load the checkpoint with the `config.json` it was saved with, or re-fetch a torn checkpoint. The raise is world-uniform (rank-consensus fenced), so the first error names the real cause. | [Expert Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/expert-parallelism.md#model-loading) |
| **`N checkpoint key(s) align to no model tensor`** at an EP lazy load | Checkpoint keys the loader's mapping never claimed — the same causes as the row above, most often a stale hub rename. This one is a **warning**, not a raise, and it is not rank-fenced: those tensors are skipped and the run proceeds. | Read the named keys before ignoring it. Genuinely surplus keys (a checkpoint carrying extra state this architecture drops) are fine; a renamed weight here means the model tensor was filled by something else or left to the load-coverage gate. | [Expert Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/expert-parallelism.md#model-loading) |
| **`N tensor(s) of <ModelClass> are absent from the checkpoint and were randomly initialized`** at model load | The checkpoint does not carry weights the live model needs — truncated, partially uploaded, or written for a different architecture. Without the gate the run would train part-pretrained, part-noise with no other symptom. | Check the named keys against the checkpoint's index and re-fetch it. A task head the architecture adds on top of the backbone, a tied `lm_head` and the class's own `_keys_to_ignore_on_load_missing` are excused already; `HALO_ALLOW_MISSING_CHECKPOINT_KEYS=1` accepts the rest deliberately. | [Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#load-coverage-gate) |
| Qwen3.8-Flash-Next refused at startup: **`init_from_scratch builds … with random weights, but its external tables are frozen`**, **`… holds external tables …, and under accelerate-managed FSDP every save of it goes through transformers' save_pretrained`**, **`eos_token sets eos_token_id=…, but […] copied the checkpoint's eos id`**, **`….save_pretrained writes its state dict, which holds none of its external tables`**, or, on the sentence-transformers embedding load (no EP, TP or grouped-GEMM EP layers), **`… builds its external tables (Qwen3.8-Flash-Next's n-gram table) in host memory only from the config its family patches stamp`** | The 95 GiB n-gram table lives outside the module state: only the checkpoint supplies it, only the toolkit's gathered writers save it, its embedding copies `eos_token_id` when it is built, and sentence-transformers builds the backbone from a config that carries no stamp. | Train from the checkpoint; launch with `torchrun` (or train adapters); leave `eos_token` unset; save through the trainer's `save_model`; the sentence-transformers embedding load (no EP, TP or grouped-GEMM EP layers) takes no such backbone. | [Qwen3.8-Flash-Next](https://whitecircle.com/halo/docs/agent-docs/models/qwen4_exp.md#refused-modes) |
| **`Resolving the checkpoint '<path>' failed on K of N rank(s)`** at model load; the failing ranks raise **`'<path>' is not a directory on <host>, and this rank's Hub cache (...) holds no snapshot of it`**, or every rank names one whose cache **`holds the snapshot of '<repo>' at commit <c> without M of the N weight file(s) global rank 0's snapshot names`** | Some ranks cannot see the source the load names: a local checkpoint present on some nodes only, or a Hub snapshot that global rank 0 fetched into an `HF_HOME` the other nodes do not mount (a shared input declaration over node-local caches; there the config and tokenizer reads leave a weightless snapshot, which counts as missing). A fetching rank's own snapshot comes up short when its fetch ran offline (`HF_HUB_OFFLINE`) or against an unreachable hub, which serves whatever the cache holds. | Put the checkpoint, or `HF_HOME`, on a mount every node sees. Otherwise declare per-node input with `DIST_INPUT_SHARED_FILESYSTEM=0`, so each node's local rank 0 fetches its own copy. A short fetch: fetch with the hub reachable, or stage the complete snapshot. | [Multi-Node](https://whitecircle.com/halo/docs/agent-docs/parallelism/multi-node.md#model-loading) |
| **`The checkpoint '<repo>' each rank resolved differs across ranks`** at model load | Per-node Hub caches hold different commits of the repo: staged at different times and loaded offline, or a revision that moved between two nodes' fetches. The nodes would otherwise train different weights. | Pin the revision (`model_revision`, `teacher_model_revision`) to one commit, or refresh every node's cache to the same one. | [Multi-Node](https://whitecircle.com/halo/docs/agent-docs/parallelism/multi-node.md#model-loading) |
| **`'<path>' probes as multimodal on some ranks and text-only on others`** before the model load | The modality probe read the config on some ranks and fell back to the model name on others (a node that cannot see the source), or per-node caches hold different commits whose configs disagree. The ranks would build different model classes. | Make the same checkpoint readable on every node, and pin `model_revision`. | [Multi-Node](https://whitecircle.com/halo/docs/agent-docs/parallelism/multi-node.md#model-loading) |
| **CPU RAM spikes / OOM per node at model load** | Batched per-node loading. Left unset it adapts to the node — `min(4, max(1, local_world_size // 2))`, so 4 ranks at once on an 8-GPU node and 2 on a 4-GPU tray. | Set `max_concurrent_loading` explicitly: toward `1` (fully sequential) on CPU-RAM-constrained machines; `0` (all-parallel) only with ample RAM. | [Configuration](https://whitecircle.com/halo/docs/agent-docs/reference/configuration-reference.md) |
| **`net.ipv4.ip_local_port_range starts at N, which leaves K test worker(s) no port`** from a test | `tests/common/ports.py` hands out test ports between 20000 and the kernel's ephemeral range, so no kernel-picked port can take one; this host's range starts less than one port per xdist worker above 20000. | Move the range up to the Linux default: `sysctl -w net.ipv4.ip_local_port_range="32768 60999"` on the host (a `--network host` container shares it), or `docker run --sysctl net.ipv4.ip_local_port_range="32768 60999"` for a container with its own network. | [Contributing](https://whitecircle.com/halo/docs/agent-docs/contributing.md#tests) |
| **`The launcher declares world_size=N but ['MASTER_ADDR', 'MASTER_PORT'] is unset`** at startup | A bare `srun --ntasks-per-node=N <script>`: SLURM declares the world through `SLURM_NTASKS` but supplies no `env://` rendezvous, so no process group can be built. | Launch one `torchrun` per node — `srun --ntasks-per-node=1 bash -c 'torchrun --nnodes=$SLURM_NNODES --node_rank=$SLURM_NODEID …'`, quoted so each task expands its own node rank — or export `MASTER_ADDR`/`MASTER_PORT` identically on every task. | [Launch Recipes](https://whitecircle.com/halo/docs/agent-docs/parallelism/launch-recipes.md#slurm) |
| **`--numa-binding is a torchrun option and needs a torchrun launch`** from `halo launch` | `accelerate launch` and `python` have no NUMA binding, so `--accelerate` or a single process refuses the flag. | Drop the flag, or launch more than one process on torchrun. | [NUMA binding](https://whitecircle.com/halo/docs/agent-docs/optimization/throughput-benchmarks.md#numa-binding) |
| **`ddp_timeout=… would never apply`** at startup | The process group starts before the config is parsed, under the NCCL watchdog `DIST_NCCL_TIMEOUT_MINUTES` sets. | Remove `ddp_timeout` and set `DIST_NCCL_TIMEOUT_MINUTES` in the launch environment, alike on every node. | [Environment variables](https://whitecircle.com/halo/docs/agent-docs/reference/configuration-reference.md#environment-variables) |
| **`Output-filesystem probe: … output_dir must be writable by global rank 0`** at startup | The multi-node startup probe could not write its sentinel under `output_dir`. | Fix the directory's permissions or mount. | [Filesystem Handling](https://whitecircle.com/halo/docs/agent-docs/data/filesystem-handling.md) |
| **`Output filesystem is declared SHARED but N of M ranks … cannot see a file global rank 0 wrote`**, or the PER-NODE mirror image, at startup | The multi-node startup probe wrote a sentinel under `output_dir` from global rank 0 and the declaration contradicts what the ranks see. Declared shared but invisible: only rank 0 writes `trainer_state.json`, so a resume would find every checkpoint incomplete on the other nodes. Declared per-node but visible everywhere: every node's local rank 0 would write the same checkpoint paths. | Match the declaration to the mount with `DIST_OUTPUT_SHARED_FILESYSTEM` (or the `DIST_SHARED_FILESYSTEM` umbrella), or move `output_dir` — onto the shared mount, or one per node. On a shared mount whose attribute cache outlasts the probe (`actimeo`/`acdirmax` above 60 s), raise `DIST_FS_PROBE_TIMEOUT_SECONDS` instead. | [Filesystem Handling](https://whitecircle.com/halo/docs/agent-docs/data/filesystem-handling.md) |
| **`Input filesystem is declared SHARED but N of M ranks … cannot see a file global rank 0 wrote under HF_HUB_CACHE`** (or `HF_DATASETS_CACHE`), or the PER-NODE mirror image, at startup | The multi-node startup probe wrote a sentinel under each HF cache from global rank 0 and the declaration contradicts what the ranks see. Declared shared but invisible: the other nodes would read caches nobody filled. Declared per-node but visible everywhere: every node's local rank 0 would fill the same cache files. | Match the declaration to the caches with `DIST_INPUT_SHARED_FILESYSTEM` (or the `DIST_SHARED_FILESYSTEM` umbrella), or move the cache the refusal names — onto the shared mount, or onto each node's local storage. One declaration covers both caches, so `HF_HUB_CACHE` and `HF_DATASETS_CACHE` must be on the same kind of storage (an explicit `HF_DATASETS_CACHE` does not follow `HF_HOME`). On a shared mount whose attribute cache outlasts the probe, raise `DIST_FS_PROBE_TIMEOUT_SECONDS`. | [Filesystem Handling](https://whitecircle.com/halo/docs/agent-docs/data/filesystem-handling.md) |
| **`Multi-Node NVLink prerequisite check … failed on N of M rank(s)`** at config time | `NVLINK_DOMAIN_SIZE > gpus_per_node` declares node-local groups that span OS nodes over NVLink, and some rank has no IMEX channels at `/dev/nvidia-caps-imex-channels`, reports a fabric registration other than `COMPLETED`, or sees no fabric clique at all. | Bring up NVIDIA Fabric Manager plus the IMEX service with matching channels on every node (NCCL >= 2.25.2), or drop `NVLINK_DOMAIN_SIZE` to `gpus_per_node` and keep the groups within one OS node. | [Multi-Node](https://whitecircle.com/halo/docs/agent-docs/parallelism/multi-node.md) |
| **Weight sync hangs at the first collective** — the group formed, both ends idle, `/health` unanswered past the one-time init | The two containers drive different NCCL transports (`NCCL_NET` / `NCCL_NET_PLUGIN` differ) or different `aws-ofi-nccl` + libfabric builds (an upstream server image, a host-installed plugin against the image's): the pair is wire-sensitive and forms the group before hanging. | Serve from the Halo images and run the same fabric recipe on both ends — compose EFA overlay + `make ... EFA=1`, or both on the no-fabric defaults; `python scripts/profiling/weight_sync_transport.py --server_url http://<server>:8000 --expect efa` reports the transport and plugin build the group formed on. | [Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#servers-on-other-nodes-efa) |
| **`server ... is on a remote host ... but the NCCL weight-sync group address resolved to loopback`** at sync init | The trainer advertised a loopback master address, which the remote engine's workers would dial back to themselves. | Set `VLLM_GROUP_HOST` / `SGLANG_GROUP_HOST`, or the entry's `group_host` in `rollout_server_configs`, to the trainer IP on the subnet the server reaches. | [Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md) |
| **`weight-sync group address ... is not a single address of this host that the server ... can reach`** at sync init | The advertised group address is one the trainer cannot bind alone (a NAT or port-mapped address, `0.0.0.0`, or a hostname resolving to loopback for a remote server), and the unauthenticated rendezvous store listens only on the address it advertises. | Set `VLLM_GROUP_HOST` / `SGLANG_GROUP_HOST`, or the entry's `group_host`, to this host's IP on an interface the server reaches. Behind NAT or a port mapping, set `HALO_WEIGHT_SYNC_BIND_ALL=1`, on a network where only trusted hosts reach the group port. | [Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#group-rendezvous) |
| **`weight-sync group address ... has no IPv4 address on this host`** at sync init | The group address is an IPv6 literal or a name without an IPv4 address; the rendezvous store listens on IPv4 only, so IPv6 is unsupported. | Set `VLLM_GROUP_HOST` / `SGLANG_GROUP_HOST`, or the entry's `group_host`, to this host's IPv4 address on an interface the server reaches. | [Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#group-rendezvous) |
| **`Could not bind the weight-transfer group port`** at sync init | Another socket holds the configured port: a live client, or an outbound connection given it as its source port (inside `net.ipv4.ip_local_port_range`). Raised before the server is asked to join. | Give each server its own `group_port`, reserved or outside the ephemeral range. | [Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#group-rendezvous) |
| **`rollout_backend=... cannot serve model_type [...]`**, or `... does not support weight sync`, at startup or trainer construction | The engine release cannot take an online update for this family (the client's `UNSERVABLE_MODEL_TYPES`), or the family spec's `weight_sync_refusal` says why no sync into it is validated (refused on both engines). | Train it offline (offline GRPO needs no server, or SFT). Switching `rollout_backend` helps only against a client's `UNSERVABLE_MODEL_TYPES` entry, since the two engines list different families; a `weight_sync_refusal` holds on both. | [Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#which-families-each-engine-serves) |
| **`… resolves for model_type [...], but the model carries no live EP layer`**, **`… has no routed experts`** or **`… is a MoE with no Halo EP family`** at trainer construction | A family with an EP layer runs without one (`use_grouped_gemm: false` at `ep_size: 1`), a dense checkpoint sits under a spelling a MoE family claims, or the MoE has no family spec, so no engine layout covers its stock tree. | On a family with an EP layer, set `use_grouped_gemm: true`. Otherwise train it offline, or add the family. | [Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#construction-gates) |
| **`vLLM server … runs vLLM …; this client speaks the weight-transfer protocol of vLLM >= …`** at sync init | The client reads the server's `/version` first. A server older than 0.32.0, a 1.1.0 rollout image among them, speaks another weight-transfer wire. | Serve from this release's toolkit image (`make build-vllm`, or the published `vllm-0.32.0` tag). | [Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#vllm) |
| **`use_adaptive_entropy is not supported`**, **`entropy_coef=… trains nothing with use_chunked_grpo_logprobs`**, **`… is an IterableDataset, which the DP-sharded loader this run uses cannot group into prompt groups`** or **`ref_model was passed, but this run holds no KL reference`** at online or async GRPO construction | A sharded resume does not restore TRL's adaptive coefficient. The chunked sweep returns a detached entropy. The prompt-group sampler needs a dataset with a length. A run at `beta: 0`, or a policy scored with its adapter disabled, reads no `ref_model`. | Set a fixed `entropy_coef`; set `entropy_coef: 0.0` or turn the chunked sweep off; load a map-style `Dataset`; drop `ref_model`. | [Online GRPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/online.md) |
| **`train_sampling_strategy='batch_rebalance' shards batches across every process`** at trainer construction | That sampler shards over every process, while TP, CP, ETP, PP and a pre-sharded dataset shard the loader over data-parallel replicas, so ranks sharing a batch would draw different rows. | Use `random`, `sequential` or `group_by_length`. | [Data Loading](https://whitecircle.com/halo/docs/agent-docs/parallelism/data-loading.md) |
| **`length_ordered_microbatches: true, but …`** at construction, or **`TRL refilled its generation buffer without calling shuffle_sequence_dict`** at a round | The deal would change the objective or order nothing on this run, as the message says, or the installed TRL moved the shuffle the deal replaces. | Unset the option or set it to `false`. | [Length-ordered micro-batches](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/performance.md#length-ordered-micro-batches) |
| **`loss_forward_importance_weights: true, but …`** at construction, or **`The loss forward scored … time(s) with entropy`** / **`TRL's loss never read …`** at a micro-batch | A consumer needs the round's log-probs before the first backward, as the message names it, or the loss forward no longer reaches the scoring hook once or TRL moved where it reads the weight. | Set the option to `false` (the default), which keeps the no-grad pass. | [Weight from the loss forward](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/performance.md#weight-from-the-loss-forward) |
| **`The rollout engine cannot enforce a thinking budget for model_type …`** at trainer construction (Mistral 4 on vLLM) | vLLM serves the family with its `mistral_common` tokenizer, which encodes `[THINK]` as plain text while the model writes it as a control token, so the budget never starts counting. | Unset `rollout_max_thinking_tokens` and `rollout_max_answer_tokens`, and steer with `reasoning_effort`. | [Reasoning Budgets](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/reasoning.md) |
| **`GRPOConfig 'vllm_group_port' is not read by environmental GRPO`** at construction | Async GRPO's weight-sync group port is `AsyncRolloutConfig`'s, on either backend. | Set `rollout_group_port`. | [Weight synchronization](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/setup.md#weight-synchronization) |
| **`isr_engine_rescore_per_server set with isr_engine_reference off`**, or **`reasoning_terms_in_failed_groups set with reasoning_price and reasoning_floor both off`** at config time | The option modifies another that is off, so it would change nothing. | Set the option it modifies, or remove it. | [AsyncRolloutConfig](https://whitecircle.com/halo/docs/agent-docs/reference/configuration-reference.md#asyncrolloutconfig) |
| **`scratch_allowlist, scratch_env and episode_scratch give a spawned server its scratch directory`**, **`episode_scratch scopes the server's scratch directory to the episode, but this server takes none`**, **`server_env sets …, which scratch_env would point into the scratch directory`** or **`env_vars sets …, one file every episode's own server would read and write at once`** at MCP environment construction | An `sse` server is already running, so nothing spawns it into a scratch directory. A server without a scratch option has no directory to scope. The last two name one variable twice, or one file for every episode's server. | Use the stdio transport or drop the scratch options; drop `episode_scratch`; drop the variable from one of the two. | [MCP](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/environments/protocols/mcp.md#configuration) |
| **`judge term(s) … list 'checks: …', the checks the environment declares`**, **`judge check(s) … stand exempt under episode facts`** or **`context_file … is not a file`** at config or environment construction | The environment declares no checks to read, a check's `unless` names a fact the environment never establishes, or the path does not resolve (a relative one reads from the launch directory). | List the checks in the term, fix the `unless`, or use an absolute path. | [Judge → Environment checks](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/rewards/judge.md#environment-checks) |
| **`per_device_train_batch_size * steps_per_generation (N) must be divisible by num_generations`** at construction | Async GRPO groups advantages rank-locally, so a generation group cannot straddle ranks. TRL checks only the global generation batch. | Size `per_device_train_batch_size × steps_per_generation` (which defaults to `gradient_accumulation_steps`) to a multiple of `num_generations`. | [Async GRPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/objective.md#batch-construction) |
| **`Could not verify the head path of <ModelClass> (...)`** at GRPO trainer construction with `use_chunked_grpo_logprobs: true` | The chunked log-probs compute logits from the hidden state, so the family's forward is first probed on a meta-device shell with a stubbed backbone and a stand-in output embedding. The probe could not run it; the parenthesized error says why (an output embedding that is not a plain `nn.Linear`, a class with no backbone below it, a forward reading other model state on its head path). | Set `use_chunked_grpo_logprobs: false`. For a family you are adding, make its head path pass the probe. | [Adding a Model](https://whitecircle.com/halo/docs/agent-docs/models/adding-a-model.md#declare-the-head-transform) |
| **`<ModelClass>'s forward transforms its head path beyond what is declared for it`**, at the same points | The forward applies something between the backbone and the logits (a hidden-state scale, a logit scale, a softcap, a vocabulary cut) that the family's `HeadTransformSpec` does not declare, so the hidden-state paths would score a different distribution than the model samples from. | Declare the transform on a `HeadTransformSpec` in `halo/models/head_transform.py`, or set `use_chunked_grpo_logprobs: false`. | [Adding a Model](https://whitecircle.com/halo/docs/agent-docs/models/adding-a-model.md#declare-the-head-transform) |
| **`…'s attention reads a token's row index, and offline GRPO under pipeline parallelism left-pads a batch's prompts`** at trainer construction (DeepSeek-V4) | Pipeline stages forward the collated batch whole, so a shorter row's tokens would score at the wrong positions. Off PP these families score each row alone. | Set `per_device_train_batch_size` and `per_device_eval_batch_size` to 1, or drop `pipeline_parallel_size`. | [Offline GRPO → Row positions](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/offline.md#row-positions) |
| **`Offline GRPO evaluates a map-style datasets.Dataset`** or **`KL evaluation requires a finite tokenized datasets.Dataset`** from `evaluate()` | The sampler cuts rows by group and equalizes the batch counts across ranks, which a stream or a plain list cannot give it. | Pass a `datasets.Dataset`. | [Offline GRPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/offline.md) |
| **`The GRPO loss forward materializes full logits: …`** (`RuntimeError`) at GRPO trainer construction, or `Another rank's full-logits plane cannot fit` | With `use_chunked_grpo_logprobs` off, the loss forward holds a `rows × width × vocab` logits plane at 4 B a logit, and this device's free memory (or a peer's) cannot hold it. The message states the size and the batch-size and width settings behind it. | Set `use_chunked_grpo_logprobs: true`, or lower the batch size or the width setting the message names. | [Memory and Throughput](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/performance.md#the-logits-wall) |
| **`ncclP2pImportShareableBuffer ... invalid argument`** on the first update, `The full weights of the ModelRunner are partially updated` in the SGLang log | cuMem mismatch: SGLang's entry point sets `NCCL_CUMEM_ENABLE=0` process-wide unless it is pre-set, and the trainer's NCCL has cuMem on. | `NCCL_CUMEM_ENABLE=1` in the server container (`docker-compose.sglang.yml` sets it), then restart the server — it holds a half-written model. | [Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#nccl-transport-sglang) |
| **`rollout_max_thinking_tokens` or `carry_reasoning` refused** with `rollout_backend: sglang` | Neither is available on SGLang: the trainer wires no SGLang thinking-budget mechanism (`thinking_token_budget` is a vLLM-only request field), and SGLang's forwarding of `reasoning_content` is unverified, so the reasoning carry is refused. | Drop the knob, or run that recipe against vLLM. | [Async GRPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/setup.md#rollout-backend) |
| **`reasoning_effort_profiles['<level>'] sets reasoning_price without reasoning_budget`** at environment construction; **`reasoning_effort_profiles set [...] but the environment's reasoning_effort is unset`**, **`reasoning_price_cap set while no level the environment draws sets reasoning_price`**, **`reasoning_terms_in_failed_groups set while no drawn level prices reasoning and reasoning_floor is off`**, **`reasoning_floor charges a failed episode its shortfall under its level's reasoning_budget, but the drawn level(s) [...] set none above 0`**, **`the drawn level(s) [...] set reasoning_budget with no reasoning_price while reasoning_floor is off`**, **`rollout_chat_template_kwargs carries [...]`** at trainer construction | The reasoning terms read the episode's level profile: the price charges past the level's `reasoning_budget`, the floor under it. With no level drawn nothing reads them; the cap is read only while a drawn level prices reasoning, and the failed-group knob only while the price or the floor is on; the floor needs a budget above 0 at every level the environment draws (all three under `random`), and a budget neither its price nor the floor reads changes nothing; a profile key among the template variables would reach the template. | Add the level's `reasoning_budget` (`0` prices every token); set `reasoning_effort`, or drop the profile keys; price a level, or remove the knob; give every drawn level a `reasoning_budget` above 0, or turn the floor off; price the level or drop its budget; move the key into `reasoning_effort_profiles`. | [Construction checks](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/reasoning.md#construction-checks) |
| **`reasoning_floor charges an episode that earned no objective, but the reward prices no success above zero`**, **`the reasoning terms are waived in a training group no episode of which earned the objective, but the reward prices no success above zero`** at trainer construction | No environment term, or one of weight ≤ 0: every episode reads as failed, so the floor would charge every episode and a price waived in failed groups would charge none. | Give the environment term a positive weight, or turn the floor off; for the price, `reasoning_terms_in_failed_groups: true` also clears it. | [Construction checks](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/reasoning.md#construction-checks) |
| **`save_sharded_ep=True` rejected at trainer construction** | The run has a shape whose per-rank shards `merge_ep_shards.py` could not merge (multiple EP groups, expert TP, CP, expert LoRA, a non-shared multi-node output filesystem, …); the error names it, and the full list is on the linked page. | Use the gathered save (`save_sharded_ep: false`) — it streams, so it costs no more host memory, and it needs no merge. | [Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#expert-parallelism-ep-eptp-epcp) |
| **`Cannot resume precompute_ref_log_probs from <checkpoint>`**, **`reference_logps.pt ... does not belong to this '<split>' dataset`**, or **`Regenerate the '<split>' reference log-probs for this run`**, at DPO/KTO construction on resume | The resume built the policy from the checkpoint, so a reference sweep would score the trained weights, and the checkpoint holds no saved columns for that split — or holds them for other rows, reference settings or log-prob precision. | Resume with the saving run's data and reference settings. Otherwise run the config one step from the base into the scratch `output_dir` the error names and copy its `reference_logps.pt` in (on every node when checkpoints are node-local); a checkpoint without the file can also take the previous checkpoint's. | [DPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/preference/dpo.md#resuming-a-precompute-run) |
| **Exact resume warm-restarted although `world_size` is unchanged** | A fingerprint field other than `world_size` changed — any of `ep_size`, `expert_tp_size`, `cp_size`, `tp_size`, `pp_size`, `fsdp_shard_ep1_experts`, `optimizer_class`, `ep_scope`, `use_grouped_gemm`, `hsdp` (`use_hsdp`), `nvlink_domain_size`, `expert_replica_size` (`saved='<absent>'` for a key the saved meta lacks). A pipeline-parallel run raises on any drift instead. | The warning names every differing field with `saved=`/`current=` — restore it, or accept the warm restart (weights and LR schedule still resume). | [Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#warm-restart-vs-exact-resume-torchrun) |
| **`Restoring the per-rank optimizer state from <checkpoint> failed on N of M rank(s) [...]`** or **`Reading the per-rank optimizer shards from <checkpoint> failed on …`** at resume, on every rank | The shards match this run's topology, but restoring them failed on the listed ranks: a truncated or unreachable shard file, or a CUDA OOM inside `set_optimizer_state_dict`. Resuming without them would reset the optimizer moments while weights, step and LR schedule resume, so every rank raises. | Fix the cause the message quotes (each failing rank's log carries its error) and resume again. `allow_optimizer_warm_restart: true` resumes with a fresh optimizer instead. | [Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#warm-restart-vs-exact-resume-torchrun) |
| **`Optimizer resume from <checkpoint>: optimizer_meta.pt is missing beside the per-rank optimizer shards`** (or `is unreadable`), or **`… it holds a replicated optimizer.pt / optimizer.bin but no per-rank optimizer shards`**, at resume on every rank | An interrupted save (the meta is written only after every rank's shard), or an `optimizer.pt` a sharded resume cannot read: a non-sharded run's state, or an earlier release's sharded save stopped before its shards (a sharded save writes no `optimizer.pt`). Resuming past either would reset the optimizer moments while weights, step and LR schedule resume. | Resume from an earlier complete checkpoint, or set `allow_optimizer_warm_restart: true` to resume with a fresh optimizer. | [Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#warm-restart-vs-exact-resume-torchrun) |
| **`Resume checkpoint(s) [...] are incomplete on at least one node: trainer_state.json, which a save publishes last, is missing`** at startup | A save stopped partway (a preemption, a full disk), or a node's copy is incomplete. `resume_from_checkpoint: true` passes over such a directory when an older complete checkpoint exists, and moves it into `output_dir/_incomplete_checkpoints/` with a warning; an explicit path to it, or an `output_dir` holding only incomplete ones, raises. | Resume from an earlier complete checkpoint, or remove the incomplete one. | [Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#interrupted-saves) |
| **`… present on some ranks but missing on others at …`**, **`… is present but unreadable`** or **`… parsed but carries no 'weight_map'`** at resume | A torn or partial checkpoint, or a per-node copy this rank reads mid-write. | Resume from a complete checkpoint. | [Interrupted saves](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#interrupted-saves) |
| **`No node directory holds …`**, **`… and … carry different safetensors indexes`**, **`… differs in size across …`** or **`… vouches for … optimizer shards, but no input holds the optimizer shards of ranks …`** from `merge_pp_stages.py` | The inputs are not every node's directory of one pipeline-parallel save, or a copy is torn, or a node's optimizer save did not complete. | Pass every node's directory of that save and re-copy a torn one. For weights only, delete `optimizer_meta.pt` and every optimizer shard from the input. | [Per-node export](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md#per-node-export) |
| **`… names … itself: replacing it would delete the only copy`**, **`… is a fused 3-D expert tensor, but this checkpoint's model_type … matches no MoE family spec`** or **`… which a PEFT merge silently drops`** from a checkpoint tool | `--output_dir` is the source; the block-scaling axis of an unclaimed family is unknown; a native expert adapter reached the in-memory merge (`convert_to_bf16.py --peft`), which cannot fold its deltas. | Write to another directory; declare the family's `MoEFamilySpec` or pass `--exclude`; merge the adapter with `merge_peft_adapters.py` (it streams) or train with `merge_expert_lora_on_save: true`. | [Post-training scripts](https://whitecircle.com/halo/docs/agent-docs/reference/scripts-reference.md#post-training-scripts) |
| **`Target modules <name> not found in the base model`** at PEFT injection, after the load, for a module the model has | `lora_target_modules` is a bare string, which PEFT reads as a regex fullmatched against each full module path, so a module name alone matches nothing. | Write it as a list (`lora_target_modules: [o_proj]`), which names modules at any length; keep a bare string for a regex or `all-linear`. | [PEFT](https://whitecircle.com/halo/docs/agent-docs/optimization/peft.md#configuration) |
| **`lora_target_modules entries [...] are regexes, but a list names modules`** at startup | A regex written into the list. PEFT compares list entries to module paths verbatim, so it would match nothing (unnoticed beside an entry that does). | Give the regex as a bare string in the YAML, or list the module names. | [PEFT](https://whitecircle.com/halo/docs/agent-docs/optimization/peft.md#configuration) |
| **`Growing deepseek_v4's vocabulary to <n> rows leaves gate.tid2eid without a row for the added tokens`** at tokenizer setup or from `patch_vocab.py` | The hash routers index that table by token id, one frozen expert assignment per row, so an added id has none: a forward on it would raise `IndexError`, and a grown export would not load. | Keep added tokens within the checkpoint's embedding rows. | [DeepSeek-V4](https://whitecircle.com/halo/docs/agent-docs/models/deepseek-v4.md) |
| **`Growing <ModelClass>'s vocabulary to <n> rows would leave its attached MTP head scoring the old output embedding`** at tokenizer setup | `added_special_tokens` runs past the checkpoint's rows, and the resize replaces the output embedding the head's term holds: part of the term would train a module no longer in the model, and an added id would raise `IndexError`. | Grow the vocabulary before the run with `scripts/before_training/patch_vocab.py`, which carries the head, and train from its output. | [MTP Heads](https://whitecircle.com/halo/docs/agent-docs/optimization/mtp-heads.md#refusals) |
| **`The PEFT adapter at <checkpoint> does not match the adapters this model builds`** at resume, on every rank (after the `PEFT adapter resume from <checkpoint> failed on …` prefix) | The saved adapters differ from those the resumed run builds, key for key or in shape: a changed `lora_target_modules`, `lora_r`, `modules_to_save` or `added_special_tokens`, a file without the expert half the run trains, or a short copy on one node. Every rank checks before any tensor is copied, so nothing was written. | Resume with the adapter settings the checkpoint was saved with, from a complete checkpoint. | [Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#resume-by-parallelism-mode) |
| **`added_special_tokens grew the vocabulary of an adapter run under accelerate-managed FSDP`**, or **`added_special_tokens grew the vocabulary, but PEFT's adapter selection leaves out the input embedding`** (or `output head`), at trainer construction | The added rows exist in no base checkpoint, so the adapter carries them for an exact resume and a complete merge. Under accelerate-managed FSDP the base Trainer saves the adapter with PEFT's own resized-embedding check, which reads the base config again and skips it offline or without the Hub; once `lora_target_modules` adapts one layer of an untied embedding pair, PEFT saves only the adapted layers' base weights. Either would leave the added rows out of every save. | Launch with `torchrun` (toolkit FSDP2), adapt both embedding layers or neither, or grow the vocabulary before the run with `scripts/before_training/patch_vocab.py`. | [Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#peft-lora-saving) |
| **`LoRA for embedding training is not supported with Expert Parallelism (EP or expert-TP) or on the grouped-GEMM EP layers`** at embedding startup, before the model loads (at trainer construction for a hand-built run) | The save folds injected LoRA over the backbone's own parameters. Under EP they hold each rank's expert shards; on a MoE whose family has an EP layer, the default `use_grouped_gemm` puts the grouped-GEMM EP layers on the backbone at `expert_parallel_size: 1`, whose parameters hold the layers' own expert layout. Either checkpoint would load nowhere. | Drop `--expert_parallel_size` / `--expert_tensor_parallel_size` and set `use_grouped_gemm: false` (LoRA then trains the stock experts), or full fine-tune. | [Embedding](https://whitecircle.com/halo/docs/agent-docs/training-methods/embedding.md#peft--lora) |
| **`<model> is a sentence-transformers pipeline holding [...] past its backbone`** at embedding startup, before any weight loads | Under EP or TP, and on a MoE an EP layer class wraps at the default `use_grouped_gemm`, the backbone loads through the parallel loader, whose pipeline rebuilds only the backbone, `Pooling` and `Normalize`; any other module (a `Dense` head) would be left out of training and the export. | Train where sentence-transformers loads the whole pipeline: no EP or TP, and `use_grouped_gemm: false` on a MoE (one GPU or accelerate DDP if a module outside the backbone trains). Or start from the bare backbone. | [Embedding](https://whitecircle.com/halo/docs/agent-docs/training-methods/embedding.md#launch) |
| **`max_length=… reached the embedding trainer unresolved`** at a hand-built trainer's construction | sentence-transformers' collator truncates every text column to it, and only the embedding script resolves the context-window spelling. | Pass a positive length, or `null` to keep the model's `max_seq_length`. | [Embedding](https://whitecircle.com/halo/docs/agent-docs/training-methods/embedding.md#configuration) |
| **`pytest's basetemp is too deep for the AF_UNIX sockets the GPU tests bind`** from the GPU test launcher | An AF_UNIX socket path caps at 108 bytes, and `datasets` binds a multiprocessing socket (`pymp-…/listener-…`, 32 bytes) under `TMPDIR` while it tokenizes; the launcher reserves 40 bytes for it. It gives each test a `TMPDIR` under pytest's basetemp, which sits under the host's `TMPDIR`, and moves one too deep for the socket to a short directory beside it; when even that leaves too few bytes, it refuses the run. | Point `TMPDIR` (or `--basetemp`) at a shorter path on the scratch volume; the message names how many of the 40 bytes the current one leaves. | [Contributing](https://whitecircle.com/halo/docs/agent-docs/contributing.md#tests) |
| **Multi-node: ranks never rendezvous**, then `ProcessGroup not initialized` or an NCCL timeout at init | `MASTER_ADDR` is a public or management IP the fabric cannot route, the nodes sit on different fabrics, or one node's `torchrun` started outside the rendezvous timeout (10 min) or with different parallelism flags. | Use the fabric-routable address of node 0 and test its port from another node; launch every node with identical flags except `--node_rank`; kill stale `torchrun` processes. | [RunPod](https://whitecircle.com/halo/docs/agent-docs/infrastructure/runpod.md) |
| **RDMA connection timeout**, or InfiniBand down (`ibstat` shows `State: Down`) | The node was provisioned without the fabric, or a link is down. | Verify the fabric (`ibstat` on IB/RoCE, `fi_info -p efa` on EFA), then re-run with `NCCL_DEBUG=INFO` for the transport logs. Without a fabric, use node-local EP (`ep_scope: node`) so only gradient sync crosses the network. | [Multi-Node → RDMA fabrics](https://whitecircle.com/halo/docs/agent-docs/parallelism/multi-node.md#rdma-fabrics) |
| **Resume finds no checkpoint** on a multi-node run with per-node storage | Each node resumes from its own disk (`DIST_SHARED_FILESYSTEM=0` forces `save_on_each_node`), so one node lacks `checkpoint-*` with its `trainer_state.json` at the same path. | Check every node holds the step directory at the same path. | [Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md) |
| **`torch.cuda.is_available()` is `False`** inside the container, or `nvidia-smi` lists no GPU | The container started without `--gpus all`, the Docker default runtime rejects that flag (`sysbox-runc`), or the host lacks the NVIDIA driver or Container Toolkit. | Start the container with `--gpus all`; add `DOCKER_RUNTIME=nvidia` to a `make` target on a sysbox host; check `nvidia-smi` on the host. | [Docker](https://whitecircle.com/halo/docs/agent-docs/infrastructure/docker.md#verifying-and-debugging) |
| **`make check-lock` fails** on the lock export | It exports `uv.lock` with `--locked` and installs it, plus an editable `-e .`, over the image's compiled torch, Flash Attention and DeepEP in a throwaway container, so a `pyproject.toml` the lock does not match fails. | Re-lock with `uv lock` (the image ships uv), not `--upgrade`, which drifts from the versions the Dockerfile compiles. Nothing `make check-lock` installs outlives the command: a dependency change takes an image rebuild. | [Development Environment](https://whitecircle.com/halo/docs/agent-docs/contributing/development-environment.md) |

## OOM

Peak memory is dominated by saved activations, not weights. Cut it in this order:

1. **Gradient checkpointing on** — trades recompute for activation memory. Turn it off only when the
   batch already fits; the recompute costs wall-clock.
2. **Lower `per_device_train_batch_size` or `max_length`** — both scale activations linearly.

3. **Move to a sharded mode.** Dense model that OOMs in FSDP2 → add node-local TP. MoE model that
   OOMs → EP (experts across ranks) or pure ETP (`ep_size=1`, expert FFN sharded).

A large **"reserved by PyTorch but unallocated"** figure in the OOM message points at allocator
fragmentation rather than at the steps above: launch with
`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`. Under EP it costs throughput, so set it there only to fit
([measured](https://whitecircle.com/halo/docs/agent-docs/optimization/throughput-benchmarks.md#allocator-expandable-segments)).

To find *what* holds memory, dump a CUDA snapshot and drag it onto
[memory_viz](https://pytorch.org/memory_viz) ([Debugging §3](https://whitecircle.com/halo/docs/agent-docs/reference/debugging.md#3-gpu-memory-profiling)).

## NCCL timeout / hang

A hang is almost always a collective mismatch: one rank issues an all-reduce / all-gather / barrier
that another never reaches, so NCCL blocks until the watchdog fires.

```bash
python scripts/profiling/py_spy_diag.py dump   # py-spy stacks for EVERY rank on the node
```

Attaching needs ptrace access ([Debugging §1c](https://whitecircle.com/halo/docs/agent-docs/reference/debugging.md#1c-py-spy--cpu-flame-graph-dataloader--python-stalls)). Dumps land in `$HALO_DATA_ROOT/profiling/py_spy/stacks`, and
the rank *not* in a collective is the culprit.
For the definitive "which collective did rank K miss?", enable the NCCL flight recorder
(`TORCH_NCCL_TRACE_BUFFER_SIZE`, `TORCH_NCCL_DUMP_ON_TIMEOUT`).

One class of hang is refused up front instead: a job whose ranks disagree on an EP, timeout, save, attention
or matmul-precision knob (`HALO_EP_CAPACITY_DEDUP`, the DeepEP wire parameters, `HALO_GRAD_BUCKET_MB`,
`HALO_CHECKPOINT_WRITE_THREADS`, `HALO_CHECKPOINT_WRITERS_PER_NODE`, `DIST_NCCL_TIMEOUT_MINUTES`, `DIST_STORE_TIMEOUT_HOURS`,
`DIST_FS_PROBE_TIMEOUT_SECONDS`, `FLASH_ATTENTION_DETERMINISTIC`, `HALO_FLEX_SLIDING`,
`HALO_FP32_MATMUL_PRECISION`) raises
`Rank-uniform toolkit environment` in distributed setup, before the weight load — a per-node `--env-file`
rollout that reached only some nodes would otherwise diverge the collective counts from the second MoE layer
on, or load a head_dim-256 model on a different attention backend per rank, naming nothing.

Raise the watchdog for large cross-node all-to-all, or for a gathered save on storage that writes one
part (`save_max_shard_size`) slower than it: a gathered EP/TP save holds the non-writers in the next
chunk's gather while the writer flushes a part. The joins after a save's unbounded writes are exempt:
every rank's optimizer shard, a sharded EP save's per-rank shards, a gathered save's last part and
index and each pipeline stage's parts are joined over the c10d store (`DIST_STORE_TIMEOUT_HOURS`,
default 4 h). So are main-first work (model downloads, dataset load, tokenize/pack) and the
node-batched model load (`max_concurrent_loading`): the peers wait on the store, not inside a
collective.

```bash
export DIST_NCCL_TIMEOUT_MINUTES=60   # default 30; PyTorch's own default is 10
```

Full procedure: [Debugging §4](https://whitecircle.com/halo/docs/agent-docs/reference/debugging.md#4-diagnosing-multi-node-hangs).

## DeepEP build / runtime faults

DeepEP is required for EP — there is no NCCL fallback. The recurring failure modes:

- **`import deep_ep` fails (NVSHMEM / undefined symbol).** On Blackwell + PyTorch 2.11+cu130,
  NVSHMEM ships as `nvidia-nvshmem-cu13`; installing `nvidia-nvshmem-cu12` clobbers the headers. Use
  the prebuilt image.

- **`Assertion exception ... != NCCL_GIN_TYPE_NONE` / "NCCL GIN is unavailable" at buffer init on
  cross-node EP (EFA).** Proxy GIN could not come up on at least one node. See
  [DeepEP → AWS EFA](https://whitecircle.com/halo/docs/agent-docs/infrastructure/deepep.md#expert-parallelism-over-aws-efa).

    Check, on **every** node: the `gdrdrv` kernel module is loaded (`ls /dev/gdrdrv`; the host must
    install the gdrcopy driver), the container was started with `--device /dev/gdrdrv`, and
    `NCCL_GIN_TYPE=2` is exported. One node missing any of the three fails the whole job with this
    assertion while node-local EP on the same machine runs fine; NCCL's log names the gap as
    `Failed to initialize GDRCopy`.

    A host can load the module without creating the device node (`grep gdrdrv /proc/devices` prints a
    major number, `ls /dev/gdrdrv` fails). Create the node on the host, or inside the container:
    start it with `--device-cgroup-rule='c <major>:0 rwm'` instead of `--device /dev/gdrdrv`, and run
    `mknod -m 666 /dev/gdrdrv c <major> 0` before `torchrun`.

- **`CUBLAS_STATUS_EXECUTION_FAILED` / DeepEP `combine.hpp` `CUDA_ERROR_LAUNCH_FAILED` + Xid 43 on
  several ranks at once, single-node EP.** Before reading it as a kernel or hardware fault, grep the
  full log for `OutOfMemoryError`.

    A rank-local OOM inside a distributed step abandons the DeepEP barrier its peers are waiting in
    (`DeepEP NVLink barrier timeout` lines). As its context tears down, the peers die on launch
    failures inside whatever kernel is on-stream. The OOMing rank's traceback is the primary failure
    and is easily buried under the collateral.

    The trainer logs a `RANK n: ... THIS RANK IS THE PRIMARY FAILURE` banner for it, and warns after
    the first optimizer step when a rank's peak sits above ~92% of its device.

    The failing rank exits without entering distributed teardown (both halves are themselves
    collectives, and its peers are still inside the step's own), so torchrun reaps the job in
    seconds rather than at the NCCL watchdog.

    On MoE the usual cause is routing skew. A cold (unbalanced) router concentrates dispatch buffers
    and expert activations on hot ranks, and under gradient checkpointing every MoE layer's saved
    dispatch/combine results scale with it.

    Under `bias_update` the skew (watch `moe/load_max`) falls over the first few hundred steps, so a
    batch shape that OOMs at cold start can fit when resumed from a balanced checkpoint. Cold-start
    at the smaller batch, or lower `per_device_train_batch_size` / `max_length`.

- **`DeepEP receive counts disagree with the received expert ids` in a `device-side assert triggered`
  on one rank, its peers then timing out in DeepEP's barrier.** The dispatch's receive counts do not
  match the expert ids it delivered. See
  [DeepEP → Runtime errors](https://whitecircle.com/halo/docs/agent-docs/infrastructure/deepep.md#runtime-errors).

- **`Dispatch CPU wait ... received count 0`, or Xid 109 `CTX SWITCH TIMEOUT` → Xid 43 /
  `unspecified launch failure`, on cross-node EP (EFA).** The dispatch exceeded the proxy-GIN
  per-rank payload ceiling and wedged in transit. See
  [DeepEP → AWS EFA](https://whitecircle.com/halo/docs/agent-docs/infrastructure/deepep.md#expert-parallelism-over-aws-efa).

    The Xid 43 / launch-failure form surfaces in whatever kernel is on-stream (often the expert
    GEMM) and reads like a compute bug, but the trigger is the oversized dispatch. A declared
    `per_device_train_batch_size × max_length` (under PP, one microbatch's rows × `max_length`) over `8192` tokens/rank
    (`HALO_DEEPEP_GIN_MAX_TOKENS_PER_RANK`) is rejected at config time, before the model loads. That
    gate reads the train batch only, so an evaluation over the ceiling (`per_device_eval_batch_size`
    defaults to 8) passes it, starts training, and raises at the dispatcher's buffer-sizing check on
    its first oversized batch. A wedge with no raise means the ceiling was disabled or raised. Lower
    `per_device_train_batch_size`, `per_device_eval_batch_size` or `max_length`, or use
    `ep_scope=node` with DP across nodes.

`CUDA_DEVICE_MAX_CONNECTIONS=1` is baked into the images as `ENV` (the driver latches it at the
`deep_ep` import's `cuInit`, so a Python `os.environ` write is too late). Measured effect:
[DeepEP](https://whitecircle.com/halo/docs/agent-docs/infrastructure/deepep.md).

## Parallelism config rejections

Which axis combinations may run is an allowlist, checked before any rank math, and a refusal names
the mechanism that would break: [Supported combinations](https://whitecircle.com/halo/docs/agent-docs/parallelism.md#supported-combinations).

| Combo | Status | Use instead |
|---|---|---|
| An axis set outside the allowlist: `TP + CP`, `ETP + CP`, `TP + ETP` (with or without EP), PP with anything but EP or pure ETP | Rejected at config time, naming the mechanism | Pick a [supported combination](https://whitecircle.com/halo/docs/agent-docs/parallelism.md#supported-combinations): EP+TP or pure ETP instead of TP+ETP; PP+EP or PP with pure ETP |
| `pipeline_parallel_size > 1` with a trainer that does not support PP (online / async GRPO, SDPG, distillation, embedding) | Rejected at config time: `<Trainer> does not support Pipeline Parallelism (PP): <its PP_REFUSAL>` | Shard with EP/TP/ETP, or use a [PP-capable trainer](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md#launching) |
| `pipeline_parallel_size > 1` on an unregistered model: `<Backbone> is not registered for pipeline parallelism (PP_FAMILY_SPECS); registered: ...` | Rejected at the model split, on every rank before a weight loads | The family has no `validated=True` `PPFamilySpec`. Train it with EP/TP/ETP/FSDP2 instead, or use a [registered family](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md#model-support) |
| `pipeline_parallel_size > 1` with MoE balancing on the router aux loss: `<Model> balances its experts through the HuggingFace router aux loss (moe_balancing=… resolves to 'aux_loss', router_aux_loss_coef=…), which pipeline parallelism drops` | Rejected at the model split | A stage never runs the `*ForCausalLM.forward` that adds the aux loss. Use `moe_balancing: bias_update` where the family's EP layers export the bias (the error names it first), else `none`; `bias_update_transient` is refused under PP ([PP + EP](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md#pp--ep-moe)) |
| `The MTP head fit failed its acceptance gate: eval_mtp_acceptance_… is …, under …` and a non-zero exit | The fit ended below its first held-out reading plus `mtp_min_acceptance_gain`, whatever its loss did; the named checkpoint holds the rejected head | Serve the head the fit started from, or refit at a lower `learning_rate` with `warmup_steps` ([Acceptance](https://whitecircle.com/halo/docs/agent-docs/optimization/mtp-heads.md#acceptance)) |
| `pipeline_parallel_size > 1` on a tied checkpoint: `… ties its output head to its input embedding (tie_word_embeddings)`; on a refused family: `… is refused for pipeline parallelism` (Zaya, Gemma 4); with built MTP layers: `… declares … and its decoder list holds the extra layers` | Rejected on the meta shell, on every rank before a weight loads | An untied checkpoint (Qwen3 8B and up), EP/TP/ETP without PP for a refused family, or every MTP count field at 0 ([Model support](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md#model-support)) |
| A stage boundary or split the family cannot take: `pipeline stage … starts at layer …, which is not a whole number of periods`, `DeepSeek-V4 hash_moe layer(s) … fall outside pipeline stage 0`, `GLM-5 pipeline stage boundary at layer(s) … severs a DSA indexer chain`, `PP stage …/… carries no EP-patched modules` | Rejected at the split | Choose a `pipeline_parallel_size` or `pipeline_split` that keeps the boundary on the pattern period, the state on one stage and a MoE layer on every stage ([Split-offset constraint](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md#split-offset-constraint)) |
| PP batch shapes: `Pipeline parallelism requires a fixed sequence length`, `Packing under pipeline parallelism needs one packed row per microbatch`, `… (padding-free) is incompatible with pipeline parallelism`, `per_device_eval_batch_size (…) must equal per_device_train_batch_size`, `A paired (preference) batch of … cannot be split into … microbatches` | Rejected at trainer construction, or at the first split batch | The P2P shapes freeze on the first step: set `max_length`, one packed row per microbatch, an eval batch size equal to the train one, and a batch that is a multiple of `2 × pipeline_microbatches` on paired rows ([Configuration](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md#configuration)) |
| `torch …'s FSDP2 state is missing …`, or `torch …'s … is missing …` under PP | Raised when the runner is built | `PipelineRunner` drives private torch members. Use the image's torch; a torch upgrade re-validates PP first ([Constraints](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md#constraints-inherited-from-torchdistributedpipelining)) |
| Any other `… under pipeline parallelism` refusal (`save_sharded_ep`, `torch_compile`, `activation_offloading`, `optim: muon` / `flash_adamw`, `push_to_hub` on per-node storage, HSDP, image data, a live `ref_model`, DPO/KTO without precomputed columns, `compute_metrics`, reentrant checkpointing) | Rejected before the split, each naming its mechanism and remedy | The full list with each reason: [PP → Rejected combinations](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md#rejected-combinations) |
| `world_size / pipeline_parallel_size` not a multiple of the NVLink domain | Rejected at config time | Pick a `pipeline_parallel_size` dividing the NVLink-domain count, so every stage is a whole number of domains |
| `fsdp_defer_grad_sync: true` with TP at `data_parallel_size == 1`, or with `fsdp_reshard_after_forward: true` (ZeRO-3) | Rejected at config time | TP at DP=1 applies no FSDP2 wrap — drop the flag. Under ZeRO-3 the deferred reduce holds the full unsharded gradient FULL_SHARD exists to shard: keep the flag on ZeRO-2 (`fsdp_reshard_after_forward: false`), or drop it |
| `moe_balancing: aux_loss` on a model whose `forward` omits `output_router_logits`, with `router_aux_loss_coef > 0` | Rejected when the perf callbacks build | HF's config fallback lives on that parameter, so the flag never reaches the aux term while still switching router-logit recording on — a `[tokens, num_experts]` plane per MoE layer, every forward. Use `moe_balancing: auto`, which resolves to the route the loaded class supports, or `bias_update_transient` where the architecture has no exportable bias slot (not under PP) |
| `moe_balancing: bias_update` rejected: "would train a routing bias that NO EXPORT CARRIES" | Rejected when the perf callbacks build | The family's architecture has no checkpoint slot for a selection bias (Qwen3, Qwen3.5/3.6, Qwen3-Next, Qwen3.8-Flash-Next, Mistral4, Cohere2 MoE) — the exported model would silently serve without the bias it trained with. Opt in deliberately with `bias_update_transient` (not under PP), or use `none` |
| Expert LoRA `lora_r` not a multiple of 8 | Rejected when the EP layer builds its adapters | `torch._grouped_mm` reads the adapter's rank dimension as a stride spanning a multiple of 16 bytes (`GROUPED_MM_STRIDE_ALIGNMENT_BYTES`), and the adapter GEMMs run at bf16 at the narrowest; any other rank fails at the first expert matmul with `strides should be multiple of 16 bytes`. Use a multiple of 8 |
| Expert LoRA + `expert_tp_size > 1` | Rejected at config time by `ParallelismConfig`, before the checkpoint downloads | The replicated adapter half gets partial gradients under expert TP and drifts across ranks. Use EP without ETP for expert adapters, or drop expert projections from `lora_target_modules` |
| `use_peft: true` with no attention target AND no EP layer that built expert adapters | Rejected at PEFT setup | No adapter would be created and the base stays fully trainable — a full fine-tune at the LoRA learning rate. A genuine expert-only run is fine: its list is empty only after the peel. Name the modules to adapt, or set `use_peft: false` |
| Expert projections in `lora_target_modules` + `use_dora` | Rejected before model load, at the expert peel | DoRA's magnitude decomposition has no grouped implementation, so it would apply to the attention half only. `use_rslora` **is** carried (expert scaling `alpha/sqrt(r)`). Drop `use_dora`, or drop the expert projections |
| `lora_target_parameters` under EP | Rejected at PEFT setup | PEFT would wrap the EP layer itself and swap it into its parent — adapters outside the EP gradient sync and outside both EP validators. Name the expert projections in `lora_target_modules` instead. Allowed without EP layers, where it is ordinary PEFT |
| Expert-only `lora_target_modules` + `lora_modules_to_save` | Rejected at PEFT setup | The model is never PEFT-wrapped, so those modules would stay frozen. Add an attention target (they route to stock PEFT, which owns `modules_to_save`), or drop `lora_modules_to_save`. Either way `merge_expert_lora_on_save` still produces a merged servable checkpoint — it folds the attention half of a mixed run too |
| Resuming expert adapters no EP layer can receive | Rejected in `load_expert_lora_adapters` | EP off, `use_grouped_gemm: false`, or the expert projections dropped — every saved expert delta would be discarded while the restore reported success. Resume with the run's expert configuration |
| LoRA/PEFT + TP (TP-only or EP+TP) | Rejected at construction | The adapter is not in the TP graph and trains rank-inconsistent. Native EP expert adapters are counted by their own check (`model_has_expert_lora`), so an **expert-only** EP+TP run is refused too. LoRA works under FSDP/CP/EP/ETP |
| LoRA/PEFT + PP: a `peft_config`, a model passed already PEFT-wrapped (`... the model passed to the trainer is already PEFT-wrapped`), or expert projections in `lora_target_modules` (`Expert LoRA (native grouped adapters) is not supported under pipeline parallelism`, at config time) | Rejected at trainer construction, on every rank before the stage split | Pass the plain model and fully fine-tune under PP, or train the adapter without PP. The adapter save and resume are not stage-aware ([PP rejected combinations](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md#rejected-combinations)) |
| `lora_target_modules` matching only modules no stock LoRA adapter fits | Rejected at PEFT setup | A wrapper holding its weight in a child module (a multimodal tower's `Gemma4ClippableLinear`) or a layer subclass whose forward is not its base layer's affine map (DeepSeek-V4's grouped `o_a_proj`) is excluded from injection; excluding every match would leave the adapter run training nothing. Name the backbone's own projections, or use `all-linear` ([PEFT](https://whitecircle.com/halo/docs/agent-docs/optimization/peft.md#targets-peft-cannot-adapt)) |
| CP + an unsupported trainer | Rejected at config time and again at trainer construction (`<TrainerClass> does not support Context Parallelism (CP)`) | `SUPPORTS_CP` is the gate (`reject_undeclared_axes`), not loss inspection. SFT, SMPO and offline GRPO (full fine-tuning) declare it; every other trainer is refused |
| Offline GRPO + CP + PEFT/native expert LoRA | Rejected at trainer construction | Full fine-tune under CP, or run adapters without CP |
| Offline GRPO full-FT KL + pre-sharded inputs, or supplied grouped `ref_per_token_logps` | Rejected before the model load (script) and at trainer construction | Use a finite, unsharded dataset and the trainer's persisted run-start reference ([reference contract](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/offline.md#reference-model)) |
| Offline GRPO KL resume from a checkpoint without `reference_logps.pt`, or without one of its splits (`Cannot resume offline GRPO from …`) | Rejected at trainer construction: rescoring would take the trained policy as its own reference | Restore the sidecar from a complete checkpoint, or recover it with the same data/settings and exact original model into a separate scratch run; copy `checkpoint-1/reference_logps.pt` into the resume checkpoint on every node with node-local output ([recovery](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/offline.md#reference-model)) |

TP and node-local EP must stay within one NVLink domain. EP can cross domains under
`ep_scope: global`. The full matrix is in [Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism.md) and the per-mode
guides: [Expert](https://whitecircle.com/halo/docs/agent-docs/parallelism/expert-parallelism.md) ·
[Tensor](https://whitecircle.com/halo/docs/agent-docs/parallelism/tensor-parallelism.md) · [Context](https://whitecircle.com/halo/docs/agent-docs/parallelism/context-parallelism.md) ·
[Expert-Tensor](https://whitecircle.com/halo/docs/agent-docs/parallelism/expert-tensor-parallelism.md) ·
[Pipeline](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md).

## Turning off a fused kernel

The toolkit's fused kernels, the MoE ones, the hyper-connection ones and the flex-sliding attention, have no runtime fallback: one that fails to compile or launch raises. A MoE or hyper-connection kernel raises at its first forward or backward. The flex-sliding attention compiles at model load when the world size is above 1, so it raises there, and on a single rank at its first masked call longer than the window or one tile. They are validated on B300; on other GPUs, turn off the one that fails.

- **Flex-sliding attention** (models with sliding-window layers or heads wider than 256, such as Step-3.7 and Gemma 4, on a run resolved to `sdpa`): `HALO_FLEX_SLIDING=0` sends every attention layer back to plain SDPA.
- **Masked matmul attention** (GLM-5 Next's DSA layers on `sdpa`, `halo/kernels/masked_softmax.py`, warmed at model load at world size above 1, otherwise compiled at the first backward): `HALO_DSA_MATMUL=0` keeps stock SDPA.
- **Block-sparse QSA attention** (`halo/kernels/block_sparse_attention.py`, Qwen3.8-Flash-Next): `HALO_QSA_KERNEL=off` keeps the tiled SDPA path; the default `auto` falls back on its own where the kernel cannot run.
- **Fused GLU** (`halo/kernels/fused_glu.py`, every expert and dense-MLP combine, grouped and loop paths alike): `HALO_FUSED_GLU=0` runs each combine through its eager form on CUDA too, and the EP layers' construction lines read `glu_combine=eager`. On CPU tensors, and for an activation the kernels do not compute exactly, the eager form runs anyway.
- **Hyper-connection kernels** (`halo/kernels/hyper_connection.py`, GLM-5 Next and DeepSeek-V4): `HALO_HC_KERNELS=0` runs upstream's forwards unpatched.
- **Gated-residual kernels** (`halo/kernels/gated_residual.py`, Qwen3.8-Flash-Next): `HALO_GATED_RESIDUAL_KERNELS=0` runs upstream's forward.
- **Native RMSNorm** (a Liger role): `liger_kernel_config: {rms_norm: false}` keeps the family's own norm, and `use_liger_kernel: false` turns off every Liger role.
- **Fused weighted un-permute and routing sort** (`halo/kernels/moe_permute.py` and the counting sort of `halo/kernels/moe_route_sort.py`, on every grouped-GEMM path): no switch of their own. `use_grouped_gemm: false` avoids both by running the per-expert loop instead of the grouped path. That costs throughput, most with many local experts per rank ([Grouped GEMM](https://whitecircle.com/halo/docs/agent-docs/optimization/grouped-gemm.md#grouped-vs-the-loop-path)).
