Troubleshooting
Symptom→fix lookup for OOM, NCCL hangs, DeepEP faults, parallelism-config rejections, NaN loss, and
attention-backend mismatches. Run /debug to route a live symptom to the exact helper.
A step that is slow rather than broken is a bottleneck question: read power, not util %
(GPU Training Theory §11).
The trainer-side weight-sync errors are rows below; server-side symptoms are on Rollout Servers, and Ray-actor, rollout and environment symptoms on Async GRPO → Troubleshooting.
First things to check
Most "it won't start" reports are environment, not code:
| Check | Why it bites |
|---|---|
Right image for the GPU. halo:blackwell on B200 (SM100) / B300 (SM103), halo:hopper on H100/H200 (SM90). | The host has no usable Python — PyTorch, DeepEP, and Flash Attention live only inside the image. FA3 is Hopper-only, FA4 Blackwell-only; the wrong image gives import errors or kernel hangs. |
--env-file .env passed to docker run. | The code does not auto-load .env. Without it, WANDB_API_KEY / HF_TOKEN / AWS_DEFAULT_REGION are unset, so a run that reports to W&B, pulls a gated repo, or reads an s3:// dataset fails at that step — each needs your own account. Start from .env.example. |
No poetry run / uv run prefix. | The image installs deps into the system interpreter, so python / torchrun / accelerate / pytest run directly. |
| The scratch mount is actually large. | The root filesystem is often small and shares the Docker image pool; a path named /mnt is not guaranteed to be a separate volume. Check findmnt / df -h, then point HF_DATASETS_CACHE, TMPDIR, HALO_DATA_ROOT, and logs at whichever mount has capacity. |
Symptom → cause → fix
| Symptom | Cause | Fix | Deep page |
|---|---|---|---|
torch.OutOfMemoryError: CUDA out of memory | Activations + weights + optimizer states exceed VRAM. | Gradient checkpointing on, lower per_device_train_batch_size / max_length, or move to a sharded mode. See OOM. | Debugging §3 |
CUDA out of memory after step 2 under gradient_checkpointing_plan: auto | The measured step did not hold the run's largest footprint: a pack fuller than any the step measured, EP routing drift past the 75% budget, or fragmentation growing past the measured ratio. | Relaunch into the same output_dir (resume or not): the unfinished plan's record makes the next plan checkpoint at least one step more. A timestamped output_dir gets a new directory, so no floor: pin it. gradient_checkpointing_plan: off keeps full checkpointing. | GradientCheckpointingPlanCallback |
gradient_checkpointing_plan: auto is refused for this run: … at startup | The plan relaxes per-layer checkpointing from one measured step of packed rows; the message lists every reason this run cannot take one: pipeline parallelism, torch_compile, auto_find_batch_size, a trainer that generates rollouts, unpacked or multimodal rows, activation offload, an explicit every_n_layers, model-level checkpointing, or checkpointing off. | Remove what the message names, or set gradient_checkpointing_plan: off. | GradientCheckpointingPlanCallback |
gradient_checkpointing_plan: auto measures CUDA allocator readings and needs a GPU, or The checkpointing plan record names every_n_layers=… at startup | The plan reads CUDA allocator stats, so a CPU run has nothing to measure. The record in output_dir names a policy the plan does not choose from at this layer count: another run's file, or an edited one. | Run on a GPU, or set gradient_checkpointing_plan: off. Remove a stale _halo_run_plan.json from output_dir. | GradientCheckpointingPlanCallback |
| Job hangs at step 0 or first cross-rank sync, no error, then a watchdog timeout | Collective mismatch — one rank issued a collective another never reached (divergent control flow, a straggler still in the dataloader). | Dump every rank's stack; the rank not in a collective is the culprit. See NCCL hang. | Debugging §4 |
| Every GPU at 100% utilization but idle power, all ranks spinning a CPU core, no NCCL error | A per-rank backward graph: a masked or empty row left disconnected from the loss let autograd prune that row's backward on one rank, so its FSDP2/EP collective never fired. | Keep every row connected to the loss (a value masked downstream is fine). Confirm with the NCCL flight recorder — mismatched collectives at one collective_seq_id. | Debugging §4 |
trains or takes a gradient under reentrant gradient checkpointing, and takes no floating tensor at a training forward | A checkpointed layer holding a trainable parameter, or taking a gradient in another argument, gets no float positionally and no tensor as its first argument, so a reentrant checkpoint could give its outputs no gradient. Every family on the roster passes one. | Pass the hidden state positionally in the layer's caller, or train without gradient checkpointing. | Data Parallelism |
None of the inputs have requires_grad=True. Gradients will be None (torch UserWarning) on an image batch | A frozen tower's checkpointed block under reentrant checkpointing, a whole frozen tower or the blocks before the first trained one, with nothing to train. | None needed. A block holding an adapter or a trained parameter does not warn. | Data Parallelism |
This rank's batch holds no image while the vision tower trains under data parallelism at a training forward | Under accelerate launch, or with a frozen input embedding beside a trainable tower, gradient checkpointing off, and a trained decoder layer or ZeRO-3 (unfreeze_layers_patterns matching the tower but not the embedding, or an adapter reaching the tower, which the message names), a text rank's reduces cannot pair with an image rank's, and the step would hang to the collective timeout. | Launch with torchrun; or train the embedding, or turn checkpointing on; or drop the text-only rows. A fine-tune can also set freeze_vision_tower: true or train adapters; an adapter run, which refuses that knob, keeps lora_target_modules off the tower. | Data Parallelism |
ValueError at startup naming expert_parallel_size=N on a single M-GPU NVLink domain forms K concurrent >2-rank DeepEP dispatch groups (after the parallelism config failed on … First (rank 0): prefix under torchrun) | Racy single-domain multi-group EP: one NVLink domain, ep_size > 2, and ep_group_size smaller than the domain (ep_size=4 on 8 GPUs). The combine barriers race FSDP2's DP-wide collectives, so the shape is refused at config time. | ep_size=2, ep_size = the domain, or raise ep_size × expert_tp_size to the domain (ep4 + etp2 on 8). Attention TP leaves ep_group_size unchanged, so ep4 + tp2 is refused too. | Expert Parallelism |
EP capacity dedup: a MoE layer dispatched N tokens/rank, over the capacity C cached at forward generation G on one rank, its peers stuck in the dispatch | The forward reuses the first MoE layer's all-reduced capacity for every later layer, and a layer outgrew it: either the model's later MoE layers dispatch more tokens than its first, or this forward entered the backbone directly and opened no capacity scope of its own. The raising rank leaves its peers in the dispatch collective until HALO_DEEPEP_GPU_TIMEOUT_SECONDS, so it reads as a hang plus one traceback. | A caller that peels the backbone off the wrapper calls bump_forward_generation() once per forward (the chunked log-prob path does). Otherwise HALO_EP_CAPACITY_DEDUP=0 sizes every layer with its own all-reduce — at one arena per layer. | DeepEP |
An expert id outside the local experts at ep_size 1, where the routing hands over only local ids in a device-side assert triggered (ep_size: 1, grouped GEMM) | At ep_size 1 nothing is dispatched, so every expert id the routing hands the expert compute must lie in [0, num_experts): the routing sort counts the ids in that range and asserts they cover every slot, since an ID outside would misalign the expert groups. A -1 is not padding here; only DeepEP pads with it. The ID came from the family's routing (a custom router, a changed top-k) or a routing-replay mask row; engine masks are range-checked when they are assembled. | Not a DeepEP fault and not a config problem: report it with the model, the parallelism shape and whether routing replay was on. | Grouped GEMM |
ImportError / NVSHMEM / undefined symbol on import deep_ep | Wrong NVSHMEM package, or DeepEP used without a GPU / the right image. | Run inside the image (DeepEP faults). | DeepEP |
FlashAttention-4's head_dim-256 kernel is not validated with a deterministic backward at load — a full_determinism run of Qwen3.5 / Qwen3.6 / Qwen3-Next or any other head_dim-256 model FA4 otherwise trains | full_determinism sets FLASH_ATTENTION_DETERMINISTIC=1, which transformers forwards to the dedicated head_dim-256 backward, whose deterministic mode is not validated here. | Auto-handled: resolve_attn_implementation demotes FA4 to SDPA (to FA2 under Context Parallelism, the kernel Ulysses runs then) and says so; SFT packing and padding_free rows on a GatedDeltaNet decoder load FA2 instead, an explicit attn_implementation: flash_attention_4 included; elsewhere an explicit FA4 is demoted to SDPA too. Drop full_determinism to train on FA4. | Flash Attention |
FlashAttention-4 trains head_dim-256 attention on a dedicated backward kernel: it takes no attention logit softcapping (or no sliding-window attention, or multi-head latent attention at this width stays on FA2 as a precaution) at load — Gemma 2 / Gemma 3 at head_dim 256, GLM-4.7-Flash | FA4 trains a 256-wide query/key head only through its dedicated backward kernel, which asserts on a softcap and on a window; GLM-4.7-Flash's latent attention is kept on FA2 as a precaution. | Auto-handled: the resolver demotes them to FA2, which takes all three; SDPA is never chosen for a softcapped model, since it drops the softcap. Nothing to do; a host with no FA2 build refuses instead, naming flex_attention and eager. | Flash Attention |
FlashAttention-4's varlen kernels are not validated with attention logit softcapping at load (Gemma 2), or attn_implementation='flash_attention_4' cannot train this model on packed, padding-free or padded rows (Gemma 2 27B; the resolver moves a head_dim-256 pin to FA2) | FA4's varlen kernels are not validated with a softcap; packed, padding-free and padded rows (more than one per batch), training or evaluation, all reach them. | Unset: auto-handled, the run takes FA2. Pinned FA4: unset it or set flash_attention_2, or train unpacked at per_device_train_batch_size: 1 (and per_device_eval_batch_size: 1 where the run evaluates). | Flash Attention |
A softcapped attention call reached FlashAttention-4's varlen kernels in the first forward (Gemma 2 on a pinned flash_attention_4, a padded script such as DPO, KTO, SMPO, reward, classification or GRPO) | The batch unpads to FA4's varlen entry point, which refuses a softcap (not validated there). The scripts outside SFT, self-distillation, teacher distillation and embedding read no row shape before the load, so the entry point refuses the call itself. | Leave attn_implementation unset (the padded scripts then take FA2) or set flash_attention_2. | Flash Attention |
full_determinism: true cannot run with the sdpa_flex_sliding attention at trainer construction (Gemma 4 or Step-3.7 on SDPA, world size > 1) | HF's Trainer turns on deterministic-algorithms mode after the model loaded, and the masked layers' FlexAttention graphs were compiled at load without it. Dynamo guards on that mode, so each rank would recompile at its own first long masked call while its peers wait in the next collective (a DeepEP dispatch times out). | HALO_FLEX_SLIDING=0 (plain SDPA), or drop full_determinism. | Flash Attention |
reset_sinks: true would neutralize the attention sinks of the GptOss teacher too at teacher distillation startup, before any weight loads, or, from a teacher built outside the script, The GptOss teacher was loaded with its attention sinks neutralized at trainer construction | reset_sinks applies to both models. The student trains away from neutralized sinks, but the frozen teacher never trains back: gpt-oss-20b without its sinks predicts ~6 nats/token (KL) from itself, so the student would fit a degenerate target. | Set reset_sinks: false: both models keep the sinks their checkpoints hold, frozen, on a sink-carrying backend (flash_attention_4 on Blackwell; flex_attention or eager). A teacher built outside the script loads with reset_sinks=False through load_frozen_auxiliary_model, on the same backends. | Teacher Distillation |
This GptOss model runs attn_implementation='…', which drops its attention sinks, while the sinks are live at model load, in any loader (the reward-model scorer and a sentence-transformers backbone included) | The built model's backend takes no sink column (FA2, SDPA, an FA3 build without s_aux) while its sinks are live, so every logprob would shift by nats against the served model. The check reads the built model, so it also catches a backend no resolver chose. | Load on a sink-carrying backend (flash_attention_4 on Blackwell; flex_attention or eager), or neutralize the sinks (reset_sinks: true in a training run). | GPT-OSS → Attention sinks |
full_determinism cannot run over an EP group spanning NVLink domains at startup, before the load | full_determinism on an EP group spanning NVLink domains. DeepEP's cross-domain (hybrid RDMA) dispatch has no deterministic mode: it places tokens in atomic claim order, so expert weight gradients change from step to step. | ep_scope: node with DP across domains (on a multi-node NVLink fabric, NVLINK_DOMAIN_SIZE set to its size), or drop full_determinism. | Expert Parallelism |
full_determinism is set but FLASH_ATTENTION_DETERMINISTIC is not, so this row plan would pick … from a custom script, before any weight loads | The script built its config without parse_script_args, which turns transformers' deterministic flash switch on as it parses a full_determinism config. plan_row_attention reads that switch to pick the attention, as the loader does, so planning before it is set would pick a non-deterministic run's backend for kernels the trainer then runs deterministic. | Parse the config through parse_script_args, or call enable_flash_attention_deterministic() (halo/models/attention/selection.py) before planning. | Configuration |
Async GRPO with Environments: episode/length_cutoff_in_call_turns climbs (turns cut while writing a tool call) | A turn reaches its token cap inside a tool call: either the call outgrew the turn's own cap (rollout_max_tokens), or the engine closed the turn's reasoning at its cap and the model carried the thought on inside the call's argument, as comment lines of the program (episode/thinking_cap_turns shows it starting; the run's judge term handles reasoning carried into programs: Code Contests). Both rollout drivers read the cap off usage.completion_tokens, so the turn is a length cut whatever finish the server reports: the salvaged call is never run, the turn takes the nudge-and-retry path, and its fragment trains only on a negative advantage (Untrainable turns). | Under rollout_max_answer_tokens an in-call cut after a forced close is the bound at work: the call carried the reasoning on in its comments; raise the bound only when honest calls outgrow it. Otherwise raise the cap that binds: rollout_max_tokens (on vLLM, the answer room: rollout_max_tokens less the reasoning cap, or rollout_max_answer_tokens where smaller) when the call itself outgrows the turn, the reasoning cap (rollout_max_thinking_tokens, else rollout_max_tokens less rollout_max_answer_tokens) when it cuts the reasoning short. | Native tool-use |
Async GRPO with Environments: episode/empty_turns climbs, episodes loop to max_turns while episode/length_cutoff_turns stays near zero | The model closes its reasoning and stops with neither a tool call nor an answer, below its budget. Each such turn is nudged and retried like an engine cut, and at max_length_cutoff_recoveries: null an episode can spend every turn this way. | Cap the recoveries (max_length_cutoff_recoveries, one cap for cuts and empty turns) and price them (length_cutoff_penalty); the turn itself trains only on a negative advantage, and the price charges the episodes that recover. | Native tool-use |
Async GRPO with Environments: completions fill with Error: <tool>: unknown argument '<name>'; its arguments are … | The call named an argument the tool does not declare: a misspelled name (<parameter=…> in the Qwen XML format), or arguments the gpt-oss parser could not read, which it passes on as '_raw'. The call is refused before it runs, its error naming the declared arguments; no tool budget is spent, though the turn is and tool_error_penalty applies. | A few are the model correcting itself on the next turn. Many mark a policy drifting off its call format: check entropy and the trainer-vs-sampler gap (sampling/logratio_mean). | Native tool-use |
Async GRPO with Environments: a vLLM reasoning cap can be enforced but the importance-sampling correction is off at trainer construction | A vLLM reasoning cap can bind (rollout_max_thinking_tokens, or rollout_max_tokens less rollout_max_answer_tokens) and the marker resolves, but the IS correction is off; the reasoning closes the engine forces are kept out of the loss only through its ratio. | Turn train_on_sampled_tokens and vllm_importance_sampling_correction back on (both default on). | Objective |
multiple-choice answer '…' is a digit string from exam_qa | A digit string names different choices as a 0-based index and as a 1-based label (ARC's answerKey), so the grader refuses to guess. | Convert the answer to a choice letter or an int index when preparing the dataset. | QA Benchmarks |
| NaN loss, other models, mid-run | LR too high for bf16, or a precision/grad-reduce mismatch. | AdamWBF16 with stochastic rounding is auto-on under bf16: true (FSDP/EP/TP). Lower the LR; set fp32_grad_reduce: true for a tighter grad reduce. | BF16 Optimizer |
Optimizer step N: the global gradient norm is nan; the update is skipped (warning; the logs carry nonfinite_steps_skipped, and a logging window with no applied step logs loss: nan) | One step's gradients went non-finite. The step was not applied: weights and optimizer moments are those of step N-1, and training continues. | Nothing, if it stays rare. A recurring one points at a data or numerics fault in the forward or backward of those batches. | Trainer Architecture |
Optimizer steps A-B: K consecutive steps had a non-finite global gradient norm, on every rank | More than max_consecutive_nonfinite_steps non-finite steps in a row. None was applied. | Find the cause (the batches of those steps, or the model's numerics), then resume from the last checkpoint. | Trainer Architecture |
The finite-weights check at optimizer step N failed on … rank(s) naming parameters that already hold NaN/inf before the update | The weights went non-finite before this step's gradients did: corrupted outside the optimizer update, which a skip cannot repair. | Resume from the last checkpoint. If it recurs at the same step, the batches of that step trigger it. | Trainer Architecture |
FSDP2 post-backward left undone after complete_fsdp2_post_backward (RuntimeError) after a backward | A shard group still holds an unreduced gradient, or stays gathered, after the trainer completed the post-backward that reentrant checkpointing skips for a layer recomputed more than once per backward. Either a backward ran outside the trainer's training_step, or the FSDP2 internals the completion drives changed with the installed torch. | Run every backward through training_step. After a torch upgrade, run tests/cpu/parallelism/test_fsdp2_post_backward_internals.py, which pins every member the completion touches. | Data Parallelism |
| Per-token loss degrades sharply past ~2048 tokens, or logprobs differ between the image and a host Python install | The NGC image defaults fp32 matmuls to TF32; TF32's 10-bit mantissa collapses adjacent RoPE token positions past 2048, corrupting the positional encoding wherever RoPE is a matmul: remote code such as Ling and Ring. transformers' own RoPE multiplies in the pinned release. | Auto-handled: configure_float32_matmul_precision pins fp32 matmuls to highest at model load and the image sets TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=0. Opt into TF32 with HALO_FP32_MATMUL_PRECISION=high. | Configuration |
LIGER_KERNEL_IMPL_ROPE=<value> selects a Liger rope implementation the toolkit does not run (or the same for LIGER_KERNEL_BACKEND_ROPE) at startup | The toolkit pins Liger's RoPE to its Triton kernel: Liger's CuTe DSL RoPE returns wrong results under CUTE_DSL_ENABLE_TVM_FFI, which the images set for FA4. | Unset the variable; only nvidia-triton (or triton) is accepted. | Liger Kernels |
The Liger applier for <model_type> rebound torch.nn.LayerNorm to <class> for the whole process (RuntimeError) at startup, before the model builds | An upstream Liger applier left torch.nn.LayerNorm rebound after the dispatch forced its layer_norm flag off, so every LayerNorm built afterwards (a teacher, a reference, a vision tower) would run Liger's class. | Set the flag that installs it to false in liger_kernel_config, or set use_liger_kernel: false. | Liger Kernels |
mean_token_accuracy reads 0 and entropy ln(vocab) in a training log, beside a sane loss | Liger's cross-entropy kernel wrote into fp32 logits the forward returned, before the metrics read them. | Read eval metrics that mask label -100; the loss and gradients are right. | Liger Kernels |
| OOM on Gemma 4 at long sequence, or FA2 rejects the model | Gemma 4's global layers have head_dim=512, above every flash kernel's limit; cuDNN SDPA rejects it too and the math kernel OOMs on the full score matrix. | Automatic: the run resolves to SDPA and is built as sdpa_flex_sliding. Sliding layers run FlexAttention; global layers run matmul attention within a 2 GiB per-layer score budget (about 4,700 tokens at batch 1) and, past it, mem-efficient attention per document on a packed row and mem-efficient SDPA on any other. Without gradient checkpointing the matmul global layers hold about 7 GiB more than SDPA at 4,096 tokens; HALO_FLEX_SLIDING=0 builds plain SDPA. Set attn_implementation: sdpa to skip the redirect warning. | Gemma 4 |
Hang inside ptxas at build/first kernel, Hopper, building FA2 | FA2's split-K forward kernels hang ptxas under CUDA 13.2 on SM90. | The Hopper image strips every flash_fwd_split_* source, links 24 throw-stubs in their place and pins the dense and varlen forwards to one split, so no forward Halo runs reaches a stub. Use the prebuilt image; building FA2 for Hopper outside Docker needs the same two changes or CUDA 13.1. | Flash Attention |
CheckpointError (recomputed tensor shape ≠ forward) under EP or any MoE with gradient checkpointing, often in GRPO | use_reentrant=False validates recompute shapes, and a recomputed router can flip a near-tie pick (the extra no-grad GRPO forwards make it likelier), changing the routing tensors' shapes. | Let the override stand: the trainer forces use_reentrant=True under EP, CP or any MoE (resolve_use_reentrant, halo/trainers/parallelism/ep_layers.py) with a warning at init, even over a configured false. Pipeline parallelism is the exception: it runs non-reentrant and refuses use_reentrant: true. | Expert Parallelism |
gradient_checkpointing_enable raises on Zaya | cuDNN Conv1d backward-recompute fault under CUDA 13.2; the toolkit clears the model's GC support flag so it fails at enable time rather than deep in the first backward. | Run Zaya as plain FSDP2 without GC, EP without GC, or EP+ETP without GC — the supported configs. TP and CP are unsupported for Zaya. | Zaya |
fp32_non_ep_params=True cannot combine with fsdp_shard_ep1_experts=True at model load (MoE, ep_size: 1) | The upcast skips every EP-layer parameter, so FSDP-managed replicated experts would stay bf16 inside an fp32 shard group and FSDP2 asserts a uniform original parameter dtype at the first forward. Rejected off config.json instead, before the process groups and the meta shell. | Set fsdp_shard_ep1_experts: false, or drop fp32_non_ep_params. Under TP/CP the first remedy is itself refused, so dropping fp32_non_ep_params is the only one; the message names whichever applies. | Bailing/Ling |
fp32_non_ep_params=True is not supported for gemma4/gemma4_text under Expert Parallelism at model load (ep_size > 1 only) | Gemma 4's norms re-emit activations at weight dtype, so the upcast feeds fp32 tokens into DeepEP's 2-byte transport. The gate turns that into a named refusal instead of a raw C++ assert at the first dispatch, after the whole multi-GPU load. | Train Gemma 4 experts-distributed in plain bf16; fp32_experts still applies. | Gemma 4 |
ValueError: ... must divide ... / Node-local EP group size (N) cannot exceed the NVLink domain (M) at startup | ParallelismConfig validation rejected the layout. | Fix the degrees so they divide world_size; keep TP and node-local EP within one NVLink domain. See Config rejections. | Parallelism |
text_only_model=True cannot load model_type=… at load | The family has no text-only *ForCausalLM sibling: transformers maps none, or the class it maps is the conditional-generation one (Gemma 4). | Set text_only_model: false and load the multimodal class. | Supported Models |
flash-attn is required for Context Parallelism at load | Ulysses attention calls a flash kernel directly, and neither the run's backend nor flash_attention_2 imports. | Run in the Blackwell or Hopper image. | Context Parallelism |
lowp_precision=… cannot run …: it contracts over … at load | The fake quant blocks both GEMM operands along the contraction axis, and this module's axis does not split into the format's blocks. | Lower the parallel size that shards the axis where the message names one, pick a lowp_precision whose block divides it, or set the opt-out flag the message names. | Low-Precision Kernels |
…: TP style … keeps a …-wide activation full width, which tensor_parallel_size=… does not divide at load, or … ran with its parameters FSDP2-sharded over … in a forward | A rowwise or gathering TP linear splits that width across the group, which needs an exact divisor. The second is a projection called around its FSDP unit's forward, with the (dp, tp) shard still registered, which would compute part of its output. | Pick a tensor_parallel_size that divides the width. Call the projection through the FSDP-wrapped module. | Tensor Parallelism |
… holds an external table …, The checkpoint in … holds no keys that load into …, The … parts in … hold … rows of widths … or … has no table attached on a Qwen3.8-Flash-Next load or save | The n-gram table attaches from the local directory the weights were read from. A model loaded outside the toolkit's loaders whose config records a Hub id has no revision to pair the table with, a config object names no checkpoint, and a directory without the table's keys or sizes does not match the config. | Load through a Halo loader, or from a local directory that holds the table, with the config.json it was saved with. | Qwen3.8-Flash-Next |
A qwen4_exp row whose position_ids restart at 0 (a packed row) arrived without seq_idx, seq_idx marks different documents than the position_ids restarts of the same row or Tiled QSA … in a Qwen3.8-Flash-Next forward | The per-document forwards read documents from seq_idx and padding from a 2-D mask, so a row without them, or with dropout or interleaved padding, would read across documents. | Pack with the toolkit's collators, which emit seq_idx for the family. Set attention_dropout to 0, and pad on one side. | Qwen3.8-Flash-Next |
The document-aware PLE conv reads the weights of …, Zaya's document-aware CCA reads … weights directly or Nemotron-H's Mamba mixer reads conv1d.weight directly at the first forward | These forwards read the conv's weights themselves, so an adapter on that conv would be bypassed. | Leave the conv (conv1d, or conv_qk_depthwise / conv_qk_grouped) out of lora_target_modules. | Qwen3.8-Flash-Next, Zaya, Nemotron-H |
trust_remote_code loaded … through the release's own modeling file, which cannot train here at load | The remote composite (Nemotron-H's) drops what this toolkit's rows read on the way to its text tower. | trust_remote_code: false, which reads the same checkpoint through transformers' classes, or text_only_model: true. | Nemotron-H |
A remote modeling file replaced transformers' conversion mapping for … at load | A hub modeling file (Nemotron-H's) registers its conversions with overwrite=True for the whole process, so a model built from transformers' own classes would load under names it does not have. | Load the family one way per process: every model with trust_remote_code, or none. | Correct the conversion mapping |
Expert Parallelism (…) needs routed experts or … needs a family EP layer at model load, before the process groups | An EP or expert-TP group on a dense model, or on a MoE with no family spec (Mixtral). No EP layer would wrap it, so the experts would stay replicated at a DP size that assumes them sharded. | Set expert_parallel_size and expert_tensor_parallel_size to 1, or add the family. | Adding a Model |
Expert Parallelism (…) on 'bailing_hybrid' needs the release's own modeling file at model load | trust_remote_code: false built transformers' native Ling 3.0 class, whose BailingHybridMoE no EP layer claims. | Set trust_remote_code: true and pin model_revision. | Bailing/Ling |
Loss near ln(vocab_size) from step 1, or a new release trains far worse than its siblings | The weights did not land where the class reads them. | Read rank 0's LOAD REPORT from from_pretrained: its UNEXPECTED, MISMATCH and CONVERSION lines name the keys. | Adding a Model |
| QLoRA rejected at model load under EP / TP / grouped-GEMM MoE | The EP lazy loader, the grouped-GEMM MoE loader and TP DTensor sharding all materialize plain de-quantized weights, so bitsandbytes Params4bit are lost and PEFT's 4-bit dispatch fails. An ep_size=1 MoE run hits this through the grouped-GEMM EP layers alone. | QLoRA on plain DDP/FSDP (either launcher; a MoE also needs use_grouped_gemm: false), or CP on a dense model; plain LoRA (no quantization) for EP. TP and EP+TP reject every adapter shape, native expert LoRA included. | Scale & Limits |
use_hsdp / fsdp_reshard_after_forward / fsdp_reshard_after_backward / fsdp_defer_grad_sync rejected at trainer construction under QLoRA | fully_shard cannot wrap bitsandbytes Params4bit, so QLoRA skips FSDP2 and syncs gradients with a post-backward all-reduce instead. The knobs shape a wrap that never happens — allowed through, a multi-node run asking for use_hsdp would silently train on flat replicated gradients. | Remove the flag(s), or drop the 4-bit quantization (plain LoRA or full fine-tuning) to get the sharding the config names. | PEFT |
Dataset <path>: the ranks … that must hold the same rows loaded different ones or The shard index of <path> differs across ranks, at dataset load on every rank | Ranks read their own copies of the source and the copies differ: a node serving an old per-node S3 cache while S3 is unreachable from it, a pre-staged directory synced at another time, another cached hub revision, or a re-prepare racing the launch. The row message names the ranks and nodes holding each version; the shard-index one lists each split's distinct digests and rank 0's. | Re-sync the stale copy, or delete it ($HALO_DATA_ROOT/s3_datasets/<md5> for an S3 source) so the node re-downloads, and relaunch. | Data Loading |
Dataset-map cache key cannot hash the content of tokenizer ... and keys on its name_or_path ... instead at the dataset map | Reading the tokenizer's content raised — a custom Python component the fast backend cannot serialize, or a remote-code get_vocab — so the map cache keys on the tokenizer's path. A resume that loads it from the run's checkpoint then re-runs every map, and a tokenizer changed in place at the same path reuses the previous run's rows. | Nothing to do for an unchanged tokenizer beyond the re-map on resume. After editing one in place, save it to a new path or clear HF_DATASETS_CACHE. | Data Loading |
| Embedding index / OOB crash right after the dataset map, wrong-model token IDs | A map whose output depends on a closure value the cache fingerprint cannot hash. Tokenizer identity and chat-template hash are already in the key, so a plain model swap cannot collide. | Look for the Dataset-map cache fingerprint ... skips a value of type ... it cannot fingerprint warning and thread that value through cache_key_extras; clear HF_DATASETS_CACHE only as a blunt fallback. | Data Loading |
Dataset path '…' is spelled as a HuggingFace Hub id, but a local directory of that name exists here, or Dataset … carries a ':config' or '@split' suffix from a dataset tool | Either reading of the first would be a guess, and the wrong one fails far from the cause. A tool with --config and --split reads a bare repo id. | Write ./org/name or an absolute path for the directory, or run where no such directory exists for the Hub dataset. Pass the bare org/name with --config / --split. | Dataset path formats |
… names its bucket and key itself; drop --bucket/--subfolder, or pass the bare key with them from s3.py | A full s3://bucket/key URI is taken as given, so a second bucket or subfolder contradicts it. | Pass one or the other. | S3 CLI |
The preprocessed rows were rendered with …, but this run ends with … at startup | The checkpoint exports the run's template, so the model would train on one template and be served with another. | Set chat_template to the rows' template with force_chat_template: true, or re-preprocess with the template the run exports. | Pre-Processing |
chat_template differs from the template … ships at startup | Without force_chat_template: true the run would keep the shipped template, train on it and export it. | Set force_chat_template: true, or remove chat_template. | Chat templates |
does not support these config fields (they would be silently ignored): ['chat_template_path'] from SFT or self-distillation | The rows are rendered before the trainer applies chat_template_path, so the checkpoint would export a template they never saw. | Set chat_template, with force_chat_template: true over a template the checkpoint ships and added_special_tokens for any tokens it needs. | SFT |
assistant_message_template … does not occur in this tokenizer's rendered chat template where a collator is built | Completion-only masking would mask every label and train zero tokens at a loss near 0. | Set the marker to the exact substring the template emits before assistant content, or set train_on_completions_only: false. | Collators → Constraints |
tree_packing needs: …, tree packing is not available for model_type=…, tree packing is verified on … or tree_full_attention='split' … before the load; the row is not a depth-first tree layout or a tree node starts on a trained token at collation | A requirement is unmet, or the family or backend is not admitted. The split form needs a gradient for the kernel's log-sum-exp, which only FA4 returns and its head_dim-256 kernel does not. The collation errors mean a row was cut below a tree. | Meet what the message lists; set tree_full_attention: gather for the split refusals. Keep packing_strategy: bfd, so rows reach the collator whole. | Tree Packing → Limits |
tree packing got an empty … split on at least one data-parallel rank, or Cannot pack the … split: it has no rows at the dataset map | The split has nothing left to pack after the dataset's own filters, or on a rank's share of it. | Check the split's size and test_size against the data-parallel degree. | Tree Packing |
dataset_ratio=… cannot apply to the pre-processed dataset at … at the dataset load | A pre-processed dataset fixed its rows at preparation, so the run would train all of them. | Unset dataset_ratio, or subset the source before preparing it. | Dataset Formats |
packing / padding_free are not supported for self-distillation at startup | The self-distillation collators pad one row per conversation in each branch, and OPD pairs those rows one to one. | Remove both from the YAML. | Self-Distillation |
Dataset '…' can be neither used nor tokenized at SMPO construction | The dataset holds neither the prepared columns nor the raw columns the tokenize map reads. | Pass the raw prompt, chosen, rejected rows (and images for a VLM run). | SMPO |
glm5-chat.jinja renders system, user and assistant text turns only at the dataset map | chat_template: jinja-templates/glm/glm5-chat.jinja, the GLM-5 training template, renders plain text turns only, so it refuses a tool definition, an assistant turn with tool_calls or a tool turn instead of dropping it from the row. | Render tool data with the hub template (leave chat_template unset), or drop the tool turns from the dataset. | GLM-5 Next |
packing=True on attn_implementation='sdpa' is refused for model_type=… (or eager, flex_attention) at startup — GPT-OSS, Bailing / Ling, GLM-5 Next | These families' forwards never pass position_ids into their mask construction, so on a dense backend a packed row runs as one causal sequence and its documents attend across each other. | Use a flash backend (GPT-OSS, Ling 2.0, Ring), or turn packing off. Ling 3.0 and GLM-5 Next have no flash backend: train on padded rows, or GLM-5 Next on tree_packing. | Collators |
packing=True is refused for model_type=…: its linear-attention layers receive no document boundary (or its depthwise causal short convolutions … receive no document boundary, or padding_free=True …) at startup — Ring on a flash backend, Inkling, or any hybrid whose linear-attention layers read no segment marker | The state of those layers (Ring's Lightning Attention-2, Inkling's short convolutions) carries from each document of a row into the next on every attention backend, and nothing in the loss shows it. | Turn packing / padding_free off, or set allow_cross_document_linear_attention: true where that mixing is acceptable. | Collators |
DPO on image data cannot train '<model>' (model_type='paligemma'): its decoder is a prefix LM (or KTO, SMPO) at startup, before any weight loads | PaliGemma's decoder needs token_type_ids marking each completion as the causal suffix, and no vision preference collator writes them: TRL's (DPO, KTO) calls the processor on the completions without images, which PaliGemma's raises on, and SMPO's marks the whole row as prefix, so each completion would score with its own next tokens in view. | Train PaliGemma on images with SFT or distillation, or run the preference method on text-only pairs. | Modality support |
VLM … holds a row without an image on a prefix-LM (PaliGemma) model in the collator, at the first batch holding such a row | PaliGemma's processor splices <bos> after each row's last <image>; in a row without one it lands inside the text, and a batch with no image raises inside the processor. | Drop the text-only rows from the image dataset, or train them in a separate text-only run. | SFT → Vision-language models |
The chat template renders <bos>, which this prefix-LM (PaliGemma) processor already puts after each row's last image where the VLM collator is built | PaliGemma's processor puts <bos> after each row's last image itself, so a chat template that renders one too (a Gemma template opens with {{ bos_token }}) would leave two in every row, and in every prompt it renders for serving. | Remove the {{ bos_token }} from the template and point chat_template at the edited copy, with force_chat_template: true when the checkpoint ships a template (without it a differing one raises). | SFT → Vision-language models |
<collator>: no assistant turn of the chat template ends in a turn terminator [...] where a VLM collator is built, under train_on_completions_only | A completion span ends at a turn terminator, the tokenizer's eos or the config's eos_token_id. A template that closes assistant turns with another token (a Gemma template on PaliGemma closes them with <end_of_turn>, its eos is <eos>) leaves every span open, so completion-only masking would train no token. | Set eos_token to the token the template closes an assistant turn with (eos_token: <end_of_turn> for a Gemma template), or set train_on_completions_only: false. | Collators → Constraints |
packing under pipeline parallelism is refused for model_type=… at startup | The model's conv / linear-attention layers read no document boundary at all (GLM-5 Next's KDA, Lightning Attention-2, Inkling's short convolutions), so their state carries across the documents of a packed row. GatedDeltaNet without its segment-aware kernels is refused as packing=True is refused for the linear-attention family. | Turn packing off (padded rows hold one document each), or train without PP. | Pipeline Parallelism |
OSError: AF_UNIX path too long in a SyncManager process, then EOFError out of Map (num_proc=N) | datasets.map with num_proc > 1 starts a manager that listens on an AF_UNIX socket under $TMPDIR (pymp-*/listener-*, 32 characters deeper). Linux caps a socket path at 108 bytes including the terminating NUL, so a TMPDIR longer than 75 characters fails. | Keep TMPDIR short: a top-level directory on the large volume ($D/tmp in the standard launch), or a short symlink to a deep one. | Docker |
NVRM: Xid ... 145, RLW_RXPIPE Nonfatal flooding dmesg, thousands per hour | A physically marginal NVLink lane. Forward error correction absorbs the errors, so the link keeps full bandwidth and no data is at risk — the Xid line is informational. | Run python scripts/profiling/nvlink_health.py: it exits 1 only on errors FEC did not absorb or a down link, and reports correctable churn as marginal. Escalate for a physical reseat only if a link turns unhealthy or codewords reach the deep FEC bins. | Debugging |
| A serving engine rejects or silently drops a MoE checkpoint's experts (missing per-expert keys) | The engine's loader reads only per-expert experts.{i}.* keys and the checkpoint holds the fused pair: an EP-gathered DeepSeek-V4 save on vLLM, or a Cohere2 MoE or DeepSeek-V4 one on SGLang. GLM-5 Next's gathered saves are per-expert, so only a checkpoint an earlier release saved with the fused pair hits it. GLM-4 MoE Lite (either engine) and LFM-2 (vLLM) hit it only with a checkpoint fused before Halo loaded it: the EP-gathered save writes both per expert, and the other writers keep the layout the load read. | Rewrite it with scripts/after_training/unfuse_moe_experts.py. | Checkpoints |
N model PARAMETER(s) reached device placement still on the meta device | The checkpoint load lost those tensors — a key the loader's disk→model mapping did not resolve, or state the checkpoint does not carry that nothing initialized. They hold no weights, and filling them would train uninitialized memory that differs on every rank. | Check the named keys against the checkpoint's index; load with ep_lazy_loading=False to route through from_pretrained. | Expert Parallelism |
N model BUFFER(s) reached device placement still on the meta device | Same gate, buffer side: a load path that skipped finalize_loaded_model, or a buffer none of its fixers recomputes. The lazy shell builds under init_empty_weights(include_buffers=False), so buffers computed in __init__ (a config-less rotary such as Qwen VL vision) arrive intact; reaching the gate is a loader or fixer gap, not an unsupported model or EP config. | Route the load through finalize_loaded_model, or add the family's buffer to the _ROTARY_FIXERS / _NON_PERSISTENT_FIXERS chain in halo/model_loading/buffer_repair.py. A zero inv_freq degenerates RoPE to NoPE at a plausible loss, which is why the gate is fatal. | Checkpoints |
Lazy load: tensor from checkpoint key(s) ... has shape ... but model tensor ... expects ..., or global expert index(es) ... are absent at an EP lazy load | The checkpoint and the config disagree: a patch_vocab-grown checkpoint paired with the base config.json, a changed intermediate_size or expert count, a truncated per-expert upload, or a stale hub rename. from_pretrained catches these itself; the lazy path re-implements the same gates. | Fix the pairing — load the checkpoint with the config.json it was saved with, or re-fetch a torn checkpoint. The raise is world-uniform (rank-consensus fenced), so the first error names the real cause. | Expert Parallelism |
N checkpoint key(s) align to no model tensor at an EP lazy load | Checkpoint keys the loader's mapping never claimed — the same causes as the row above, most often a stale hub rename. This one is a warning, not a raise, and it is not rank-fenced: those tensors are skipped and the run proceeds. | Read the named keys before ignoring it. Genuinely surplus keys (a checkpoint carrying extra state this architecture drops) are fine; a renamed weight here means the model tensor was filled by something else or left to the load-coverage gate. | Expert Parallelism |
N tensor(s) of <ModelClass> are absent from the checkpoint and were randomly initialized at model load | The checkpoint does not carry weights the live model needs — truncated, partially uploaded, or written for a different architecture. Without the gate the run would train part-pretrained, part-noise with no other symptom. | Check the named keys against the checkpoint's index and re-fetch it. A task head the architecture adds on top of the backbone, a tied lm_head and the class's own _keys_to_ignore_on_load_missing are excused already; HALO_ALLOW_MISSING_CHECKPOINT_KEYS=1 accepts the rest deliberately. | Checkpoints |
Qwen3.8-Flash-Next refused at startup: init_from_scratch builds … with random weights, but its external tables are frozen, … holds external tables …, and under accelerate-managed FSDP every save of it goes through transformers' save_pretrained, eos_token sets eos_token_id=…, but […] copied the checkpoint's eos id, ….save_pretrained writes its state dict, which holds none of its external tables, or, on the sentence-transformers embedding load (no EP, TP or grouped-GEMM EP layers), … builds its external tables (Qwen3.8-Flash-Next's n-gram table) in host memory only from the config its family patches stamp | The 95 GiB n-gram table lives outside the module state: only the checkpoint supplies it, only the toolkit's gathered writers save it, its embedding copies eos_token_id when it is built, and sentence-transformers builds the backbone from a config that carries no stamp. | Train from the checkpoint; launch with torchrun (or train adapters); leave eos_token unset; save through the trainer's save_model; the sentence-transformers embedding load (no EP, TP or grouped-GEMM EP layers) takes no such backbone. | Qwen3.8-Flash-Next |
Resolving the checkpoint '<path>' failed on K of N rank(s) at model load; the failing ranks raise '<path>' is not a directory on <host>, and this rank's Hub cache (...) holds no snapshot of it, or every rank names one whose cache holds the snapshot of '<repo>' at commit <c> without M of the N weight file(s) global rank 0's snapshot names | Some ranks cannot see the source the load names: a local checkpoint present on some nodes only, or a Hub snapshot that global rank 0 fetched into an HF_HOME the other nodes do not mount (a shared input declaration over node-local caches; there the config and tokenizer reads leave a weightless snapshot, which counts as missing). A fetching rank's own snapshot comes up short when its fetch ran offline (HF_HUB_OFFLINE) or against an unreachable hub, which serves whatever the cache holds. | Put the checkpoint, or HF_HOME, on a mount every node sees. Otherwise declare per-node input with DIST_INPUT_SHARED_FILESYSTEM=0, so each node's local rank 0 fetches its own copy. A short fetch: fetch with the hub reachable, or stage the complete snapshot. | Multi-Node |
The checkpoint '<repo>' each rank resolved differs across ranks at model load | Per-node Hub caches hold different commits of the repo: staged at different times and loaded offline, or a revision that moved between two nodes' fetches. The nodes would otherwise train different weights. | Pin the revision (model_revision, teacher_model_revision) to one commit, or refresh every node's cache to the same one. | Multi-Node |
'<path>' probes as multimodal on some ranks and text-only on others before the model load | The modality probe read the config on some ranks and fell back to the model name on others (a node that cannot see the source), or per-node caches hold different commits whose configs disagree. The ranks would build different model classes. | Make the same checkpoint readable on every node, and pin model_revision. | Multi-Node |
| CPU RAM spikes / OOM per node at model load | Batched per-node loading. Left unset it adapts to the node — min(4, max(1, local_world_size // 2)), so 4 ranks at once on an 8-GPU node and 2 on a 4-GPU tray. | Set max_concurrent_loading explicitly: toward 1 (fully sequential) on CPU-RAM-constrained machines; 0 (all-parallel) only with ample RAM. | Configuration |
net.ipv4.ip_local_port_range starts at N, which leaves K test worker(s) no port from a test | tests/common/ports.py hands out test ports between 20000 and the kernel's ephemeral range, so no kernel-picked port can take one; this host's range starts less than one port per xdist worker above 20000. | Move the range up to the Linux default: sysctl -w net.ipv4.ip_local_port_range="32768 60999" on the host (a --network host container shares it), or docker run --sysctl net.ipv4.ip_local_port_range="32768 60999" for a container with its own network. | Contributing |
The launcher declares world_size=N but ['MASTER_ADDR', 'MASTER_PORT'] is unset at startup | A bare srun --ntasks-per-node=N <script>: SLURM declares the world through SLURM_NTASKS but supplies no env:// rendezvous, so no process group can be built. | Launch one torchrun per node — srun --ntasks-per-node=1 bash -c 'torchrun --nnodes=$SLURM_NNODES --node_rank=$SLURM_NODEID …', quoted so each task expands its own node rank — or export MASTER_ADDR/MASTER_PORT identically on every task. | Launch Recipes |
--numa-binding is a torchrun option and needs a torchrun launch from halo launch | accelerate launch and python have no NUMA binding, so --accelerate or a single process refuses the flag. | Drop the flag, or launch more than one process on torchrun. | NUMA binding |
ddp_timeout=… would never apply at startup | The process group starts before the config is parsed, under the NCCL watchdog DIST_NCCL_TIMEOUT_MINUTES sets. | Remove ddp_timeout and set DIST_NCCL_TIMEOUT_MINUTES in the launch environment, alike on every node. | Environment variables |
Output-filesystem probe: … output_dir must be writable by global rank 0 at startup | The multi-node startup probe could not write its sentinel under output_dir. | Fix the directory's permissions or mount. | Filesystem Handling |
Output filesystem is declared SHARED but N of M ranks … cannot see a file global rank 0 wrote, or the PER-NODE mirror image, at startup | The multi-node startup probe wrote a sentinel under output_dir from global rank 0 and the declaration contradicts what the ranks see. Declared shared but invisible: only rank 0 writes trainer_state.json, so a resume would find every checkpoint incomplete on the other nodes. Declared per-node but visible everywhere: every node's local rank 0 would write the same checkpoint paths. | Match the declaration to the mount with DIST_OUTPUT_SHARED_FILESYSTEM (or the DIST_SHARED_FILESYSTEM umbrella), or move output_dir — onto the shared mount, or one per node. On a shared mount whose attribute cache outlasts the probe (actimeo/acdirmax above 60 s), raise DIST_FS_PROBE_TIMEOUT_SECONDS instead. | Filesystem Handling |
Input filesystem is declared SHARED but N of M ranks … cannot see a file global rank 0 wrote under HF_HUB_CACHE (or HF_DATASETS_CACHE), or the PER-NODE mirror image, at startup | The multi-node startup probe wrote a sentinel under each HF cache from global rank 0 and the declaration contradicts what the ranks see. Declared shared but invisible: the other nodes would read caches nobody filled. Declared per-node but visible everywhere: every node's local rank 0 would fill the same cache files. | Match the declaration to the caches with DIST_INPUT_SHARED_FILESYSTEM (or the DIST_SHARED_FILESYSTEM umbrella), or move the cache the refusal names — onto the shared mount, or onto each node's local storage. One declaration covers both caches, so HF_HUB_CACHE and HF_DATASETS_CACHE must be on the same kind of storage (an explicit HF_DATASETS_CACHE does not follow HF_HOME). On a shared mount whose attribute cache outlasts the probe, raise DIST_FS_PROBE_TIMEOUT_SECONDS. | Filesystem Handling |
Multi-Node NVLink prerequisite check … failed on N of M rank(s) at config time | NVLINK_DOMAIN_SIZE > gpus_per_node declares node-local groups that span OS nodes over NVLink, and some rank has no IMEX channels at /dev/nvidia-caps-imex-channels, reports a fabric registration other than COMPLETED, or sees no fabric clique at all. | Bring up NVIDIA Fabric Manager plus the IMEX service with matching channels on every node (NCCL >= 2.25.2), or drop NVLINK_DOMAIN_SIZE to gpus_per_node and keep the groups within one OS node. | Multi-Node |
Weight sync hangs at the first collective — the group formed, both ends idle, /health unanswered past the one-time init | The two containers drive different NCCL transports (NCCL_NET / NCCL_NET_PLUGIN differ) or different aws-ofi-nccl + libfabric builds (an upstream server image, a host-installed plugin against the image's): the pair is wire-sensitive and forms the group before hanging. | Serve from the Halo images and run the same fabric recipe on both ends — compose EFA overlay + make ... EFA=1, or both on the no-fabric defaults; python scripts/profiling/weight_sync_transport.py --server_url http://<server>:8000 --expect efa reports the transport and plugin build the group formed on. | Rollout Servers |
server ... is on a remote host ... but the NCCL weight-sync group address resolved to loopback at sync init | The trainer advertised a loopback master address, which the remote engine's workers would dial back to themselves. | Set VLLM_GROUP_HOST / SGLANG_GROUP_HOST, or the entry's group_host in rollout_server_configs, to the trainer IP on the subnet the server reaches. | Rollout Servers |
weight-sync group address ... is not a single address of this host that the server ... can reach at sync init | The advertised group address is one the trainer cannot bind alone (a NAT or port-mapped address, 0.0.0.0, or a hostname resolving to loopback for a remote server), and the unauthenticated rendezvous store listens only on the address it advertises. | Set VLLM_GROUP_HOST / SGLANG_GROUP_HOST, or the entry's group_host, to this host's IP on an interface the server reaches. Behind NAT or a port mapping, set HALO_WEIGHT_SYNC_BIND_ALL=1, on a network where only trusted hosts reach the group port. | Rollout Servers |
weight-sync group address ... has no IPv4 address on this host at sync init | The group address is an IPv6 literal or a name without an IPv4 address; the rendezvous store listens on IPv4 only, so IPv6 is unsupported. | Set VLLM_GROUP_HOST / SGLANG_GROUP_HOST, or the entry's group_host, to this host's IPv4 address on an interface the server reaches. | Rollout Servers |
Could not bind the weight-transfer group port at sync init | Another socket holds the configured port: a live client, or an outbound connection given it as its source port (inside net.ipv4.ip_local_port_range). Raised before the server is asked to join. | Give each server its own group_port, reserved or outside the ephemeral range. | Rollout Servers |
rollout_backend=... cannot serve model_type [...], or ... does not support weight sync, at startup or trainer construction | The engine release cannot take an online update for this family (the client's UNSERVABLE_MODEL_TYPES), or the family spec's weight_sync_refusal says why no sync into it is validated (refused on both engines). | Train it offline (offline GRPO needs no server, or SFT). Switching rollout_backend helps only against a client's UNSERVABLE_MODEL_TYPES entry, since the two engines list different families; a weight_sync_refusal holds on both. | Rollout Servers |
… resolves for model_type [...], but the model carries no live EP layer, … has no routed experts or … is a MoE with no Halo EP family at trainer construction | A family with an EP layer runs without one (use_grouped_gemm: false at ep_size: 1), a dense checkpoint sits under a spelling a MoE family claims, or the MoE has no family spec, so no engine layout covers its stock tree. | On a family with an EP layer, set use_grouped_gemm: true. Otherwise train it offline, or add the family. | Rollout Servers |
vLLM server … runs vLLM …; this client speaks the weight-transfer protocol of vLLM >= … at sync init | The client reads the server's /version first. A server older than 0.32.0, a 1.1.0 rollout image among them, speaks another weight-transfer wire. | Serve from this release's toolkit image (make build-vllm, or the published vllm-0.32.0 tag). | Rollout Servers |
use_adaptive_entropy is not supported, entropy_coef=… trains nothing with use_chunked_grpo_logprobs, … is an IterableDataset, which the DP-sharded loader this run uses cannot group into prompt groups or ref_model was passed, but this run holds no KL reference at online or async GRPO construction | A sharded resume does not restore TRL's adaptive coefficient. The chunked sweep returns a detached entropy. The prompt-group sampler needs a dataset with a length. A run at beta: 0, or a policy scored with its adapter disabled, reads no ref_model. | Set a fixed entropy_coef; set entropy_coef: 0.0 or turn the chunked sweep off; load a map-style Dataset; drop ref_model. | Online GRPO |
train_sampling_strategy='batch_rebalance' shards batches across every process at trainer construction | That sampler shards over every process, while TP, CP, ETP, PP and a pre-sharded dataset shard the loader over data-parallel replicas, so ranks sharing a batch would draw different rows. | Use random, sequential or group_by_length. | Data Loading |
length_ordered_microbatches: true, but … at construction, or TRL refilled its generation buffer without calling shuffle_sequence_dict at a round | The deal would change the objective or order nothing on this run, as the message says, or the installed TRL moved the shuffle the deal replaces. | Unset the option or set it to false. | Length-ordered micro-batches |
loss_forward_importance_weights: true, but … at construction, or The loss forward scored … time(s) with entropy / TRL's loss never read … at a micro-batch | A consumer needs the round's log-probs before the first backward, as the message names it, or the loss forward no longer reaches the scoring hook once or TRL moved where it reads the weight. | Set the option to false (the default), which keeps the no-grad pass. | Weight from the loss forward |
The rollout engine cannot enforce a thinking budget for model_type … at trainer construction (Mistral 4 on vLLM) | vLLM serves the family with its mistral_common tokenizer, which encodes [THINK] as plain text while the model writes it as a control token, so the budget never starts counting. | Unset rollout_max_thinking_tokens and rollout_max_answer_tokens, and steer with reasoning_effort. | Reasoning Budgets |
GRPOConfig 'vllm_group_port' is not read by environmental GRPO at construction | Async GRPO's weight-sync group port is AsyncRolloutConfig's, on either backend. | Set rollout_group_port. | Weight synchronization |
isr_engine_rescore_per_server set with isr_engine_reference off, or reasoning_terms_in_failed_groups set with reasoning_price and reasoning_floor both off at config time | The option modifies another that is off, so it would change nothing. | Set the option it modifies, or remove it. | AsyncRolloutConfig |
scratch_allowlist, scratch_env and episode_scratch give a spawned server its scratch directory, episode_scratch scopes the server's scratch directory to the episode, but this server takes none, server_env sets …, which scratch_env would point into the scratch directory or env_vars sets …, one file every episode's own server would read and write at once at MCP environment construction | An sse server is already running, so nothing spawns it into a scratch directory. A server without a scratch option has no directory to scope. The last two name one variable twice, or one file for every episode's server. | Use the stdio transport or drop the scratch options; drop episode_scratch; drop the variable from one of the two. | MCP |
judge term(s) … list 'checks: …', the checks the environment declares, judge check(s) … stand exempt under episode facts or context_file … is not a file at config or environment construction | The environment declares no checks to read, a check's unless names a fact the environment never establishes, or the path does not resolve (a relative one reads from the launch directory). | List the checks in the term, fix the unless, or use an absolute path. | Judge → Environment checks |
per_device_train_batch_size * steps_per_generation (N) must be divisible by num_generations at construction | Async GRPO groups advantages rank-locally, so a generation group cannot straddle ranks. TRL checks only the global generation batch. | Size per_device_train_batch_size × steps_per_generation (which defaults to gradient_accumulation_steps) to a multiple of num_generations. | Async GRPO |
Could not verify the head path of <ModelClass> (...) at GRPO trainer construction with use_chunked_grpo_logprobs: true | The chunked log-probs compute logits from the hidden state, so the family's forward is first probed on a meta-device shell with a stubbed backbone and a stand-in output embedding. The probe could not run it; the parenthesized error says why (an output embedding that is not a plain nn.Linear, a class with no backbone below it, a forward reading other model state on its head path). | Set use_chunked_grpo_logprobs: false. For a family you are adding, make its head path pass the probe. | Adding a Model |
<ModelClass>'s forward transforms its head path beyond what is declared for it, at the same points | The forward applies something between the backbone and the logits (a hidden-state scale, a logit scale, a softcap, a vocabulary cut) that the family's HeadTransformSpec does not declare, so the hidden-state paths would score a different distribution than the model samples from. | Declare the transform on a HeadTransformSpec in halo/models/head_transform.py, or set use_chunked_grpo_logprobs: false. | Adding a Model |
…'s attention reads a token's row index, and offline GRPO under pipeline parallelism left-pads a batch's prompts at trainer construction (DeepSeek-V4) | Pipeline stages forward the collated batch whole, so a shorter row's tokens would score at the wrong positions. Off PP these families score each row alone. | Set per_device_train_batch_size and per_device_eval_batch_size to 1, or drop pipeline_parallel_size. | Offline GRPO → Row positions |
Offline GRPO evaluates a map-style datasets.Dataset or KL evaluation requires a finite tokenized datasets.Dataset from evaluate() | The sampler cuts rows by group and equalizes the batch counts across ranks, which a stream or a plain list cannot give it. | Pass a datasets.Dataset. | Offline GRPO |
The GRPO loss forward materializes full logits: … (RuntimeError) at GRPO trainer construction, or Another rank's full-logits plane cannot fit | With use_chunked_grpo_logprobs off, the loss forward holds a rows × width × vocab logits plane at 4 B a logit, and this device's free memory (or a peer's) cannot hold it. The message states the size and the batch-size and width settings behind it. | Set use_chunked_grpo_logprobs: true, or lower the batch size or the width setting the message names. | Memory and Throughput |
ncclP2pImportShareableBuffer ... invalid argument on the first update, The full weights of the ModelRunner are partially updated in the SGLang log | cuMem mismatch: SGLang's entry point sets NCCL_CUMEM_ENABLE=0 process-wide unless it is pre-set, and the trainer's NCCL has cuMem on. | NCCL_CUMEM_ENABLE=1 in the server container (docker-compose.sglang.yml sets it), then restart the server — it holds a half-written model. | Rollout Servers |
rollout_max_thinking_tokens or carry_reasoning refused with rollout_backend: sglang | Neither is available on SGLang: the trainer wires no SGLang thinking-budget mechanism (thinking_token_budget is a vLLM-only request field), and SGLang's forwarding of reasoning_content is unverified, so the reasoning carry is refused. | Drop the knob, or run that recipe against vLLM. | Async GRPO |
reasoning_effort_profiles['<level>'] sets reasoning_price without reasoning_budget at environment construction; reasoning_effort_profiles set [...] but the environment's reasoning_effort is unset, reasoning_price_cap set while no level the environment draws sets reasoning_price, reasoning_terms_in_failed_groups set while no drawn level prices reasoning and reasoning_floor is off, reasoning_floor charges a failed episode its shortfall under its level's reasoning_budget, but the drawn level(s) [...] set none above 0, the drawn level(s) [...] set reasoning_budget with no reasoning_price while reasoning_floor is off, rollout_chat_template_kwargs carries [...] at trainer construction | The reasoning terms read the episode's level profile: the price charges past the level's reasoning_budget, the floor under it. With no level drawn nothing reads them; the cap is read only while a drawn level prices reasoning, and the failed-group knob only while the price or the floor is on; the floor needs a budget above 0 at every level the environment draws (all three under random), and a budget neither its price nor the floor reads changes nothing; a profile key among the template variables would reach the template. | Add the level's reasoning_budget (0 prices every token); set reasoning_effort, or drop the profile keys; price a level, or remove the knob; give every drawn level a reasoning_budget above 0, or turn the floor off; price the level or drop its budget; move the key into reasoning_effort_profiles. | Construction checks |
reasoning_floor charges an episode that earned no objective, but the reward prices no success above zero, the reasoning terms are waived in a training group no episode of which earned the objective, but the reward prices no success above zero at trainer construction | No environment term, or one of weight ≤ 0: every episode reads as failed, so the floor would charge every episode and a price waived in failed groups would charge none. | Give the environment term a positive weight, or turn the floor off; for the price, reasoning_terms_in_failed_groups: true also clears it. | Construction checks |
save_sharded_ep=True rejected at trainer construction | The run has a shape whose per-rank shards merge_ep_shards.py could not merge (multiple EP groups, expert TP, CP, expert LoRA, a non-shared multi-node output filesystem, …); the error names it, and the full list is on the linked page. | Use the gathered save (save_sharded_ep: false) — it streams, so it costs no more host memory, and it needs no merge. | Checkpoints |
Cannot resume precompute_ref_log_probs from <checkpoint>, reference_logps.pt ... does not belong to this '<split>' dataset, or Regenerate the '<split>' reference log-probs for this run, at DPO/KTO construction on resume | The resume built the policy from the checkpoint, so a reference sweep would score the trained weights, and the checkpoint holds no saved columns for that split — or holds them for other rows, reference settings or log-prob precision. | Resume with the saving run's data and reference settings. Otherwise run the config one step from the base into the scratch output_dir the error names and copy its reference_logps.pt in (on every node when checkpoints are node-local); a checkpoint without the file can also take the previous checkpoint's. | DPO |
Exact resume warm-restarted although world_size is unchanged | A fingerprint field other than world_size changed — any of ep_size, expert_tp_size, cp_size, tp_size, pp_size, fsdp_shard_ep1_experts, optimizer_class, ep_scope, use_grouped_gemm, hsdp (use_hsdp), nvlink_domain_size, expert_replica_size (saved='<absent>' for a key the saved meta lacks). A pipeline-parallel run raises on any drift instead. | The warning names every differing field with saved=/current= — restore it, or accept the warm restart (weights and LR schedule still resume). | Checkpoints |
Restoring the per-rank optimizer state from <checkpoint> failed on N of M rank(s) [...] or Reading the per-rank optimizer shards from <checkpoint> failed on … at resume, on every rank | The shards match this run's topology, but restoring them failed on the listed ranks: a truncated or unreachable shard file, or a CUDA OOM inside set_optimizer_state_dict. Resuming without them would reset the optimizer moments while weights, step and LR schedule resume, so every rank raises. | Fix the cause the message quotes (each failing rank's log carries its error) and resume again. allow_optimizer_warm_restart: true resumes with a fresh optimizer instead. | Checkpoints |
Optimizer resume from <checkpoint>: optimizer_meta.pt is missing beside the per-rank optimizer shards (or is unreadable), or … it holds a replicated optimizer.pt / optimizer.bin but no per-rank optimizer shards, at resume on every rank | An interrupted save (the meta is written only after every rank's shard), or an optimizer.pt a sharded resume cannot read: a non-sharded run's state, or an earlier release's sharded save stopped before its shards (a sharded save writes no optimizer.pt). Resuming past either would reset the optimizer moments while weights, step and LR schedule resume. | Resume from an earlier complete checkpoint, or set allow_optimizer_warm_restart: true to resume with a fresh optimizer. | Checkpoints |
Resume checkpoint(s) [...] are incomplete on at least one node: trainer_state.json, which a save publishes last, is missing at startup | A save stopped partway (a preemption, a full disk), or a node's copy is incomplete. resume_from_checkpoint: true passes over such a directory when an older complete checkpoint exists, and moves it into output_dir/_incomplete_checkpoints/ with a warning; an explicit path to it, or an output_dir holding only incomplete ones, raises. | Resume from an earlier complete checkpoint, or remove the incomplete one. | Checkpoints |
… present on some ranks but missing on others at …, … is present but unreadable or … parsed but carries no 'weight_map' at resume | A torn or partial checkpoint, or a per-node copy this rank reads mid-write. | Resume from a complete checkpoint. | Interrupted saves |
No node directory holds …, … and … carry different safetensors indexes, … differs in size across … or … vouches for … optimizer shards, but no input holds the optimizer shards of ranks … from merge_pp_stages.py | The inputs are not every node's directory of one pipeline-parallel save, or a copy is torn, or a node's optimizer save did not complete. | Pass every node's directory of that save and re-copy a torn one. For weights only, delete optimizer_meta.pt and every optimizer shard from the input. | Per-node export |
… names … itself: replacing it would delete the only copy, … is a fused 3-D expert tensor, but this checkpoint's model_type … matches no MoE family spec or … which a PEFT merge silently drops from a checkpoint tool | --output_dir is the source; the block-scaling axis of an unclaimed family is unknown; a native expert adapter reached the in-memory merge (convert_to_bf16.py --peft), which cannot fold its deltas. | Write to another directory; declare the family's MoEFamilySpec or pass --exclude; merge the adapter with merge_peft_adapters.py (it streams) or train with merge_expert_lora_on_save: true. | Post-training scripts |
Target modules <name> not found in the base model at PEFT injection, after the load, for a module the model has | lora_target_modules is a bare string, which PEFT reads as a regex fullmatched against each full module path, so a module name alone matches nothing. | Write it as a list (lora_target_modules: [o_proj]), which names modules at any length; keep a bare string for a regex or all-linear. | PEFT |
lora_target_modules entries [...] are regexes, but a list names modules at startup | A regex written into the list. PEFT compares list entries to module paths verbatim, so it would match nothing (unnoticed beside an entry that does). | Give the regex as a bare string in the YAML, or list the module names. | PEFT |
Growing deepseek_v4's vocabulary to <n> rows leaves gate.tid2eid without a row for the added tokens at tokenizer setup or from patch_vocab.py | The hash routers index that table by token id, one frozen expert assignment per row, so an added id has none: a forward on it would raise IndexError, and a grown export would not load. | Keep added tokens within the checkpoint's embedding rows. | DeepSeek-V4 |
Growing <ModelClass>'s vocabulary to <n> rows would leave its attached MTP head scoring the old output embedding at tokenizer setup | added_special_tokens runs past the checkpoint's rows, and the resize replaces the output embedding the head's term holds: part of the term would train a module no longer in the model, and an added id would raise IndexError. | Grow the vocabulary before the run with scripts/before_training/patch_vocab.py, which carries the head, and train from its output. | MTP Heads |
The PEFT adapter at <checkpoint> does not match the adapters this model builds at resume, on every rank (after the PEFT adapter resume from <checkpoint> failed on … prefix) | The saved adapters differ from those the resumed run builds, key for key or in shape: a changed lora_target_modules, lora_r, modules_to_save or added_special_tokens, a file without the expert half the run trains, or a short copy on one node. Every rank checks before any tensor is copied, so nothing was written. | Resume with the adapter settings the checkpoint was saved with, from a complete checkpoint. | Checkpoints |
added_special_tokens grew the vocabulary of an adapter run under accelerate-managed FSDP, or added_special_tokens grew the vocabulary, but PEFT's adapter selection leaves out the input embedding (or output head), at trainer construction | The added rows exist in no base checkpoint, so the adapter carries them for an exact resume and a complete merge. Under accelerate-managed FSDP the base Trainer saves the adapter with PEFT's own resized-embedding check, which reads the base config again and skips it offline or without the Hub; once lora_target_modules adapts one layer of an untied embedding pair, PEFT saves only the adapted layers' base weights. Either would leave the added rows out of every save. | Launch with torchrun (toolkit FSDP2), adapt both embedding layers or neither, or grow the vocabulary before the run with scripts/before_training/patch_vocab.py. | Checkpoints |
LoRA for embedding training is not supported with Expert Parallelism (EP or expert-TP) or on the grouped-GEMM EP layers at embedding startup, before the model loads (at trainer construction for a hand-built run) | The save folds injected LoRA over the backbone's own parameters. Under EP they hold each rank's expert shards; on a MoE whose family has an EP layer, the default use_grouped_gemm puts the grouped-GEMM EP layers on the backbone at expert_parallel_size: 1, whose parameters hold the layers' own expert layout. Either checkpoint would load nowhere. | Drop --expert_parallel_size / --expert_tensor_parallel_size and set use_grouped_gemm: false (LoRA then trains the stock experts), or full fine-tune. | Embedding |
<model> is a sentence-transformers pipeline holding [...] past its backbone at embedding startup, before any weight loads | Under EP or TP, and on a MoE an EP layer class wraps at the default use_grouped_gemm, the backbone loads through the parallel loader, whose pipeline rebuilds only the backbone, Pooling and Normalize; any other module (a Dense head) would be left out of training and the export. | Train where sentence-transformers loads the whole pipeline: no EP or TP, and use_grouped_gemm: false on a MoE (one GPU or accelerate DDP if a module outside the backbone trains). Or start from the bare backbone. | Embedding |
max_length=… reached the embedding trainer unresolved at a hand-built trainer's construction | sentence-transformers' collator truncates every text column to it, and only the embedding script resolves the context-window spelling. | Pass a positive length, or null to keep the model's max_seq_length. | Embedding |
pytest's basetemp is too deep for the AF_UNIX sockets the GPU tests bind from the GPU test launcher | An AF_UNIX socket path caps at 108 bytes, and datasets binds a multiprocessing socket (pymp-…/listener-…, 32 bytes) under TMPDIR while it tokenizes; the launcher reserves 40 bytes for it. It gives each test a TMPDIR under pytest's basetemp, which sits under the host's TMPDIR, and moves one too deep for the socket to a short directory beside it; when even that leaves too few bytes, it refuses the run. | Point TMPDIR (or --basetemp) at a shorter path on the scratch volume; the message names how many of the 40 bytes the current one leaves. | Contributing |
Multi-node: ranks never rendezvous, then ProcessGroup not initialized or an NCCL timeout at init | MASTER_ADDR is a public or management IP the fabric cannot route, the nodes sit on different fabrics, or one node's torchrun started outside the rendezvous timeout (10 min) or with different parallelism flags. | Use the fabric-routable address of node 0 and test its port from another node; launch every node with identical flags except --node_rank; kill stale torchrun processes. | RunPod |
RDMA connection timeout, or InfiniBand down (ibstat shows State: Down) | The node was provisioned without the fabric, or a link is down. | Verify the fabric (ibstat on IB/RoCE, fi_info -p efa on EFA), then re-run with NCCL_DEBUG=INFO for the transport logs. Without a fabric, use node-local EP (ep_scope: node) so only gradient sync crosses the network. | Multi-Node → RDMA fabrics |
| Resume finds no checkpoint on a multi-node run with per-node storage | Each node resumes from its own disk (DIST_SHARED_FILESYSTEM=0 forces save_on_each_node), so one node lacks checkpoint-* with its trainer_state.json at the same path. | Check every node holds the step directory at the same path. | Checkpoints |
torch.cuda.is_available() is False inside the container, or nvidia-smi lists no GPU | The container started without --gpus all, the Docker default runtime rejects that flag (sysbox-runc), or the host lacks the NVIDIA driver or Container Toolkit. | Start the container with --gpus all; add DOCKER_RUNTIME=nvidia to a make target on a sysbox host; check nvidia-smi on the host. | Docker |
make check-lock fails on the lock export | It exports uv.lock with --locked and installs it, plus an editable -e ., over the image's compiled torch, Flash Attention and DeepEP in a throwaway container, so a pyproject.toml the lock does not match fails. | Re-lock with uv lock (the image ships uv), not --upgrade, which drifts from the versions the Dockerfile compiles. Nothing make check-lock installs outlives the command: a dependency change takes an image rebuild. | Development Environment |
OOM
Peak memory is dominated by saved activations, not weights. Cut it in this order:
Gradient checkpointing on — trades recompute for activation memory. Turn it off only when the batch already fits; the recompute costs wall-clock.
Lower
per_device_train_batch_sizeormax_length— both scale activations linearly.Move to a sharded mode. Dense model that OOMs in FSDP2 → add node-local TP. MoE model that OOMs → EP (experts across ranks) or pure ETP (
ep_size=1, expert FFN sharded).
A large "reserved by PyTorch but unallocated" figure in the OOM message points at allocator
fragmentation rather than at the steps above: launch with
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. Under EP it costs throughput, so set it there only to fit
(measured).
To find what holds memory, dump a CUDA snapshot and drag it onto memory_viz (Debugging §3).
NCCL timeout / hang
A hang is almost always a collective mismatch: one rank issues an all-reduce / all-gather / barrier that another never reaches, so NCCL blocks until the watchdog fires.
Bash
python scripts/profiling/py_spy_diag.py dump # py-spy stacks for EVERY rank on the nodeAttaching needs ptrace access (Debugging §1c). Dumps land in $HALO_DATA_ROOT/profiling/py_spy/stacks, and
the rank not in a collective is the culprit.
For the definitive "which collective did rank K miss?", enable the NCCL flight recorder
(TORCH_NCCL_TRACE_BUFFER_SIZE, TORCH_NCCL_DUMP_ON_TIMEOUT).
One class of hang is refused up front instead: a job whose ranks disagree on an EP, timeout, save, attention
or matmul-precision knob (HALO_EP_CAPACITY_DEDUP, the DeepEP wire parameters, HALO_GRAD_BUCKET_MB,
HALO_CHECKPOINT_WRITE_THREADS, HALO_CHECKPOINT_WRITERS_PER_NODE, DIST_NCCL_TIMEOUT_MINUTES, DIST_STORE_TIMEOUT_HOURS,
DIST_FS_PROBE_TIMEOUT_SECONDS, FLASH_ATTENTION_DETERMINISTIC, HALO_FLEX_SLIDING,
HALO_FP32_MATMUL_PRECISION) raises
Rank-uniform toolkit environment in distributed setup, before the weight load — a per-node --env-file
rollout that reached only some nodes would otherwise diverge the collective counts from the second MoE layer
on, or load a head_dim-256 model on a different attention backend per rank, naming nothing.
Raise the watchdog for large cross-node all-to-all, or for a gathered save on storage that writes one
part (save_max_shard_size) slower than it: a gathered EP/TP save holds the non-writers in the next
chunk's gather while the writer flushes a part. The joins after a save's unbounded writes are exempt:
every rank's optimizer shard, a sharded EP save's per-rank shards, a gathered save's last part and
index and each pipeline stage's parts are joined over the c10d store (DIST_STORE_TIMEOUT_HOURS,
default 4 h). So are main-first work (model downloads, dataset load, tokenize/pack) and the
node-batched model load (max_concurrent_loading): the peers wait on the store, not inside a
collective.
Bash
export DIST_NCCL_TIMEOUT_MINUTES=60 # default 30; PyTorch's own default is 10Full procedure: Debugging §4.
DeepEP build / runtime faults
DeepEP is required for EP — there is no NCCL fallback. The recurring failure modes:
import deep_epfails (NVSHMEM / undefined symbol). On Blackwell + PyTorch 2.11+cu130, NVSHMEM ships asnvidia-nvshmem-cu13; installingnvidia-nvshmem-cu12clobbers the headers. Use the prebuilt image.Assertion exception ... != NCCL_GIN_TYPE_NONE/ "NCCL GIN is unavailable" at buffer init on cross-node EP (EFA). Proxy GIN could not come up on at least one node. See DeepEP → AWS EFA.Check, on every node: the
gdrdrvkernel module is loaded (ls /dev/gdrdrv; the host must install the gdrcopy driver), the container was started with--device /dev/gdrdrv, andNCCL_GIN_TYPE=2is exported. One node missing any of the three fails the whole job with this assertion while node-local EP on the same machine runs fine; NCCL's log names the gap asFailed to initialize GDRCopy.A host can load the module without creating the device node (
grep gdrdrv /proc/devicesprints a major number,ls /dev/gdrdrvfails). Create the node on the host, or inside the container: start it with--device-cgroup-rule='c <major>:0 rwm'instead of--device /dev/gdrdrv, and runmknod -m 666 /dev/gdrdrv c <major> 0beforetorchrun.CUBLAS_STATUS_EXECUTION_FAILED/ DeepEPcombine.hppCUDA_ERROR_LAUNCH_FAILED+ Xid 43 on several ranks at once, single-node EP. Before reading it as a kernel or hardware fault, grep the full log forOutOfMemoryError.A rank-local OOM inside a distributed step abandons the DeepEP barrier its peers are waiting in (
DeepEP NVLink barrier timeoutlines). As its context tears down, the peers die on launch failures inside whatever kernel is on-stream. The OOMing rank's traceback is the primary failure and is easily buried under the collateral.The trainer logs a
RANK n: ... THIS RANK IS THE PRIMARY FAILUREbanner for it, and warns after the first optimizer step when a rank's peak sits above ~92% of its device.The failing rank exits without entering distributed teardown (both halves are themselves collectives, and its peers are still inside the step's own), so torchrun reaps the job in seconds rather than at the NCCL watchdog.
On MoE the usual cause is routing skew. A cold (unbalanced) router concentrates dispatch buffers and expert activations on hot ranks, and under gradient checkpointing every MoE layer's saved dispatch/combine results scale with it.
Under
bias_updatethe skew (watchmoe/load_max) falls over the first few hundred steps, so a batch shape that OOMs at cold start can fit when resumed from a balanced checkpoint. Cold-start at the smaller batch, or lowerper_device_train_batch_size/max_length.DeepEP receive counts disagree with the received expert idsin adevice-side assert triggeredon one rank, its peers then timing out in DeepEP's barrier. The dispatch's receive counts do not match the expert ids it delivered. See DeepEP → Runtime errors.Dispatch CPU wait ... received count 0, or Xid 109CTX SWITCH TIMEOUT→ Xid 43 /unspecified launch failure, on cross-node EP (EFA). The dispatch exceeded the proxy-GIN per-rank payload ceiling and wedged in transit. See DeepEP → AWS EFA.The Xid 43 / launch-failure form surfaces in whatever kernel is on-stream (often the expert GEMM) and reads like a compute bug, but the trigger is the oversized dispatch. A declared
per_device_train_batch_size × max_length(under PP, one microbatch's rows ×max_length) over8192tokens/rank (HALO_DEEPEP_GIN_MAX_TOKENS_PER_RANK) is rejected at config time, before the model loads. That gate reads the train batch only, so an evaluation over the ceiling (per_device_eval_batch_sizedefaults to 8) passes it, starts training, and raises at the dispatcher's buffer-sizing check on its first oversized batch. A wedge with no raise means the ceiling was disabled or raised. Lowerper_device_train_batch_size,per_device_eval_batch_sizeormax_length, or useep_scope=nodewith DP across nodes.
CUDA_DEVICE_MAX_CONNECTIONS=1 is baked into the images as ENV (the driver latches it at the
deep_ep import's cuInit, so a Python os.environ write is too late). Measured effect:
DeepEP.
Parallelism config rejections
Which axis combinations may run is an allowlist, checked before any rank math, and a refusal names the mechanism that would break: Supported combinations.
| Combo | Status | Use instead |
|---|---|---|
An axis set outside the allowlist: TP + CP, ETP + CP, TP + ETP (with or without EP), PP with anything but EP or pure ETP | Rejected at config time, naming the mechanism | Pick a supported combination: EP+TP or pure ETP instead of TP+ETP; PP+EP or PP with pure ETP |
pipeline_parallel_size > 1 with a trainer that does not support PP (online / async GRPO, SDPG, distillation, embedding) | Rejected at config time: <Trainer> does not support Pipeline Parallelism (PP): <its PP_REFUSAL> | Shard with EP/TP/ETP, or use a PP-capable trainer |
pipeline_parallel_size > 1 on an unregistered model: <Backbone> is not registered for pipeline parallelism (PP_FAMILY_SPECS); registered: ... | Rejected at the model split, on every rank before a weight loads | The family has no validated=True PPFamilySpec. Train it with EP/TP/ETP/FSDP2 instead, or use a registered family |
pipeline_parallel_size > 1 with MoE balancing on the router aux loss: <Model> balances its experts through the HuggingFace router aux loss (moe_balancing=… resolves to 'aux_loss', router_aux_loss_coef=…), which pipeline parallelism drops | Rejected at the model split | A stage never runs the *ForCausalLM.forward that adds the aux loss. Use moe_balancing: bias_update where the family's EP layers export the bias (the error names it first), else none; bias_update_transient is refused under PP (PP + EP) |
The MTP head fit failed its acceptance gate: eval_mtp_acceptance_… is …, under … and a non-zero exit | The fit ended below its first held-out reading plus mtp_min_acceptance_gain, whatever its loss did; the named checkpoint holds the rejected head | Serve the head the fit started from, or refit at a lower learning_rate with warmup_steps (Acceptance) |
pipeline_parallel_size > 1 on a tied checkpoint: … ties its output head to its input embedding (tie_word_embeddings); on a refused family: … is refused for pipeline parallelism (Zaya, Gemma 4); with built MTP layers: … declares … and its decoder list holds the extra layers | Rejected on the meta shell, on every rank before a weight loads | An untied checkpoint (Qwen3 8B and up), EP/TP/ETP without PP for a refused family, or every MTP count field at 0 (Model support) |
A stage boundary or split the family cannot take: pipeline stage … starts at layer …, which is not a whole number of periods, DeepSeek-V4 hash_moe layer(s) … fall outside pipeline stage 0, GLM-5 pipeline stage boundary at layer(s) … severs a DSA indexer chain, PP stage …/… carries no EP-patched modules | Rejected at the split | Choose a pipeline_parallel_size or pipeline_split that keeps the boundary on the pattern period, the state on one stage and a MoE layer on every stage (Split-offset constraint) |
PP batch shapes: Pipeline parallelism requires a fixed sequence length, Packing under pipeline parallelism needs one packed row per microbatch, … (padding-free) is incompatible with pipeline parallelism, per_device_eval_batch_size (…) must equal per_device_train_batch_size, A paired (preference) batch of … cannot be split into … microbatches | Rejected at trainer construction, or at the first split batch | The P2P shapes freeze on the first step: set max_length, one packed row per microbatch, an eval batch size equal to the train one, and a batch that is a multiple of 2 × pipeline_microbatches on paired rows (Configuration) |
torch …'s FSDP2 state is missing …, or torch …'s … is missing … under PP | Raised when the runner is built | PipelineRunner drives private torch members. Use the image's torch; a torch upgrade re-validates PP first (Constraints) |
Any other … under pipeline parallelism refusal (save_sharded_ep, torch_compile, activation_offloading, optim: muon / flash_adamw, push_to_hub on per-node storage, HSDP, image data, a live ref_model, DPO/KTO without precomputed columns, compute_metrics, reentrant checkpointing) | Rejected before the split, each naming its mechanism and remedy | The full list with each reason: PP → Rejected combinations |
world_size / pipeline_parallel_size not a multiple of the NVLink domain | Rejected at config time | Pick a pipeline_parallel_size dividing the NVLink-domain count, so every stage is a whole number of domains |
fsdp_defer_grad_sync: true with TP at data_parallel_size == 1, or with fsdp_reshard_after_forward: true (ZeRO-3) | Rejected at config time | TP at DP=1 applies no FSDP2 wrap — drop the flag. Under ZeRO-3 the deferred reduce holds the full unsharded gradient FULL_SHARD exists to shard: keep the flag on ZeRO-2 (fsdp_reshard_after_forward: false), or drop it |
moe_balancing: aux_loss on a model whose forward omits output_router_logits, with router_aux_loss_coef > 0 | Rejected when the perf callbacks build | HF's config fallback lives on that parameter, so the flag never reaches the aux term while still switching router-logit recording on — a [tokens, num_experts] plane per MoE layer, every forward. Use moe_balancing: auto, which resolves to the route the loaded class supports, or bias_update_transient where the architecture has no exportable bias slot (not under PP) |
moe_balancing: bias_update rejected: "would train a routing bias that NO EXPORT CARRIES" | Rejected when the perf callbacks build | The family's architecture has no checkpoint slot for a selection bias (Qwen3, Qwen3.5/3.6, Qwen3-Next, Qwen3.8-Flash-Next, Mistral4, Cohere2 MoE) — the exported model would silently serve without the bias it trained with. Opt in deliberately with bias_update_transient (not under PP), or use none |
Expert LoRA lora_r not a multiple of 8 | Rejected when the EP layer builds its adapters | torch._grouped_mm reads the adapter's rank dimension as a stride spanning a multiple of 16 bytes (GROUPED_MM_STRIDE_ALIGNMENT_BYTES), and the adapter GEMMs run at bf16 at the narrowest; any other rank fails at the first expert matmul with strides should be multiple of 16 bytes. Use a multiple of 8 |
Expert LoRA + expert_tp_size > 1 | Rejected at config time by ParallelismConfig, before the checkpoint downloads | The replicated adapter half gets partial gradients under expert TP and drifts across ranks. Use EP without ETP for expert adapters, or drop expert projections from lora_target_modules |
use_peft: true with no attention target AND no EP layer that built expert adapters | Rejected at PEFT setup | No adapter would be created and the base stays fully trainable — a full fine-tune at the LoRA learning rate. A genuine expert-only run is fine: its list is empty only after the peel. Name the modules to adapt, or set use_peft: false |
Expert projections in lora_target_modules + use_dora | Rejected before model load, at the expert peel | DoRA's magnitude decomposition has no grouped implementation, so it would apply to the attention half only. use_rslora is carried (expert scaling alpha/sqrt(r)). Drop use_dora, or drop the expert projections |
lora_target_parameters under EP | Rejected at PEFT setup | PEFT would wrap the EP layer itself and swap it into its parent — adapters outside the EP gradient sync and outside both EP validators. Name the expert projections in lora_target_modules instead. Allowed without EP layers, where it is ordinary PEFT |
Expert-only lora_target_modules + lora_modules_to_save | Rejected at PEFT setup | The model is never PEFT-wrapped, so those modules would stay frozen. Add an attention target (they route to stock PEFT, which owns modules_to_save), or drop lora_modules_to_save. Either way merge_expert_lora_on_save still produces a merged servable checkpoint — it folds the attention half of a mixed run too |
| Resuming expert adapters no EP layer can receive | Rejected in load_expert_lora_adapters | EP off, use_grouped_gemm: false, or the expert projections dropped — every saved expert delta would be discarded while the restore reported success. Resume with the run's expert configuration |
| LoRA/PEFT + TP (TP-only or EP+TP) | Rejected at construction | The adapter is not in the TP graph and trains rank-inconsistent. Native EP expert adapters are counted by their own check (model_has_expert_lora), so an expert-only EP+TP run is refused too. LoRA works under FSDP/CP/EP/ETP |
LoRA/PEFT + PP: a peft_config, a model passed already PEFT-wrapped (... the model passed to the trainer is already PEFT-wrapped), or expert projections in lora_target_modules (Expert LoRA (native grouped adapters) is not supported under pipeline parallelism, at config time) | Rejected at trainer construction, on every rank before the stage split | Pass the plain model and fully fine-tune under PP, or train the adapter without PP. The adapter save and resume are not stage-aware (PP rejected combinations) |
lora_target_modules matching only modules no stock LoRA adapter fits | Rejected at PEFT setup | A wrapper holding its weight in a child module (a multimodal tower's Gemma4ClippableLinear) or a layer subclass whose forward is not its base layer's affine map (DeepSeek-V4's grouped o_a_proj) is excluded from injection; excluding every match would leave the adapter run training nothing. Name the backbone's own projections, or use all-linear (PEFT) |
| CP + an unsupported trainer | Rejected at config time and again at trainer construction (<TrainerClass> does not support Context Parallelism (CP)) | SUPPORTS_CP is the gate (reject_undeclared_axes), not loss inspection. SFT, SMPO and offline GRPO (full fine-tuning) declare it; every other trainer is refused |
| Offline GRPO + CP + PEFT/native expert LoRA | Rejected at trainer construction | Full fine-tune under CP, or run adapters without CP |
Offline GRPO full-FT KL + pre-sharded inputs, or supplied grouped ref_per_token_logps | Rejected before the model load (script) and at trainer construction | Use a finite, unsharded dataset and the trainer's persisted run-start reference (reference contract) |
Offline GRPO KL resume from a checkpoint without reference_logps.pt, or without one of its splits (Cannot resume offline GRPO from …) | Rejected at trainer construction: rescoring would take the trained policy as its own reference | Restore the sidecar from a complete checkpoint, or recover it with the same data/settings and exact original model into a separate scratch run; copy checkpoint-1/reference_logps.pt into the resume checkpoint on every node with node-local output (recovery) |
TP and node-local EP must stay within one NVLink domain. EP can cross domains under
ep_scope: global. The full matrix is in Parallelism and the per-mode
guides: Expert ·
Tensor · Context ·
Expert-Tensor ·
Pipeline.
Turning off a fused kernel
The toolkit's fused kernels, the MoE ones, the hyper-connection ones and the flex-sliding attention, have no runtime fallback: one that fails to compile or launch raises. A MoE or hyper-connection kernel raises at its first forward or backward. The flex-sliding attention compiles at model load when the world size is above 1, so it raises there, and on a single rank at its first masked call longer than the window or one tile. They are validated on B300; on other GPUs, turn off the one that fails.
- Flex-sliding attention (models with sliding-window layers or heads wider than 256, such as Step-3.7 and Gemma 4, on a run resolved to
sdpa):HALO_FLEX_SLIDING=0sends every attention layer back to plain SDPA. - Masked matmul attention (GLM-5 Next's DSA layers on
sdpa,halo/kernels/masked_softmax.py, warmed at model load at world size above 1, otherwise compiled at the first backward):HALO_DSA_MATMUL=0keeps stock SDPA. - Block-sparse QSA attention (
halo/kernels/block_sparse_attention.py, Qwen3.8-Flash-Next):HALO_QSA_KERNEL=offkeeps the tiled SDPA path; the defaultautofalls back on its own where the kernel cannot run. - Fused GLU (
halo/kernels/fused_glu.py, every expert and dense-MLP combine, grouped and loop paths alike):HALO_FUSED_GLU=0runs each combine through its eager form on CUDA too, and the EP layers' construction lines readglu_combine=eager. On CPU tensors, and for an activation the kernels do not compute exactly, the eager form runs anyway. - Hyper-connection kernels (
halo/kernels/hyper_connection.py, GLM-5 Next and DeepSeek-V4):HALO_HC_KERNELS=0runs upstream's forwards unpatched. - Gated-residual kernels (
halo/kernels/gated_residual.py, Qwen3.8-Flash-Next):HALO_GATED_RESIDUAL_KERNELS=0runs upstream's forward. - Native RMSNorm (a Liger role):
liger_kernel_config: {rms_norm: false}keeps the family's own norm, anduse_liger_kernel: falseturns off every Liger role. - Fused weighted un-permute and routing sort (
halo/kernels/moe_permute.pyand the counting sort ofhalo/kernels/moe_route_sort.py, on every grouped-GEMM path): no switch of their own.use_grouped_gemm: falseavoids both by running the per-expert loop instead of the grouped path. That costs throughput, most with many local experts per rank (Grouped GEMM).
Source on GitHubUpdated 18 November 2026