Troubleshooting
The failures people hit most, with the fix. The full symptom → cause list is Troubleshooting ↗ in the reference.
First things to check
- You are inside the right Docker image.
halo:blackwellfor B200/B300,halo:hopperfor H100/H200. The host has no usable Python, and an import error forflash_attnordeep_epalmost always means the wrong image. - The container got
--gpus alland--env-file .env. Halo never loads.envitself. The GPUmaketargets (train,test-gpu-*) and the composetrainingservice pass it; a container you start by hand needs the flag. A missingHF_TOKENorWANDB_API_KEYusually means it is missing. - Caches point at a disk with space. Run
df -hon whereverHF_HOME,HF_DATASETS_CACHE,TMPDIRandHALO_DATA_ROOTresolve.
Single-node
| Symptom | Cause → fix |
|---|---|
| CUDA out of memory | Activations dominate. In order: gradient_checkpointing: true, a smaller per_device_train_batch_size or max_length, then sharding (TP for dense, EP or ETP for MoE) or LoRA/QLoRA. |
| CUDA out of memory while a dense model loads under plain FSDP2 or DDP | Every GPU loads the whole model before sharding, so it must fit on one GPU. Use TP (Parallelism). |
Config rejected at startup (must divide, not supported, …) | The validator refuses shapes that would hang or crash mid-run. The message names the rule; valid shapes are in Parallelism. |
expert_parallel_size=N on a single M-GPU NVLink domain forms K concurrent >2-rank DeepEP dispatch groups | EP between 2 and the NVLink domain size on a single-domain job. Use ep_size=2, ep_size = the domain size, or ep4 + etp2 for a 4-way split on 8 GPUs. ep4 + tp2 is rejected the same way. |
| Missing dataset column | Each method reads fixed columns (Datasets). Mixing sources keeps only the columns common to all of them. |
| Host RAM spike or OOM while loading the model | By default half the node's ranks load at once, capped at 4. Set max_concurrent_loading: 1. |
| Loss degrades only past ~2048 tokens | TF32 rounding corrupts long-context RoPE on remote-code models that compute it with a matmul. Halo pins fp32 matmuls to full precision at model load; you see this only after setting HALO_FP32_MATMUL_PRECISION to high or medium. Unset it. |
| Garbage output or an index crash right after dataset mapping | A map whose output depends on a value the cache fingerprint cannot hash. Grep the log for Dataset-map cache fingerprint, and clear HF_DATASETS_CACHE. |
| Zaya raises when gradient checkpointing is on | Zaya cannot train with it (the recompute faults in cuDNN). Set gradient_checkpointing: false. |
None of the inputs have requires_grad=True. Gradients will be None on an image batch | A frozen vision tower under reentrant checkpointing, or its blocks before the first trained one, have nothing to train, so torch warns about them. Ignore it. Blocks holding adapters or trained parameters do not warn, and they take their gradient. |
This rank's batch holds no image while the vision tower trains under data parallelism | A text row's gradient reduces cannot pair with an image rank's, and the step would hang. It happens under accelerate launch, or with a frozen embedding, a trained tower and gradient checkpointing off. Launch with torchrun, turn checkpointing on, train the embedding, or drop the text-only rows. A fine-tune can set freeze_vision_tower: true; with LoRA, keep lora_target_modules off the tower. |
packing=True … is refused for model_type=… (or padding_free …) at startup | The rows would mix their documents with no sign in the loss. Use a flash backend, or turn the option off (Attention backends). |
Optimizer step N: the global gradient norm is nan; the update is skipped | One step's gradients went NaN or inf, so Halo skipped that update and kept training. More than max_consecutive_nonfinite_steps (default 3) in a row stop the run: look at those steps' batches or the model's numerics, then resume from the last checkpoint. |
fp32 training is not supported under Expert Parallelism at model load | bf16: false without fp16 means fp32, and DeepEP's buffer is sized for 2-byte tokens. Train in bf16, or keep fp32 masters with fp32_non_ep_params / fp32_experts (Gemma 4 refuses fp32_non_ep_params under EP). |
| A write error mid-run | A cache or output landed on the small root filesystem. Recheck the four cache variables. |
Resume checkpoint(s) … are incomplete on at least one node | A save stopped partway, so trainer_state.json (written last) is missing. Resume from an earlier checkpoint or remove the incomplete one (Checkpoints). |
optimizer_meta.pt is missing beside the per-rank optimizer shards (or is unreadable) at resume | The optimizer half of an interrupted save. Resume from an earlier checkpoint, or set allow_optimizer_warm_restart: true to keep the weights and start fresh optimizer moments. |
Multi-node and clusters
| Symptom | Cause → fix |
|---|---|
| Nodes never join, or the rendezvous times out | The torchrun arguments differ across nodes, a node started late, the master address is not routable on the fabric (a TCPStore connect timeout naming the host and port), or a stale process holds the port (pkill -f torchrun). |
| Hang at step 0, then a watchdog timeout | A rank never reached a collective. Dump every rank's stack (below, once per node); the rank that is not in a collective is the culprit. |
| Watchdog timeout during a large gathered checkpoint save | Slow work outlasted the watchdog. Raise DIST_NCCL_TIMEOUT_MINUTES (default 30), the one watchdog setting: a config's ddp_timeout is refused. |
| A rank waits hours, then aborts during a download, a dataset map or pack, or a queued model load | The rank going first outlasted DIST_STORE_TIMEOUT_HOURS (default 4). Raise it. |
Output filesystem is declared SHARED but …, Input filesystem … (or … declared PER-NODE …) at startup | The filesystem flag contradicts the mount of output_dir (Output) or of the HF caches (Input). Set DIST_OUTPUT_SHARED_FILESYSTEM or DIST_INPUT_SHARED_FILESYSTEM (or the DIST_SHARED_FILESYSTEM umbrella) to 1 for a shared FS or 0 for per-node disks, or move the directory the message names. On NFS whose attribute cache outlasts the probe, raise DIST_FS_PROBE_TIMEOUT_SECONDS. |
OSError: [Errno 116] Stale file handle while loading a model or dataset cache | A cross-node read-after-write on NFS/EFS. Set DIST_INPUT_SHARED_FILESYSTEM=0 and keep the output side shared (Clusters). |
Resolving the checkpoint '<path>' failed on K of N rank(s), or The checkpoint '<repo>' each rank resolved differs across ranks | Some nodes cannot see the model source, or their Hub caches hold different commits. Put the checkpoint or HF_HOME on a shared mount, or set DIST_INPUT_SHARED_FILESYSTEM=0, and pin model_revision to one commit. |
Dataset <path>: the ranks … that must hold the same rows loaded different ones, or The shard index of <path> differs across ranks | Nodes read different copies of the dataset. Re-sync or delete the stale copy ($HALO_DATA_ROOT/s3_datasets/<md5> for an S3 source) and relaunch. |
Each pipeline stage owns N ranks … not a multiple of the NVLink domain, or a stage left with a single rank | A stage needs whole domains and at least two ranks, so PP needs at least two nodes (Parallelism). |
does not support Pipeline Parallelism (PP), or a PP refusal naming a model, a LoRA or an option | That trainer, family or option has no PP path (what PP refuses). Drop it, or train without PP. |
| Slow cross-node traffic on AWS | EFA needs its own environment (Clusters). The same variables slow down an InfiniBand cluster if left set. |
Xid 145 NVLink messages flood dmesg | Usually benign FEC churn. halo run nvlink-health exits non-zero only on real faults (hard errors, a down link, or a GPU with fewer active links than its peers); trust it over dmesg volume. |
RL runs (vLLM / SGLang)
Server setup is in Rollout Servers, which also covers tool parsers, reasoning caps and port binding.
| Symptom | Cause → fix |
|---|---|
| The weight-sync group never forms | Both containers need network_mode: host; a bridge network hides the ports the server dials back on. Under SGLang, serve from this repo's Dockerfile.sglang image. |
vLLM server … runs vLLM …; this client speaks the weight-transfer protocol of vLLM >= 0.32.0 | The server is older than the trainer's weight-sync client. Serve from this release's vLLM image (Rollout Servers). |
Trainer exits at start with Errno 98 on the weight-sync port | The default port 51216 (plus one per extra server) sits in Linux's ephemeral range. Reserve it in net.ipv4.ip_local_reserved_ports or set group_port outside 32768–60999. |
| Weight sync hangs at the first collective after the group formed | The containers run different NCCL transports or fabric builds. Serve from Halo's images with the same fabric setup on both ends and check the transport (Rollout Servers). |
| Rewards look fine but the policy degrades | Watch sampling/logratio_mean (Monitoring): a steady negative drift means a broken sync or a drifting KL-free run. |
| Rollouts much slower in the run than on the server alone | The vLLM engine core is one CPU thread, and the trainer and sandboxes compete for it. Give each server its own cores (--cpuset-cpus) and run one engine per GPU. |
Startup rejection under rollout_backend: sglang | Weight sync is supported per family and per engine (Supported Matrix). rollout_max_thinking_tokens and carry_reasoning are vLLM-only. Drop the knob the message names, or use rollout_backend: vllm if the matrix lists the family under vLLM. |
ncclP2pImportShareableBuffer ... invalid argument in the SGLang log on the first update | cuMem differs between the containers. Keep the compose default NCCL_CUMEM_ENABLE=1 on the server and restart it, since it holds a half-written model. |
… cannot serve this run: it serves no tool parser … at startup, with could not detect tool-call parser from chat template in the SGLang log | The default auto turned the parser off on this template (the Qwen3 2507 releases, Ling-mini-2.0). Set SGLANG_TOOL_PARSER=qwen25 for Qwen3, hermes for Ling (Rollout Servers). |
… cannot serve this run: routing_replay: rollout needs the routed experts … | Start the server with VLLM_ENABLE_R3=1 or SGLANG_ENABLE_R3=1. --enable-torch-compile on SGLang is refused with it too, since it returns corrupt rows. |
… does not support weight sync … right after launch | No engine takes an online update for this family. Train it with Offline GRPO (Rollout engines). |
Rollout server restarts forever with VLLM_ENABLE_R3 or SGLANG_ENABLE_R3 set | Routed-experts capture is MoE-only; the engine exits on a dense model and the compose restart policy loops it. Unset the variable for a dense model. |
| SGLang exits at start with "Unsupported head dimensions" | GLM-4 MoE Lite on Blackwell. Start the server with --attention-backend triton (SGLANG_ATTENTION_BACKEND=triton for compose). |
GENERATION is paused or wedged at startup | A previous trainer died attached to the engine. Lift the pause (POST /resume on vLLM, /continue_generation on SGLang), or restart the container if that sync had started sending. |
"Excluded N of M lora_target_modules matches from PEFT injection" at startup | Expected. Those modules have no stock LoRA fit (a Gemma 4 vision-tower projection wrapper, DeepSeek-V4's grouped o_a_proj); everything else is adapted. |
a vLLM reasoning cap can be enforced but the importance-sampling correction is off | Turn train_on_sampled_tokens and vllm_importance_sampling_correction back on. |
consecutive steps produced no valid episode anywhere in the world | Two steps in a row had nothing to train on; the message ends with the cause. reward term 'audit' scored nothing is a judge outage: fix OPENROUTER_API_KEY or the provider and resume. A rollout error points at the server or the sandbox. |
A bubblewrap sandbox fails to start (Can't mount proc on /newroot/proc, loopback: Failed RTM_NEWADDR) | The container lacks the rights the sandbox needs (Async GRPO lists them). |
Getting eyes on a hung run
Attach from a second shell in the same container:
Bash
halo run py-spy-diag dump # Python stacks of every rank on this node
halo run py-spy-diag record --duration 30 # flame graphs (dataloader stalls)
halo run nvlink-health # NVLink preflight and verdictpy-spy needs the container started with --cap-add=SYS_PTRACE when the host's
kernel.yama.ptrace_scope is 1 or 2; at 3 it cannot attach. Start every
training container with the flag. The make targets do not add it.
For where the time goes, use enable_torch_profiler: true and
halo run trace-report (Monitoring). The full
toolbox, including the NCCL flight recorder for mismatched collectives, is
Debugging ↗.
Source on GitHubUpdated 18 November 2026