Monitoring
Console and log file
The logging rank (global rank 0, or each node's first rank on per-node disks)
copies its console output, including tqdm and NCCL messages, to
<output_dir>/log/run.log. Other ranks' output is not in it. For a detached
run, that file is your console:
Bash
tail -f checkpoints/sft-qwen3-4b-ultrachat/log/run.logThe startup banner prints the parallelism mode and its EP/CP/TP/ETP/PP/DP sizes, the attention backend the model was built with, and a parameter breakdown. Check it to confirm the run has the shape you intended.
Weights & Biases and ClearML
Tracking uses the standard HuggingFace integration:
YAML
report_to: wandb # or clearml, tensorboard, none
project_name: my-project # the W&B or ClearML project
run_name: qwen3-sft-lr1e5 # optional; defaults to <method>-<mode>-<output dir name>Credentials come from your .env: WANDB_API_KEY for W&B, the usual
clearml.conf or CLEARML_API_* setup for ClearML. Halo does not load .env
itself; pass it to the container with --env-file .env.
Halo sets WANDB_PROJECT and CLEARML_PROJECT from project_name, and
CLEARML_TASK from the run name, so you do not set them yourself.
A resumed run starts a new W&B run. To continue the same curves, export
WANDB_RUN_ID=<id> and WANDB_RESUME=allow before relaunching.
What gets logged
The base trainer logs loss, learning rate and grad norm every logging_steps
(most examples use 1). Halo adds these metric groups:
Defaults and the exact keys: CommonScriptArguments ↗.
RL health
For async GRPO with environments, keep sampling/logratio_mean on a dashboard
panel. A steady negative drift means one of two things:
- the weight sync to the rollout server is broken;
- a KL-free run is drifting, if
advantage/net_token_massstays negative andentropyclimbs after it. The early stop ↗ guards against it;balance_token_masscancels the net push, whose measured entropy effect is small.
Online GRPO logs TRL's unsigned gap sampling/sampling_logp_difference/mean
instead, with the importance-sampling correction on. Every callback and metric:
Callbacks ↗.
Profiling
Set enable_torch_profiler: true to see where the time goes. It captures a
Chrome trace on rank 0, with the EP phases (dispatch, expert compute, combine)
labeled; profiler_ranks: "all" captures every rank. halo run trace-report
turns the traces into a compute, communication and idle breakdown.
Hangs, memory snapshots and the rest of the toolbox: Debugging ↗.
Source on GitHubUpdated 18 November 2026