Monitoring

Console and log file

The logging rank (global rank 0, or each node's first rank on per-node disks) copies its console output, including tqdm and NCCL messages, to <output_dir>/log/run.log. Other ranks' output is not in it. For a detached run, that file is your console:

Bash

tail -f checkpoints/sft-qwen3-4b-ultrachat/log/run.log

The startup banner prints the parallelism mode and its EP/CP/TP/ETP/PP/DP sizes, the attention backend the model was built with, and a parameter breakdown. Check it to confirm the run has the shape you intended.

Weights & Biases and ClearML

Tracking uses the standard HuggingFace integration:

YAML

report_to: wandb          # or clearml, tensorboard, none
project_name: my-project  # the W&B or ClearML project
run_name: qwen3-sft-lr1e5 # optional; defaults to <method>-<mode>-<output dir name>

Credentials come from your .env: WANDB_API_KEY for W&B, the usual clearml.conf or CLEARML_API_* setup for ClearML. Halo does not load .env itself; pass it to the container with --env-file .env.

Halo sets WANDB_PROJECT and CLEARML_PROJECT from project_name, and CLEARML_TASK from the run name, so you do not set them yourself.

A resumed run starts a new W&B run. To continue the same curves, export WANDB_RUN_ID=<id> and WANDB_RESUME=allow before relaunching.

What gets logged

The base trainer logs loss, learning rate and grad norm every logging_steps (most examples use 1). Halo adds these metric groups:

Config fieldAdds
enable_efficiency_metricsstep time, tokens/s per GPU and cluster-wide, allocated and peak GPU memory
report_mfu_diagnosticsMFU and achieved TFLOPS; needs enable_efficiency_metrics. MoE runs also get the S-MFU variants, since plain MFU misreads sparse models
enable_moe_metricsexpert load balance averaged over MoE layers (moe/load_max, moe/load_cv, moe/dead_frac, …); under bias_update the balancing callback logs the moe/* keys instead
generate_eval_examples (SFT, DPO, SMPO, offline GRPO)a table of sample generations at each evaluation; skipped under TP, CP and PP
save_completions (online and async GRPO)each logged step's rollouts as parquet under <output_dir>/completions/, and a W&B completions table when report_to includes wandb
log_completions (online and async GRPO)also prints the per-sample table to the console

Defaults and the exact keys: CommonScriptArguments ↗.

RL health

For async GRPO with environments, keep sampling/logratio_mean on a dashboard panel. A steady negative drift means one of two things:

  • the weight sync to the rollout server is broken;
  • a KL-free run is drifting, if advantage/net_token_mass stays negative and entropy climbs after it. The early stop ↗ guards against it; balance_token_mass cancels the net push, whose measured entropy effect is small.

Online GRPO logs TRL's unsigned gap sampling/sampling_logp_difference/mean instead, with the importance-sampling correction on. Every callback and metric: Callbacks ↗.

Profiling

Set enable_torch_profiler: true to see where the time goes. It captures a Chrome trace on rank 0, with the EP phases (dispatch, expert compute, combine) labeled; profiler_ranks: "all" captures every rank. halo run trace-report turns the traces into a compute, communication and idle breakdown.

Hangs, memory snapshots and the rest of the toolbox: Debugging ↗.

Source on GitHubUpdated 18 November 2026