Training throughput

Training throughput is how many tokens a training run processes per second, usually quoted per GPU. It sets how long a run takes and what it costs.

What it measures

Throughput is the tokens in a training step (forward pass, backward pass, weight update) divided by the step time. Quoting it per GPU makes runs of different sizes comparable: multiply by the GPU count for the whole run.

The tokens you have to train on, divided by that total, give how long the run takes.

What moves it

A number means little without its conditions: model, GPU, sequence length, batch size. Within a run, four things move it:

  • Tokens per step. A small batch of short sequences leaves the GPU waiting.
  • Padding. Sequence packing stops compute going to pad tokens.
  • Gradient checkpointing. It saves memory and costs speed.
  • Sharding. Splitting a model across GPUs adds communication, so use the least that fits.

In Halo

Set enable_efficiency_metrics: true to log tokens per second per GPU. The performance guide reports gpt-oss-20b on 8 B300 GPUs at batch 4 × 4096: 28,700 tokens per second per GPU without expert parallelism or gradient checkpointing, and 13,200 with the experts split over all eight GPUs and checkpointing on.

In the Halo tech report, on gpt-oss-20b with 4,096-token sequences at batch 1 on B300 GPUs, Halo ran at 20,786 tokens per second per GPU, Megatron-LM at 8,645 and stock TRL at 4,320.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL