TRL (Transformers Reinforcement Learning) is Hugging Face's open-source library of trainers for post-training language models. It covers supervised fine-tuning, preference tuning such as DPO, and reinforcement learning such as GRPO.
What it does
TRL gives each post-training method a trainer class, built on the Trainer of Hugging Face Transformers:
A trainer takes a model, a dataset and a config, and handles the tokenization, the loss and the training loop. TRL works with the PEFT library for LoRA (low-rank adaptation) adapters and with Accelerate for multi-GPU runs.
When to use it
TRL is the usual way to post-train a Hugging Face model. It is a good default for a dense model that trains on one GPU or with plain data parallelism.
TRL and Halo
Halo is built on TRL and keeps the Hugging Face model format. It adds expert, context, tensor and expert-tensor parallelism for large and mixture-of-experts models.
In the Halo tech report, on gpt-oss-20b with 4,096-token sequences at batch 1 on B300 GPUs, Halo ran at 20,786 tokens per second per GPU, about 4.8 times stock TRL's 4,320. Halo vs TRL has the comparison.
