What is TRL?

TRL (Transformers Reinforcement Learning) is Hugging Face's open-source library of trainers for post-training language models. It covers supervised fine-tuning, preference tuning such as DPO, and reinforcement learning such as GRPO.

What it does

TRL gives each post-training method a trainer class, built on the Trainer of Hugging Face Transformers:

TrainerMethod
SFTTrainerSupervised fine-tuning on example conversations
DPOTrainerDPO, from pairs of better and worse answers
GRPOTrainerGRPO, reinforcement learning from a reward
RewardTrainerReward models that score an answer

A trainer takes a model, a dataset and a config, and handles the tokenization, the loss and the training loop. TRL works with the PEFT library for LoRA (low-rank adaptation) adapters and with Accelerate for multi-GPU runs.

When to use it

TRL is the usual way to post-train a Hugging Face model. It is a good default for a dense model that trains on one GPU or with plain data parallelism.

TRL and Halo

Halo is built on TRL and keeps the Hugging Face model format. It adds expert, context, tensor and expert-tensor parallelism for large and mixture-of-experts models.

In the Halo tech report, on gpt-oss-20b with 4,096-token sequences at batch 1 on B300 GPUs, Halo ran at 20,786 tokens per second per GPU, about 4.8 times stock TRL's 4,320. Halo vs TRL has the comparison.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL