Fine-tuning for classification

Classification fine-tuning puts a small classification head on a language model and trains it to assign labels to text. The result returns a score per label in one forward pass, with no text generated.

How it works

The language model reads the text, and a small linear head turns its final hidden state into one score per label. Training updates the head and, usually, the model under it.

  • Single-label. Each text has exactly one class. The scores go through a softmax and the loss is cross-entropy.
  • Multi-label. A text can carry several labels or none. Each label gets its own sigmoid and a threshold.

Why use it

A classifier answers in one forward pass, so it is faster and cheaper than asking a chat model to write the label. It also returns a score, so you can move the threshold to trade precision against recall. Typical uses are safety filters for content moderation, topic routing and detectors. Many guard models are classifiers of this kind.

With imbalanced classes, accuracy misleads. Weight the classes and track F1.

In Halo

Data is a prompt conversation or a raw text column named by text_field, plus a label column: one value per row for single-label, a list for multi-label. The label set is derived from the training split.

Bash

halo launch classification examples/classification/qwen3_5/clf-qwen3.5-9b-mage.yaml -n 8

For imbalanced data, set class_weights or turn on derive_class_weights. LoRA (low-rank adaptation) takes lora_task_type: SEQ_CLS. The trainer is text-only. See the classification guide.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL