Classification

Sequence classification — single-label (multi-class) and multi-label — on AutoModelForSequenceClassification. Trainer ClassificationTrainer (halo/trainers/reward/classification.py), script scripts/training/classification.py. Text-only. For a scalar preference score on the same architecture use Reward Modeling.

EP, TP, ETP, EP+TP, EP+ETP, PP, PP+EP and PP+ETP run, as does plain FSDP2 data parallelism; under PP compute_metrics is served. CP does not — the head pools the whole sequence (matrix).

Dataset

A prompt conversation plus a label column: a string for single-label, a list for multi-label. text_field names a raw text column instead, wrapped as one user turn. A dataset with neither is rejected right after the dataset load. Row shapes: Dataset Formats.

Rows are chat-templated and truncated to max_length — a label describes the whole document, so a shortened document keeps a valid label. A dataset handed to ClassificationTrainer directly (no input_ids column) is instead tokenized untruncated and its over-length rows dropped, with the fraction logged.

The label set comes from the training split: stringified, sorted for deterministic ids, and unioned across ranks on a pre-sharded dataset so every rank builds an identically shaped head. num_labels, label2id and id2label follow from it, never from YAML; single- vs multi-label follows whether label is a list. Labels seen only in validation or test are added with a warning.

ds["test"] is the eval split; a source with no test split evaluates on its validation split, and next to a test split validation contributes labels only.

-1 marks an unlabeled row. A multi-label row treats it as absence and it is dropped from the label set. A single-label split carrying it is refused before the model load when the run reads that split (train always; the eval split when the run evaluates: eval_strategy other than no, or eval_on_start), since the row has no class and the loss would read -1 as an out-of-range index. A split the run reads with no label column at all is refused the same way, by name; one it does not read adds no labels. A split the run does not read is not tokenized. Filter unlabeled rows out of train; for an unlabeled eval split (a GLUE-style test), set dataset: <id>@train and cut the eval split with test_size, or set eval_strategy: no.

Configuration

YAML

model_name_or_path: Qwen/Qwen3.5-9B
dataset: yaful/MAGE
text_field: text
test_size: 0.1
attn_implementation: sdpa
max_length: 2048
per_device_train_batch_size: 4
gradient_accumulation_steps: 8
learning_rate: 2.0e-05
num_train_epochs: 1
gradient_checkpointing: true
output_dir: checkpoints/clf-qwen3.5-9b-mage

loss_type picks cross_entropy, focal or label_smoothing_ce, and refuses the other losses' knobs; every field's type and default, and which apply to single- or multi-label heads, is in ClassificationConfig.

class_weights and derive_class_weights are mutually exclusive — setting both raises, on the CLI as well as in YAML. Every objective, the default cross-entropy included, is the trainer's own ClassificationLoss (halo/trainers/reward/classification_loss.py), evaluated in fp32; the head's built-in loss would run in the logits' bf16. A single-label head needs num_labels >= 2 — one logit is transformers' regression convention — and is refused at construction otherwise. HF's own label_smoothing_factor is rejected at construction: the trainer builds its own loss and never reaches HF's label_smoother.

Batches are right-padded, so an unpinned attn_implementation takes the padded-batch default: the loader's own pick on Blackwell (SDPA for a remote-code checkpoint, FA2 for a softcapped policy), SDPA elsewhere (Flash Attention). Under reset_sinks: false it requests nothing and lets the resolver auto-select, which there accepts only a sink-carrying backend.

Launch

Bash

# MoE, expert parallel
torchrun --nproc_per_node=8 scripts/training/classification.py \
    examples/classification/gptoss/clf-gptoss-20b-mage-ep.yaml --expert_parallel_size=8

# dense, plain FSDP2 data parallel
halo launch classification examples/classification/qwen3_5/clf-qwen3.5-9b-mage.yaml --nproc 8

LoRA takes lora_task_type: SEQ_CLS and is rejected under TP, EP+TP and PP (PEFT).

Vision-language

Not supported. The script loads through the text-only load_model_and_tokenizer, builds no processor, and refuses text_only_model as well as any image column.

A multimodal checkpoint is also checked for a sequence-classification head before the load and refused without one. transformers 5.20 ships that head for a few families and the toolkit registers two more (roster).

Testing a setup

Bash

torchrun --nproc_per_node=2 scripts/training/classification.py <config> \
    --max_steps=5 --save_strategy=no --report_to=none

Covering tests: pytest tests/cpu/trainers tests/cpu/config -m cpu and tests/gpu/trainers/other/test_classification.py; under EP, TP and ETP with save, exact resume and export, tests/gpu/parallelism/ep/test_ep_pooled_head_trainers.py --trainer classification; under PP, tests/gpu/parallelism/pp/test_pp_classification.py and test_pp_trainer_checkpoints.py --trainer classification.

What to watch

SignalReading
accuracySingle-label; multi-label reports exact_match_accuracy instead
f1Binary averaging at 2 labels, weighted for multi-class, macro for multi-label
mccSingle-label (compute_mcc); stays honest under class imbalance

Failure signatures:

  • metric_for_best_model: auc_roc raises — either compute_auc_roc is off, or an eval slice holds a class with no samples and the metric is undefined there. Rank on accuracy, f1 or mcc.
  • Documents silently shortened: max_length truncates rather than drops. Raise it when the label depends on the tail.

Source on GitHubUpdated 18 November 2026