Classification
Sequence classification — single-label (multi-class) and multi-label — on AutoModelForSequenceClassification. Trainer ClassificationTrainer (halo/trainers/reward/classification.py), script scripts/training/classification.py. Text-only. For a scalar preference score on the same architecture use Reward Modeling.
EP, TP, ETP, EP+TP, EP+ETP, PP, PP+EP and PP+ETP run, as does plain FSDP2 data parallelism; under PP compute_metrics is served. CP does not — the head pools the whole sequence (matrix).
Dataset
A prompt conversation plus a label column: a string for single-label, a list for multi-label. text_field names a raw text column instead, wrapped as one user turn. A dataset with neither is rejected right after the dataset load. Row shapes: Dataset Formats.
Rows are chat-templated and truncated to max_length — a label describes the whole document, so a shortened document keeps a valid label. A dataset handed to ClassificationTrainer directly (no input_ids column) is instead tokenized untruncated and its over-length rows dropped, with the fraction logged.
The label set comes from the training split: stringified, sorted for deterministic ids, and unioned across ranks on a pre-sharded dataset so every rank builds an identically shaped head. num_labels, label2id and id2label follow from it, never from YAML; single- vs multi-label follows whether label is a list. Labels seen only in validation or test are added with a warning.
ds["test"] is the eval split; a source with no test split evaluates on its validation split, and next to a test split validation contributes labels only.
-1 marks an unlabeled row. A multi-label row treats it as absence and it is dropped from the label set. A single-label split carrying it is refused before the model load when the run reads that split (train always; the eval split when the run evaluates: eval_strategy other than no, or eval_on_start), since the row has no class and the loss would read -1 as an out-of-range index. A split the run reads with no label column at all is refused the same way, by name; one it does not read adds no labels. A split the run does not read is not tokenized. Filter unlabeled rows out of train; for an unlabeled eval split (a GLUE-style test), set dataset: <id>@train and cut the eval split with test_size, or set eval_strategy: no.
Configuration
YAML
model_name_or_path: Qwen/Qwen3.5-9B
dataset: yaful/MAGE
text_field: text
test_size: 0.1
attn_implementation: sdpa
max_length: 2048
per_device_train_batch_size: 4
gradient_accumulation_steps: 8
learning_rate: 2.0e-05
num_train_epochs: 1
gradient_checkpointing: true
output_dir: checkpoints/clf-qwen3.5-9b-mageloss_type picks cross_entropy, focal or label_smoothing_ce, and refuses the other losses' knobs; every field's type and default, and which apply to single- or multi-label heads, is in ClassificationConfig.
class_weights and derive_class_weights are mutually exclusive — setting both raises, on the CLI as well as in YAML. Every objective, the default cross-entropy included, is the trainer's own ClassificationLoss (halo/trainers/reward/classification_loss.py), evaluated in fp32; the head's built-in loss would run in the logits' bf16. A single-label head needs num_labels >= 2 — one logit is transformers' regression convention — and is refused at construction otherwise. HF's own label_smoothing_factor is rejected at construction: the trainer builds its own loss and never reaches HF's label_smoother.
Batches are right-padded, so an unpinned attn_implementation takes the padded-batch default: the loader's own pick on Blackwell (SDPA for a remote-code checkpoint, FA2 for a softcapped policy), SDPA elsewhere (Flash Attention). Under reset_sinks: false it requests nothing and lets the resolver auto-select, which there accepts only a sink-carrying backend.
Launch
Bash
# MoE, expert parallel
torchrun --nproc_per_node=8 scripts/training/classification.py \
examples/classification/gptoss/clf-gptoss-20b-mage-ep.yaml --expert_parallel_size=8
# dense, plain FSDP2 data parallel
halo launch classification examples/classification/qwen3_5/clf-qwen3.5-9b-mage.yaml --nproc 8LoRA takes lora_task_type: SEQ_CLS and is rejected under TP, EP+TP and PP (PEFT).
Vision-language
Not supported. The script loads through the text-only load_model_and_tokenizer, builds no processor, and refuses text_only_model as well as any image column.
A multimodal checkpoint is also checked for a sequence-classification head before the load and refused without one. transformers 5.20 ships that head for a few families and the toolkit registers two more (roster).
Testing a setup
Bash
torchrun --nproc_per_node=2 scripts/training/classification.py <config> \
--max_steps=5 --save_strategy=no --report_to=noneCovering tests: pytest tests/cpu/trainers tests/cpu/config -m cpu and tests/gpu/trainers/other/test_classification.py; under EP, TP and ETP with save, exact resume and export, tests/gpu/parallelism/ep/test_ep_pooled_head_trainers.py --trainer classification; under PP, tests/gpu/parallelism/pp/test_pp_classification.py and test_pp_trainer_checkpoints.py --trainer classification.
What to watch
Failure signatures:
metric_for_best_model: auc_rocraises — eithercompute_auc_rocis off, or an eval slice holds a class with no samples and the metric is undefined there. Rank onaccuracy,f1ormcc.- Documents silently shortened:
max_lengthtruncates rather than drops. Raise it when the label depends on the tail.
Related pages
Source on GitHubUpdated 18 November 2026